{
  "id": 34441,
  "title": "Result reproducibility",
  "url": "/competitions/intel-mobileodt-cervical-cancer-screening/discussion/34441",
  "author_name": "",
  "post_date": "2017-06-09T07:13:39.950083300Z",
  "votes": 2,
  "comment_count": 15,
  "views": 0,
  "content": "<p>Hi guys,</p>\n\n<p>how do you manage to get reproductible results with:</p>\n\n<p>Keras</p>\n\n<p>Tensorflow</p>\n\n<p>cuDNN</p>\n\n<p>GPU</p>\n\n<p>I've tried some things found on the net, but I get the impression that it's not possible currently unless you run the model on the CPU, on a single thread. Would greatly appreciate some help on this !</p>",
  "messages": [
    {
      "id": "191056",
      "postDate": "06/09/2017 07:13:39",
      "content": "<p>Hi guys,</p>\n\n<p>how do you manage to get reproductible results with:</p>\n\n<p>Keras</p>\n\n<p>Tensorflow</p>\n\n<p>cuDNN</p>\n\n<p>GPU</p>\n\n<p>I've tried some things found on the net, but I get the impression that it's not possible currently unless you run the model on the CPU, on a single thread. Would greatly appreciate some help on this !</p>",
      "rawMarkdown": "Hi guys,\n\nhow do you manage to get reproductible results with:\n\nKeras\n\nTensorflow\n\ncuDNN\n\nGPU\n\nI've tried some things found on the net, but I get the impression that it's not possible currently unless you run the model on the CPU, on a single thread. Would greatly appreciate some help on this !",
      "votes": null
    },
    {
      "id": "191063",
      "postDate": "06/09/2017 07:44:11",
      "content": "<p>I've switched to Theano as Keras backend with \"conv.algo_bwd_data=deterministic\" and \"conv.algo_bwd_filter=deterministic\" in config file. Looks reproducible. Was not able to reproduce with Tensorflow.</p>",
      "rawMarkdown": "I've switched to Theano as Keras backend with \"conv.algo_bwd_data=deterministic\" and \"conv.algo_bwd_filter=deterministic\" in config file. Looks reproducible. Was not able to reproduce with Tensorflow.",
      "votes": null
    },
    {
      "id": "191109",
      "postDate": "06/09/2017 10:21:48",
      "content": "<p>I confirm this.  Can't do this with tensorflow backend...</p>",
      "rawMarkdown": "I confirm this.  Can't do this with tensorflow backend...",
      "votes": null
    },
    {
      "id": "191204",
      "postDate": "06/09/2017 15:40:18",
      "content": "<p>If you fix the tensorflow seed, numpy seed, and random.random seed it will be practically reproducible. I have done that and am still observing some small discrepancies. I am curious what is considered \"close enough\" for the competition. I find my results will be very stable over multiple runs as long as I average over multiple random seeds.</p>",
      "rawMarkdown": "If you fix the tensorflow seed, numpy seed, and random.random seed it will be practically reproducible. I have done that and am still observing some small discrepancies. I am curious what is considered \"close enough\" for the competition. I find my results will be very stable over multiple runs as long as I average over multiple random seeds.",
      "votes": null
    },
    {
      "id": "191213",
      "postDate": "06/09/2017 15:50:26",
      "content": "<p>I've found this one:</p>\n\n<p><a href=\"https://github.com/fchollet/keras/issues/2280\">https://github.com/fchollet/keras/issues/2280</a></p>\n\n<p>to almost work. However, even without using Dropout, it's still not identical - very close, but different. </p>\n\n<p>My understanding is that to claim the prize, you need to be able to reproduce exactly the same model / parameters - so 'very close' doesn't make the cut.</p>",
      "rawMarkdown": "I've found this one:\n\nhttps://github.com/fchollet/keras/issues/2280\n\nto almost work. However, even without using Dropout, it's still not identical - very close, but different. \n\nMy understanding is that to claim the prize, you need to be able to reproduce exactly the same model / parameters - so 'very close' doesn't make the cut.",
      "votes": null
    },
    {
      "id": "191871",
      "postDate": "06/12/2017 02:32:06",
      "content": "<p>I seem to be getting exact reproducibility with Theano and Keras by setting the np.random.seed first. </p>",
      "rawMarkdown": "I seem to be getting exact reproducibility with Theano and Keras by setting the np.random.seed first.",
      "votes": null
    },
    {
      "id": "192051",
      "postDate": "06/12/2017 14:45:52",
      "content": "<p>Yeah that's what I had found as well. I'm finding by averaging seeds though it is exact within an epsilon.</p>\n\n<p>I hear what you are saying though. If that is true then basically all keras/tf users will be unable to claim prizes. What can we do about this? Is it worth it just to switch to theano? </p>",
      "rawMarkdown": "Yeah that's what I had found as well. I'm finding by averaging seeds though it is exact within an epsilon.\n\nI hear what you are saying though. If that is true then basically all keras/tf users will be unable to claim prizes. What can we do about this? Is it worth it just to switch to theano?",
      "votes": null
    },
    {
      "id": "192070",
      "postDate": "06/12/2017 16:03:01",
      "content": "<p>So was the final solution for both of you to switch to Theanos? Is that our only option if we want to be eligible for phase 2?</p>",
      "rawMarkdown": "So was the final solution for both of you to switch to Theanos? Is that our only option if we want to be eligible for phase 2?",
      "votes": null
    },
    {
      "id": "192078",
      "postDate": "06/12/2017 16:30:00",
      "content": "<p>I could be wrong but my guess is that if your solution is reproducible within some reasonable epsilon and rerunning it does not shift your ranking it should not be an issue(it might be an issue if the top few positions are very very close though). There are large number of people using keras/tensorflow preventing those solutions from being accepted does not seem like the right thing to do.  </p>",
      "rawMarkdown": "I could be wrong but my guess is that if your solution is reproducible within some reasonable epsilon and rerunning it does not shift your ranking it should not be an issue(it might be an issue if the top few positions are very very close though). There are large number of people using keras/tensorflow preventing those solutions from being accepted does not seem like the right thing to do.",
      "votes": null
    },
    {
      "id": "192083",
      "postDate": "06/12/2017 16:48:40",
      "content": "<p>That is what my assumption was as well but now that this is being discussed I'm worried/curious. @WendyKan can we get an official ruling on this?</p>",
      "rawMarkdown": "That is what my assumption was as well but now that this is being discussed I'm worried/curious. @WendyKan can we get an official ruling on this?",
      "votes": null
    },
    {
      "id": "192295",
      "postDate": "06/13/2017 05:09:49",
      "content": "<p>It's highly unlikely for me to be in a prize position anyway, and some of the models are in tensorflow, so I'll just not worry about that for this comp...</p>",
      "rawMarkdown": "It's highly unlikely for me to be in a prize position anyway, and some of the models are in tensorflow, so I'll just not worry about that for this comp...",
      "votes": null
    },
    {
      "id": "192302",
      "postDate": "06/13/2017 05:29:42",
      "content": "<p>Our general rule is that you will have to generate the exact submission file with your code in order to claim a prize. However, in practice, a small epsilon shouldn't be that big of an issue. It is up to the host of the competition how strict they want to be. </p>",
      "rawMarkdown": "Our general rule is that you will have to generate the exact submission file with your code in order to claim a prize. However, in practice, a small epsilon shouldn't be that big of an issue. It is up to the host of the competition how strict they want to be.",
      "votes": null
    },
    {
      "id": "192713",
      "postDate": "06/14/2017 14:35:45",
      "content": "<p>Hi Victor, I'm curious if you saw any adverse affects when switching to Theano. I'm finding that some of my deeper networks train much slower when using Theano as a backend than with tensorflow. Not sure if that is a Keras specific problem or if I configured Theano incorrectly or what.</p>",
      "rawMarkdown": "Hi Victor, I'm curious if you saw any adverse affects when switching to Theano. I'm finding that some of my deeper networks train much slower when using Theano as a backend than with tensorflow. Not sure if that is a Keras specific problem or if I configured Theano incorrectly or what.",
      "votes": null
    },
    {
      "id": "192716",
      "postDate": "06/14/2017 14:48:32",
      "content": "<p>Theano should be even faster for some networks. Be sure you using \"image_data_format\": \"channels_first\"\nchannels_first - is native order for Theano. with channels_last it will be much slower than Tensorflow (where channels_last is native)</p>",
      "rawMarkdown": "Theano should be even faster for some networks. Be sure you using \"image_data_format\": \"channels_first\"\nchannels_first - is native order for Theano. with channels_last it will be much slower than Tensorflow (where channels_last is native)",
      "votes": null
    },
    {
      "id": "192747",
      "postDate": "06/14/2017 17:07:36",
      "content": "<p>Thanks, that seems to have been it.</p>",
      "rawMarkdown": "Thanks, that seems to have been it.",
      "votes": null
    },
    {
      "id": "192949",
      "postDate": "06/15/2017 07:09:15",
      "content": "<p>Wendy - Going forward for other Kaggle contests, will we have to code to consistently generate the same model?  I was using Keras and Tensorflow so my code probably won't generate the exact same model.  I found out about this too late to switch over to Keras/Theano.  That's fine.  I don't think that I'm in the money so it's ok.  I just want to know for the future.  In that case, I'll just switch to Keras/Theano from the start for future contests.</p>",
      "rawMarkdown": "Wendy - Going forward for other Kaggle contests, will we have to code to consistently generate the same model?  I was using Keras and Tensorflow so my code probably won't generate the exact same model.  I found out about this too late to switch over to Keras/Theano.  That's fine.  I don't think that I'm in the money so it's ok.  I just want to know for the future.  In that case, I'll just switch to Keras/Theano from the start for future contests.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 191063,
      "author_name": "victorsd",
      "author_url": "",
      "post_date": "06/09/2017 07:44:11",
      "content": "<p>I've switched to Theano as Keras backend with \"conv.algo_bwd_data=deterministic\" and \"conv.algo_bwd_filter=deterministic\" in config file. Looks reproducible. Was not able to reproduce with Tensorflow.</p>",
      "votes": null,
      "replies": [
        {
          "id": 191109,
          "author_name": "kubilai",
          "author_url": "",
          "post_date": "06/09/2017 10:21:48",
          "content": "<p>I confirm this.  Can't do this with tensorflow backend...</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 192070,
          "author_name": "gkericks",
          "author_url": "",
          "post_date": "06/12/2017 16:03:01",
          "content": "<p>So was the final solution for both of you to switch to Theanos? Is that our only option if we want to be eligible for phase 2?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 192295,
          "author_name": "kubilai",
          "author_url": "",
          "post_date": "06/13/2017 05:09:49",
          "content": "<p>It's highly unlikely for me to be in a prize position anyway, and some of the models are in tensorflow, so I'll just not worry about that for this comp...</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 192713,
          "author_name": "gkericks",
          "author_url": "",
          "post_date": "06/14/2017 14:35:45",
          "content": "<p>Hi Victor, I'm curious if you saw any adverse affects when switching to Theano. I'm finding that some of my deeper networks train much slower when using Theano as a backend than with tensorflow. Not sure if that is a Keras specific problem or if I configured Theano incorrectly or what.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 192716,
          "author_name": "victorsd",
          "author_url": "",
          "post_date": "06/14/2017 14:48:32",
          "content": "<p>Theano should be even faster for some networks. Be sure you using \"image_data_format\": \"channels_first\"\nchannels_first - is native order for Theano. with channels_last it will be much slower than Tensorflow (where channels_last is native)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 192747,
          "author_name": "gkericks",
          "author_url": "",
          "post_date": "06/14/2017 17:07:36",
          "content": "<p>Thanks, that seems to have been it.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 191204,
      "author_name": "gkericks",
      "author_url": "",
      "post_date": "06/09/2017 15:40:18",
      "content": "<p>If you fix the tensorflow seed, numpy seed, and random.random seed it will be practically reproducible. I have done that and am still observing some small discrepancies. I am curious what is considered \"close enough\" for the competition. I find my results will be very stable over multiple runs as long as I average over multiple random seeds.</p>",
      "votes": null,
      "replies": [
        {
          "id": 191213,
          "author_name": "alchemist",
          "author_url": "",
          "post_date": "06/09/2017 15:50:26",
          "content": "<p>I've found this one:</p>\n\n<p><a href=\"https://github.com/fchollet/keras/issues/2280\">https://github.com/fchollet/keras/issues/2280</a></p>\n\n<p>to almost work. However, even without using Dropout, it's still not identical - very close, but different. </p>\n\n<p>My understanding is that to claim the prize, you need to be able to reproduce exactly the same model / parameters - so 'very close' doesn't make the cut.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 192051,
          "author_name": "gkericks",
          "author_url": "",
          "post_date": "06/12/2017 14:45:52",
          "content": "<p>Yeah that's what I had found as well. I'm finding by averaging seeds though it is exact within an epsilon.</p>\n\n<p>I hear what you are saying though. If that is true then basically all keras/tf users will be unable to claim prizes. What can we do about this? Is it worth it just to switch to theano? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 192078,
          "author_name": "devinanzelmo",
          "author_url": "",
          "post_date": "06/12/2017 16:30:00",
          "content": "<p>I could be wrong but my guess is that if your solution is reproducible within some reasonable epsilon and rerunning it does not shift your ranking it should not be an issue(it might be an issue if the top few positions are very very close though). There are large number of people using keras/tensorflow preventing those solutions from being accepted does not seem like the right thing to do.  </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 192083,
          "author_name": "gkericks",
          "author_url": "",
          "post_date": "06/12/2017 16:48:40",
          "content": "<p>That is what my assumption was as well but now that this is being discussed I'm worried/curious. @WendyKan can we get an official ruling on this?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 192302,
          "author_name": "wendykan",
          "author_url": "",
          "post_date": "06/13/2017 05:29:42",
          "content": "<p>Our general rule is that you will have to generate the exact submission file with your code in order to claim a prize. However, in practice, a small epsilon shouldn't be that big of an issue. It is up to the host of the competition how strict they want to be. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 192949,
          "author_name": "ekkus93",
          "author_url": "",
          "post_date": "06/15/2017 07:09:15",
          "content": "<p>Wendy - Going forward for other Kaggle contests, will we have to code to consistently generate the same model?  I was using Keras and Tensorflow so my code probably won't generate the exact same model.  I found out about this too late to switch over to Keras/Theano.  That's fine.  I don't think that I'm in the money so it's ok.  I just want to know for the future.  In that case, I'll just switch to Keras/Theano from the start for future contests.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 191871,
      "author_name": "tanstaafl",
      "author_url": "",
      "post_date": "06/12/2017 02:32:06",
      "content": "<p>I seem to be getting exact reproducibility with Theano and Keras by setting the np.random.seed first. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "191056": "Hi guys,\n\nhow do you manage to get reproductible results with:\n\nKeras\n\nTensorflow\n\ncuDNN\n\nGPU\n\nI've tried some things found on the net, but I get the impression that it's not possible currently unless you run the model on the CPU, on a single thread. Would greatly appreciate some help on this !",
    "191063": "I've switched to Theano as Keras backend with \"conv.algo_bwd_data=deterministic\" and \"conv.algo_bwd_filter=deterministic\" in config file. Looks reproducible. Was not able to reproduce with Tensorflow.",
    "191109": "I confirm this.  Can't do this with tensorflow backend...",
    "191204": "If you fix the tensorflow seed, numpy seed, and random.random seed it will be practically reproducible. I have done that and am still observing some small discrepancies. I am curious what is considered \"close enough\" for the competition. I find my results will be very stable over multiple runs as long as I average over multiple random seeds.",
    "191213": "I've found this one:\n\nhttps://github.com/fchollet/keras/issues/2280\n\nto almost work. However, even without using Dropout, it's still not identical - very close, but different. \n\nMy understanding is that to claim the prize, you need to be able to reproduce exactly the same model / parameters - so 'very close' doesn't make the cut.",
    "191871": "I seem to be getting exact reproducibility with Theano and Keras by setting the np.random.seed first.",
    "192051": "Yeah that's what I had found as well. I'm finding by averaging seeds though it is exact within an epsilon.\n\nI hear what you are saying though. If that is true then basically all keras/tf users will be unable to claim prizes. What can we do about this? Is it worth it just to switch to theano?",
    "192070": "So was the final solution for both of you to switch to Theanos? Is that our only option if we want to be eligible for phase 2?",
    "192078": "I could be wrong but my guess is that if your solution is reproducible within some reasonable epsilon and rerunning it does not shift your ranking it should not be an issue(it might be an issue if the top few positions are very very close though). There are large number of people using keras/tensorflow preventing those solutions from being accepted does not seem like the right thing to do.",
    "192083": "That is what my assumption was as well but now that this is being discussed I'm worried/curious. @WendyKan can we get an official ruling on this?",
    "192295": "It's highly unlikely for me to be in a prize position anyway, and some of the models are in tensorflow, so I'll just not worry about that for this comp...",
    "192302": "Our general rule is that you will have to generate the exact submission file with your code in order to claim a prize. However, in practice, a small epsilon shouldn't be that big of an issue. It is up to the host of the competition how strict they want to be.",
    "192713": "Hi Victor, I'm curious if you saw any adverse affects when switching to Theano. I'm finding that some of my deeper networks train much slower when using Theano as a backend than with tensorflow. Not sure if that is a Keras specific problem or if I configured Theano incorrectly or what.",
    "192716": "Theano should be even faster for some networks. Be sure you using \"image_data_format\": \"channels_first\"\nchannels_first - is native order for Theano. with channels_last it will be much slower than Tensorflow (where channels_last is native)",
    "192747": "Thanks, that seems to have been it.",
    "192949": "Wendy - Going forward for other Kaggle contests, will we have to code to consistently generate the same model?  I was using Keras and Tensorflow so my code probably won't generate the exact same model.  I found out about this too late to switch over to Keras/Theano.  That's fine.  I don't think that I'm in the money so it's ok.  I just want to know for the future.  In that case, I'll just switch to Keras/Theano from the start for future contests."
  },
  "source": "meta"
}