We have all been experiencing the issue with reproducability. This thread is dedicated to dicuss various approaches teams and kagglers have been using.
\n\nnp.random.seed(2018)tf.set_random_seed(2)The backend is configured to use a sophisticated stack of GPU libraries, and that some of these may introduce their own source of randomness that you may or may not be able to account for. Alike GPU in our kernels
\n\nLet me know if you guys have any other awesome thoughts on making results more reproducible for this competition. I am getting deviations from 0.692 - 0.698 . My local CV and the leaderboard scores are correlating as well.
\n\nCredit - @bkkaggle
\n\nSEED = 1337\nnp.random.seed(SEED)\ntorch.manual_seed(SEED)\ntorch.cuda.manual_seed(SEED)\ntorch.backends.cudnn.deterministic = True\nCredit - @shujian
\n\nA Workaround for Non-Determinism in TensorFlow
", "messages": [ { "id": "431377", "postDate": "12/02/2018 06:29:25", "content": "We have all been experiencing the issue with reproducability. This thread is dedicated to dicuss various approaches teams and kagglers have been using.
\n\nnp.random.seed(2018)tf.set_random_seed(2)The backend is configured to use a sophisticated stack of GPU libraries, and that some of these may introduce their own source of randomness that you may or may not be able to account for. Alike GPU in our kernels
\n\nLet me know if you guys have any other awesome thoughts on making results more reproducible for this competition. I am getting deviations from 0.692 - 0.698 . My local CV and the leaderboard scores are correlating as well.
\n\nCredit - @bkkaggle
\n\nSEED = 1337\nnp.random.seed(SEED)\ntorch.manual_seed(SEED)\ntorch.cuda.manual_seed(SEED)\ntorch.backends.cudnn.deterministic = True\nCredit - @shujian
\n\nA Workaround for Non-Determinism in TensorFlow
", "rawMarkdown": "We have all been experiencing the issue with reproducability. This thread is dedicated to dicuss various approaches teams and kagglers have been using. \n\n## What causes it? \n 1. Randomness in Initialization, such as weights. \n 2. Randomness in Regularization, such as dropout. \n 3. Randomness in Layers, such as word embedding or Optimization\n 4. Randomness introduced by any randomness from third party library \n\n## Methods to reproduce on CPU \n1. Set random seeds for numpy operations/splits `np.random.seed(2018)`\n2. Set random seed for Tensorflow operations from `tf.set_random_seed(2)`\n\n## Why Reproducibility is hard when we train on GPU\nThe backend is configured to use a sophisticated stack of GPU libraries, and that some of these may introduce their own source of randomness that you may or may not be able to account for. Alike GPU in our kernels \n\n### Dealing with GPU reproducibility \n1. Rerun the experiment 10 - 15 times and average the scores to get the progress of your model.\n\nLet me know if you guys have any other awesome thoughts on making results more reproducible for this competition. I am getting deviations from 0.692 - 0.698 . My local CV and the leaderboard scores are correlating as well.\n\n\n## Various methods from kernels/discussion so far\n\n### Pytorch \nCredit - @bkkaggle\n\n SEED = 1337\n np.random.seed(SEED)\n torch.manual_seed(SEED)\n torch.cuda.manual_seed(SEED)\n torch.backends.cudnn.deterministic = True\n\n### Tensorflow \nCredit - @shujian\n\n[A Workaround for Non-Determinism in TensorFlow][1]\n\n\n [1]: http://%20https://www.twosigma.com/insights/article/a-workaround-for-non-determinism-in-tensorflow/", "votes": null }, { "id": "431423", "postDate": "12/02/2018 08:47:32", "content": "https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/72040
\n\nI think this discussion mostly contains about reproducibility. I think with keras it's gonna be tough.
", "rawMarkdown": "https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/72040\n\nI think this discussion mostly contains about reproducibility. I think with keras it's gonna be tough.", "votes": null }, { "id": "431496", "postDate": "12/02/2018 11:41:18", "content": "I missed that thread. Btw, are you using pytorch. How repeatable are the results in your case?
", "rawMarkdown": "I missed that thread. Btw, are you using pytorch. How repeatable are the results in your case?", "votes": null }, { "id": "431649", "postDate": "12/02/2018 17:16:18", "content": "No, I have not tried pytorch but with keras, the same kernel is always giving me different results.
", "rawMarkdown": "No, I have not tried pytorch but with keras, the same kernel is always giving me different results.", "votes": null }, { "id": "431834", "postDate": "12/03/2018 01:36:59", "content": "I think it is good this has it's own thread. All the keras\\cudnn specific discussion has been in kernel threads.
\n\nEventually after dealing with all the places of randomness I was able to make local code run deterministically. This hopefully will be helpful for exploration. But none of this works on kaggle kernels. I also have no issue locally with Cudnn and on Kaggle kernels even super simple non RNN kernels are not deterministic.
\n\nI shared a simple kernel to show that the issue is not Cudnn and give people a fast way to test their environment for randomness.
\n\nhttps://www.kaggle.com/joeytaj/gpu-randomness-exploration
\n\nMy local environment is \nWindows 10\nGTX 1080\nTensorflow 1.12\nKeras 2.2.4\ncudnn 7.1.4\ncuda 9.0
", "rawMarkdown": "I think it is good this has it's own thread. All the keras\\cudnn specific discussion has been in kernel threads.\n\nEventually after dealing with all the places of randomness I was able to make local code run deterministically. This hopefully will be helpful for exploration. But none of this works on kaggle kernels. I also have no issue locally with Cudnn and on Kaggle kernels even super simple non RNN kernels are not deterministic.\n\nI shared a simple kernel to show that the issue is not Cudnn and give people a fast way to test their environment for randomness.\n\nhttps://www.kaggle.com/joeytaj/gpu-randomness-exploration\n\n\nMy local environment is \nWindows 10\nGTX 1080\nTensorflow 1.12\nKeras 2.2.4\ncudnn 7.1.4\ncuda 9.0", "votes": null }, { "id": "431969", "postDate": "12/03/2018 07:36:22", "content": "I'm seeing about a 0.001 variation using the same kernel now... And my score between local and public is pretty stable. Using Keras... CuDNN-variants.
", "rawMarkdown": "I'm seeing about a 0.001 variation using the same kernel now... And my score between local and public is pretty stable. Using Keras... CuDNN-variants.", "votes": null }, { "id": "432930", "postDate": "12/04/2018 13:43:01", "content": "I checked this code on my local PC and found some randomness in results between each run (about 0.001 or lower). \nFor example:\nEpoch: 3 - Val F1 Score: 0.5564 vs Epoch: 3 - Val F1 Score: 0.5549\nEpoch: 4 - Val F1 Score: 0.5616 vs Epoch: 4 - Val F1 Score: 0.5614
How did you achieve that? Any special trick?\nAnd what is your gap between local and public score?
\n\nIf you don't mind to share for sure :)
", "rawMarkdown": "How did you achieve that? Any special trick?\nAnd what is your gap between local and public score?\n\nIf you don't mind to share for sure :)", "votes": null }, { "id": "434218", "postDate": "12/06/2018 04:05:08", "content": "Not really... I just focused on some preprocessing and then a simple model with 5 fold CV. Though I'm not sure if a 5-fold is the way to go or not, yet. I haven't really heard much about the teams using a blended approach. Are you using a single model? Your standing is quite high.
", "rawMarkdown": "Not really... I just focused on some preprocessing and then a simple model with 5 fold CV. Though I'm not sure if a 5-fold is the way to go or not, yet. I haven't really heard much about the teams using a blended approach. Are you using a single model? Your standing is quite high.", "votes": null }, { "id": "434225", "postDate": "12/06/2018 04:28:40", "content": "Confirmed that for pytorch the result seems reproducible using the following setting in both script and notebook mode (tested 3-4 times):
def set_seeds(rand_seed):\n torch.manual_seed(rand_seed)\n torch.cuda.manual_seed(rand_seed)\n torch.cuda.manual_seed_all(rand_seed)\n\n # When running on the CuDNN backend\n torch.backends.cudnn.deterministic = True\n\n np.random.seed(rand_seed)\n random.seed(rand_seed)\n\n\n
Official doc about Reproducibility:\nhttps://pytorch.org/docs/master/notes/randomness.html
", "rawMarkdown": "Confirmed that for `pytorch` the result seems reproducible using the following setting in both script and notebook mode (tested 3-4 times):\ndef set_seeds(rand_seed):\n torch.manual_seed(rand_seed)\n torch.cuda.manual_seed(rand_seed)\n torch.cuda.manual_seed_all(rand_seed)\n\n # When running on the CuDNN backend\n torch.backends.cudnn.deterministic = True\n\n np.random.seed(rand_seed)\n random.seed(rand_seed)\n\n\nOfficial doc about Reproducibility:\nhttps://pytorch.org/docs/master/notes/randomness.html", "votes": null }, { "id": "434316", "postDate": "12/06/2018 07:38:21", "content": "
Thanks, Mark! Time to port my workflow to PyTorch.
", "rawMarkdown": "Thanks, Mark! Time to port my workflow to PyTorch.", "votes": null }, { "id": "434404", "postDate": "12/06/2018 10:44:53", "content": "Thanks for the info!
\n\nYes, I have been using one model so far, without k-folds. But with some tricks for sure :) Although, I'm not able to get such a good variation with LB as you.
", "rawMarkdown": "Thanks for the info!\n\nYes, I have been using one model so far, without k-folds. But with some tricks for sure :) Although, I'm not able to get such a good variation with LB as you.", "votes": null }, { "id": "434434", "postDate": "12/06/2018 11:51:26", "content": "Great to hear Sava (@thinline72), I am using a single LSTM model in my present best with CV (0.6841) and LB (0.699 )correlation. But, I too suffer from randomness. I guess I have to make a pytorch pipeline to ensure reproducability.
", "rawMarkdown": "Great to hear Sava (@thinline72), I am using a single LSTM model in my present best with CV (0.6841) and LB (0.699 )correlation. But, I too suffer from randomness. I guess I have to make a pytorch pipeline to ensure reproducability.", "votes": null } ], "comments": [ { "id": 431423, "author_name": "suchith0312", "author_url": "", "post_date": "12/02/2018 08:47:32", "content": "https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/72040
\n\nI think this discussion mostly contains about reproducibility. I think with keras it's gonna be tough.
", "votes": null, "replies": [ { "id": 431496, "author_name": "shaz13", "author_url": "", "post_date": "12/02/2018 11:41:18", "content": "I missed that thread. Btw, are you using pytorch. How repeatable are the results in your case?
", "votes": null, "replies": [] }, { "id": 431649, "author_name": "suchith0312", "author_url": "", "post_date": "12/02/2018 17:16:18", "content": "No, I have not tried pytorch but with keras, the same kernel is always giving me different results.
", "votes": null, "replies": [] } ] }, { "id": 431834, "author_name": "joeytaj", "author_url": "", "post_date": "12/03/2018 01:36:59", "content": "I think it is good this has it's own thread. All the keras\\cudnn specific discussion has been in kernel threads.
\n\nEventually after dealing with all the places of randomness I was able to make local code run deterministically. This hopefully will be helpful for exploration. But none of this works on kaggle kernels. I also have no issue locally with Cudnn and on Kaggle kernels even super simple non RNN kernels are not deterministic.
\n\nI shared a simple kernel to show that the issue is not Cudnn and give people a fast way to test their environment for randomness.
\n\nhttps://www.kaggle.com/joeytaj/gpu-randomness-exploration
\n\nMy local environment is \nWindows 10\nGTX 1080\nTensorflow 1.12\nKeras 2.2.4\ncudnn 7.1.4\ncuda 9.0
", "votes": null, "replies": [ { "id": 432930, "author_name": "alexanderkuzmenko", "author_url": "", "post_date": "12/04/2018 13:43:01", "content": "I checked this code on my local PC and found some randomness in results between each run (about 0.001 or lower). \nFor example:\nEpoch: 3 - Val F1 Score: 0.5564 vs Epoch: 3 - Val F1 Score: 0.5549\nEpoch: 4 - Val F1 Score: 0.5616 vs Epoch: 4 - Val F1 Score: 0.5614
I'm seeing about a 0.001 variation using the same kernel now... And my score between local and public is pretty stable. Using Keras... CuDNN-variants.
", "votes": null, "replies": [ { "id": 434142, "author_name": "thinline72", "author_url": "", "post_date": "12/06/2018 00:44:25", "content": "How did you achieve that? Any special trick?\nAnd what is your gap between local and public score?
\n\nIf you don't mind to share for sure :)
", "votes": null, "replies": [] }, { "id": 434218, "author_name": "learnmower", "author_url": "", "post_date": "12/06/2018 04:05:08", "content": "Not really... I just focused on some preprocessing and then a simple model with 5 fold CV. Though I'm not sure if a 5-fold is the way to go or not, yet. I haven't really heard much about the teams using a blended approach. Are you using a single model? Your standing is quite high.
", "votes": null, "replies": [] }, { "id": 434404, "author_name": "thinline72", "author_url": "", "post_date": "12/06/2018 10:44:53", "content": "Thanks for the info!
\n\nYes, I have been using one model so far, without k-folds. But with some tricks for sure :) Although, I'm not able to get such a good variation with LB as you.
", "votes": null, "replies": [] }, { "id": 434434, "author_name": "shaz13", "author_url": "", "post_date": "12/06/2018 11:51:26", "content": "Great to hear Sava (@thinline72), I am using a single LSTM model in my present best with CV (0.6841) and LB (0.699 )correlation. But, I too suffer from randomness. I guess I have to make a pytorch pipeline to ensure reproducability.
", "votes": null, "replies": [] } ] }, { "id": 434225, "author_name": "markpeng", "author_url": "", "post_date": "12/06/2018 04:28:40", "content": "Confirmed that for pytorch the result seems reproducible using the following setting in both script and notebook mode (tested 3-4 times):
def set_seeds(rand_seed):\n torch.manual_seed(rand_seed)\n torch.cuda.manual_seed(rand_seed)\n torch.cuda.manual_seed_all(rand_seed)\n\n # When running on the CuDNN backend\n torch.backends.cudnn.deterministic = True\n\n np.random.seed(rand_seed)\n random.seed(rand_seed)\n\n\n
Official doc about Reproducibility:\nhttps://pytorch.org/docs/master/notes/randomness.html
", "votes": null, "replies": [ { "id": 434316, "author_name": "learnmower", "author_url": "", "post_date": "12/06/2018 07:38:21", "content": "Thanks, Mark! Time to port my workflow to PyTorch.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "431377": "We have all been experiencing the issue with reproducability. This thread is dedicated to dicuss various approaches teams and kagglers have been using. \n\n## What causes it? \n 1. Randomness in Initialization, such as weights. \n 2. Randomness in Regularization, such as dropout. \n 3. Randomness in Layers, such as word embedding or Optimization\n 4. Randomness introduced by any randomness from third party library \n\n## Methods to reproduce on CPU \n1. Set random seeds for numpy operations/splits `np.random.seed(2018)`\n2. Set random seed for Tensorflow operations from `tf.set_random_seed(2)`\n\n## Why Reproducibility is hard when we train on GPU\nThe backend is configured to use a sophisticated stack of GPU libraries, and that some of these may introduce their own source of randomness that you may or may not be able to account for. Alike GPU in our kernels \n\n### Dealing with GPU reproducibility \n1. Rerun the experiment 10 - 15 times and average the scores to get the progress of your model.\n\nLet me know if you guys have any other awesome thoughts on making results more reproducible for this competition. I am getting deviations from 0.692 - 0.698 . My local CV and the leaderboard scores are correlating as well.\n\n\n## Various methods from kernels/discussion so far\n\n### Pytorch \nCredit - @bkkaggle\n\n SEED = 1337\n np.random.seed(SEED)\n torch.manual_seed(SEED)\n torch.cuda.manual_seed(SEED)\n torch.backends.cudnn.deterministic = True\n\n### Tensorflow \nCredit - @shujian\n\n[A Workaround for Non-Determinism in TensorFlow][1]\n\n\n [1]: http://%20https://www.twosigma.com/insights/article/a-workaround-for-non-determinism-in-tensorflow/", "431423": "https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/72040\n\nI think this discussion mostly contains about reproducibility. I think with keras it's gonna be tough.", "431496": "I missed that thread. Btw, are you using pytorch. How repeatable are the results in your case?", "431649": "No, I have not tried pytorch but with keras, the same kernel is always giving me different results.", "431834": "I think it is good this has it's own thread. All the keras\\cudnn specific discussion has been in kernel threads.\n\nEventually after dealing with all the places of randomness I was able to make local code run deterministically. This hopefully will be helpful for exploration. But none of this works on kaggle kernels. I also have no issue locally with Cudnn and on Kaggle kernels even super simple non RNN kernels are not deterministic.\n\nI shared a simple kernel to show that the issue is not Cudnn and give people a fast way to test their environment for randomness.\n\nhttps://www.kaggle.com/joeytaj/gpu-randomness-exploration\n\n\nMy local environment is \nWindows 10\nGTX 1080\nTensorflow 1.12\nKeras 2.2.4\ncudnn 7.1.4\ncuda 9.0", "431969": "I'm seeing about a 0.001 variation using the same kernel now... And my score between local and public is pretty stable. Using Keras... CuDNN-variants.", "432930": "I checked this code on my local PC and found some randomness in results between each run (about 0.001 or lower). \nFor example:\n`Epoch: 3 - Val F1 Score: 0.5564` vs `Epoch: 3 - Val F1 Score: 0.5549`\n`Epoch: 4 - Val F1 Score: 0.5616` vs `Epoch: 4 - Val F1 Score: 0.5614`", "434142": "How did you achieve that? Any special trick?\nAnd what is your gap between local and public score?\n\nIf you don't mind to share for sure :)", "434218": "Not really... I just focused on some preprocessing and then a simple model with 5 fold CV. Though I'm not sure if a 5-fold is the way to go or not, yet. I haven't really heard much about the teams using a blended approach. Are you using a single model? Your standing is quite high.", "434225": "Confirmed that for `pytorch` the result seems reproducible using the following setting in both script and notebook mode (tested 3-4 times):\ndef set_seeds(rand_seed):\n torch.manual_seed(rand_seed)\n torch.cuda.manual_seed(rand_seed)\n torch.cuda.manual_seed_all(rand_seed)\n\n # When running on the CuDNN backend\n torch.backends.cudnn.deterministic = True\n\n np.random.seed(rand_seed)\n random.seed(rand_seed)\n\n\nOfficial doc about Reproducibility:\nhttps://pytorch.org/docs/master/notes/randomness.html", "434316": "Thanks, Mark! Time to port my workflow to PyTorch.", "434404": "Thanks for the info!\n\nYes, I have been using one model so far, without k-folds. But with some tricks for sure :) Although, I'm not able to get such a good variation with LB as you.", "434434": "Great to hear Sava (@thinline72), I am using a single LSTM model in my present best with CV (0.6841) and LB (0.699 )correlation. But, I too suffer from randomness. I guess I have to make a pytorch pipeline to ensure reproducability." }, "source": "meta" }