Hi,
\n\nUsing CuDNNLSTM we get non deterministic results using GPU. On a 5 folds CV, this creates some variance on the loss / MMC on our validation set. Also the CV is not big enough to be very reliable and stable compared to the size of the Test data. So I end up with a lot of noise between my CV / LB score.
\n\nI tried different approaches for staking to improves the predictions : hard voting, soft voting, Ridge Regression. It helps improve the local CV but the gain on LB is a lot smaller. Are you experiencing similar results ? Any suggestion on how to stack efficiently for this particular problem ?
\n\nThanks a lot,
", "messages": [ { "id": "470267", "postDate": "02/12/2019 17:04:28", "content": "Hi,
\n\nUsing CuDNNLSTM we get non deterministic results using GPU. On a 5 folds CV, this creates some variance on the loss / MMC on our validation set. Also the CV is not big enough to be very reliable and stable compared to the size of the Test data. So I end up with a lot of noise between my CV / LB score.
\n\nI tried different approaches for staking to improves the predictions : hard voting, soft voting, Ridge Regression. It helps improve the local CV but the gain on LB is a lot smaller. Are you experiencing similar results ? Any suggestion on how to stack efficiently for this particular problem ?
\n\nThanks a lot,
", "rawMarkdown": "Hi,\n\nUsing CuDNNLSTM we get non deterministic results using GPU. On a 5 folds CV, this creates some variance on the loss / MMC on our validation set. Also the CV is not big enough to be very reliable and stable compared to the size of the Test data. So I end up with a lot of noise between my CV / LB score. \n\nI tried different approaches for staking to improves the predictions : hard voting, soft voting, Ridge Regression. It helps improve the local CV but the gain on LB is a lot smaller. Are you experiencing similar results ? Any suggestion on how to stack efficiently for this particular problem ?\n\nThanks a lot,", "votes": null }, { "id": "470335", "postDate": "02/12/2019 19:27:49", "content": "You can get deterministic results with CuDNNLSTM. Assuming that you already provided other seeds to everything and you don't use sets/dicts, try to add these things: \n\ntorch.manual_seed(SEED)\ntorch.cuda.manual_seed(SEED)\ntorch.backends.cudnn.deterministic = True\n
Yes, I know that it's PyTorch and you are using Keras, but both of them are using the same CuDNN implementation. And looks liketorch.backends.cudnn.deterministic = True just sets CuDNN specific flag etc. I wasn't able to find similar thing in Keras/TF.
With this trick + seeding everything else, we were able to get fully reproducible results in Keras with CuDNN in the recent Quora competition.
", "rawMarkdown": "You can get deterministic results with CuDNNLSTM. Assuming that you already provided other seeds to everything and you don't use sets/dicts, try to add these things: \n```\ntorch.manual_seed(SEED)\ntorch.cuda.manual_seed(SEED)\ntorch.backends.cudnn.deterministic = True\n```\n\nYes, I know that it's PyTorch and you are using Keras, but both of them are using the same CuDNN implementation. And looks like`torch.backends.cudnn.deterministic = True` just sets CuDNN specific flag etc. I wasn't able to find similar thing in Keras/TF.\n\nWith this trick + seeding everything else, we were able to get fully reproducible results in Keras with CuDNN in the recent Quora competition.", "votes": null }, { "id": "470355", "postDate": "02/12/2019 20:03:16", "content": "My Single best model has LB 0.724, With voting i was able to get 0.748 on LB.
", "rawMarkdown": "My Single best model has LB 0.724, With voting i was able to get 0.748 on LB.", "votes": null }, { "id": "470384", "postDate": "02/12/2019 21:06:13", "content": "Amazing !! Thanks a lot !
", "rawMarkdown": "Amazing !! Thanks a lot !", "votes": null }, { "id": "470385", "postDate": "02/12/2019 21:09:19", "content": "Thanks for sharing this ! 0.02 is similar to what I had with voting.
", "rawMarkdown": "Thanks for sharing this ! 0.02 is similar to what I had with voting.", "votes": null }, { "id": "470410", "postDate": "02/12/2019 22:42:20", "content": "Voting also, +0.01 for me.
", "rawMarkdown": "Voting also, +0.01 for me.", "votes": null }, { "id": "470977", "postDate": "02/13/2019 21:02:13", "content": "Actually in my case it works for a few epochs (around 20 epochs) and then it diverges. After 50 epochs the difference is quite big.
\n\nI'm using a constant seed for my kfold split + the followings :
\n\n<code>\n np.random.seed(seed)\n rn.seed(seed)\n os.environ['PYTHONHASHSEED']=str(seed)\n tf.set_random_seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n</code>\nIs it working in your case with this data set ?
", "rawMarkdown": "Actually in my case it works for a few epochs (around 20 epochs) and then it diverges. After 50 epochs the difference is quite big. \n\nI'm using a constant seed for my kfold split + the followings :\n\n\n np.random.seed(seed)\n rn.seed(seed)\n os.environ['PYTHONHASHSEED']=str(seed)\n tf.set_random_seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n \n\nIs it working in your case with this data set ?",
"votes": null
},
{
"id": "470982",
"postDate": "02/13/2019 21:21:45",
"content": "Setting a seed for pytorch won't work with Keras/Tensorflow even if both run with GPU.
\n\nFor pytorch you can do something like below. BiLSTM results are 100% reproductible with Pytorch (but you need to port Keras model to Pytorch).
\n\nimport torch\n\ndef seed_numpy_and_pytorch(s):\n random.seed(s)\n os.environ['PYTHONHASHSEED'] = str(s)\n np.random.seed(s)\n torch.manual_seed(s)\n torch.cuda.manual_seed(s)\n torch.backends.cudnn.deterministic = True\n if torch.cuda.is_available():\n torch.cuda.manual_seed_all(seed)\n\nseed_numpy_and_pytorch(1234)\n@mpware It will, at least torch.backends.cudnn.deterministic = True. It was tested by several teams in Quora competition. CuDNN is the same for Keras/TF and PyTorch.
@areveillon make sure that you passes seeds to all initializers and data generators.
", "rawMarkdown": "mpware It will, at least `torch.backends.cudnn.deterministic = True`. It was tested by several teams in Quora competition. CuDNN is the same for Keras/TF and PyTorch.\n\n@areveillon make sure that you passes seeds to all initializers and data generators.", "votes": null }, { "id": "471267", "postDate": "02/14/2019 08:27:40", "content": "Interesting, do you know if there is combination of Pytorch, Keras, Tensorflow to make work? I tested it and got similar results as Antoine with:
\n\n@Antonie, PYTHONHASHSEED should be set before you start python interpreter, from within the script it does nothing.
An interesting comment from the Quora comp winner:
\n\nEmbrace the randomness\nAs it was necessary to utilize CUDNN Layers in this competition, there was some randomness involved that could be quite frustrating from time to time. I saw many people trying to fix seeds etc. and some claiming they could completely remove the randomness by using Pytorch (I still don’t believe this BTW as CUDNN has atomic operations). However, as mentioned before, a well working strategy in this competition was to combine multiple models and to end up with a good ensemble, those models should be a bit different to each other. So having different random initializations etc. can be helpful. Seeing people setting the seed as a hyperparameter is weird.", "rawMarkdown": "An interesting comment from the Quora comp winner: \n\n
Embrace the randomness\nAs it was necessary to utilize CUDNN Layers in this competition, there was some randomness involved that could be quite frustrating from time to time. I saw many people trying to fix seeds etc. and some claiming they could completely remove the randomness by using Pytorch (I still don’t believe this BTW as CUDNN has atomic operations). However, as mentioned before, a well working strategy in this competition was to combine multiple models and to end up with a good ensemble, those models should be a bit different to each other. So having different random initializations etc. can be helpful. Seeing people setting the seed as a hyperparameter is weird.", "votes": null }, { "id": "472343", "postDate": "02/15/2019 18:35:55", "content": "
Thanks Marcin !
", "rawMarkdown": "Thanks Marcin !", "votes": null } ], "comments": [ { "id": 470335, "author_name": "thinline72", "author_url": "", "post_date": "02/12/2019 19:27:49", "content": "You can get deterministic results with CuDNNLSTM. Assuming that you already provided other seeds to everything and you don't use sets/dicts, try to add these things: \n\ntorch.manual_seed(SEED)\ntorch.cuda.manual_seed(SEED)\ntorch.backends.cudnn.deterministic = True\n
Yes, I know that it's PyTorch and you are using Keras, but both of them are using the same CuDNN implementation. And looks liketorch.backends.cudnn.deterministic = True just sets CuDNN specific flag etc. I wasn't able to find similar thing in Keras/TF.
With this trick + seeding everything else, we were able to get fully reproducible results in Keras with CuDNN in the recent Quora competition.
", "votes": null, "replies": [ { "id": 470384, "author_name": "areveillon", "author_url": "", "post_date": "02/12/2019 21:06:13", "content": "Amazing !! Thanks a lot !
", "votes": null, "replies": [] }, { "id": 470977, "author_name": "areveillon", "author_url": "", "post_date": "02/13/2019 21:02:13", "content": "Actually in my case it works for a few epochs (around 20 epochs) and then it diverges. After 50 epochs the difference is quite big.
\n\nI'm using a constant seed for my kfold split + the followings :
\n\n<code>\n np.random.seed(seed)\n rn.seed(seed)\n os.environ['PYTHONHASHSEED']=str(seed)\n tf.set_random_seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n</code>\nIs it working in your case with this data set ?
", "votes": null, "replies": [] }, { "id": 470982, "author_name": "mpware", "author_url": "", "post_date": "02/13/2019 21:21:45", "content": "Setting a seed for pytorch won't work with Keras/Tensorflow even if both run with GPU.
\n\nFor pytorch you can do something like below. BiLSTM results are 100% reproductible with Pytorch (but you need to port Keras model to Pytorch).
\n\nimport torch\n\ndef seed_numpy_and_pytorch(s):\n random.seed(s)\n os.environ['PYTHONHASHSEED'] = str(s)\n np.random.seed(s)\n torch.manual_seed(s)\n torch.cuda.manual_seed(s)\n torch.backends.cudnn.deterministic = True\n if torch.cuda.is_available():\n torch.cuda.manual_seed_all(seed)\n\nseed_numpy_and_pytorch(1234)\n@mpware It will, at least torch.backends.cudnn.deterministic = True. It was tested by several teams in Quora competition. CuDNN is the same for Keras/TF and PyTorch.
@areveillon make sure that you passes seeds to all initializers and data generators.
", "votes": null, "replies": [] }, { "id": 471267, "author_name": "mpware", "author_url": "", "post_date": "02/14/2019 08:27:40", "content": "Interesting, do you know if there is combination of Pytorch, Keras, Tensorflow to make work? I tested it and got similar results as Antoine with:
\n\nAn interesting comment from the Quora comp winner:
\n\nEmbrace the randomness\nAs it was necessary to utilize CUDNN Layers in this competition, there was some randomness involved that could be quite frustrating from time to time. I saw many people trying to fix seeds etc. and some claiming they could completely remove the randomness by using Pytorch (I still don’t believe this BTW as CUDNN has atomic operations). However, as mentioned before, a well working strategy in this competition was to combine multiple models and to end up with a good ensemble, those models should be a bit different to each other. So having different random initializations etc. can be helpful. Seeing people setting the seed as a hyperparameter is weird.", "votes": null, "replies": [] } ] }, { "id": 470355, "author_name": "harshit92", "author_url": "", "post_date": "02/12/2019 20:03:16", "content": "
My Single best model has LB 0.724, With voting i was able to get 0.748 on LB.
", "votes": null, "replies": [ { "id": 470385, "author_name": "areveillon", "author_url": "", "post_date": "02/12/2019 21:09:19", "content": "Thanks for sharing this ! 0.02 is similar to what I had with voting.
", "votes": null, "replies": [] }, { "id": 470410, "author_name": "mpware", "author_url": "", "post_date": "02/12/2019 22:42:20", "content": "Voting also, +0.01 for me.
", "votes": null, "replies": [] } ] }, { "id": 471733, "author_name": "mpekalski", "author_url": "", "post_date": "02/14/2019 20:25:58", "content": "@Antonie, PYTHONHASHSEED should be set before you start python interpreter, from within the script it does nothing.
Thanks Marcin !
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "470267": "Hi,\n\nUsing CuDNNLSTM we get non deterministic results using GPU. On a 5 folds CV, this creates some variance on the loss / MMC on our validation set. Also the CV is not big enough to be very reliable and stable compared to the size of the Test data. So I end up with a lot of noise between my CV / LB score. \n\nI tried different approaches for staking to improves the predictions : hard voting, soft voting, Ridge Regression. It helps improve the local CV but the gain on LB is a lot smaller. Are you experiencing similar results ? Any suggestion on how to stack efficiently for this particular problem ?\n\nThanks a lot,", "470335": "You can get deterministic results with CuDNNLSTM. Assuming that you already provided other seeds to everything and you don't use sets/dicts, try to add these things: \n```\ntorch.manual_seed(SEED)\ntorch.cuda.manual_seed(SEED)\ntorch.backends.cudnn.deterministic = True\n```\n\nYes, I know that it's PyTorch and you are using Keras, but both of them are using the same CuDNN implementation. And looks like`torch.backends.cudnn.deterministic = True` just sets CuDNN specific flag etc. I wasn't able to find similar thing in Keras/TF.\n\nWith this trick + seeding everything else, we were able to get fully reproducible results in Keras with CuDNN in the recent Quora competition.", "470355": "My Single best model has LB 0.724, With voting i was able to get 0.748 on LB.", "470384": "Amazing !! Thanks a lot !", "470385": "Thanks for sharing this ! 0.02 is similar to what I had with voting.", "470410": "Voting also, +0.01 for me.", "470977": "Actually in my case it works for a few epochs (around 20 epochs) and then it diverges. After 50 epochs the difference is quite big. \n\nI'm using a constant seed for my kfold split + the followings :\n\n\n np.random.seed(seed)\n rn.seed(seed)\n os.environ['PYTHONHASHSEED']=str(seed)\n tf.set_random_seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n \n\nIs it working in your case with this data set ?",
"470982": "Setting a seed for pytorch won't work with Keras/Tensorflow even if both run with GPU.\n\nFor pytorch you can do something like below. BiLSTM results are 100% reproductible with Pytorch (but you need to port Keras model to Pytorch).\n\n\n import torch\n \n def seed_numpy_and_pytorch(s):\n random.seed(s)\n os.environ['PYTHONHASHSEED'] = str(s)\n np.random.seed(s)\n torch.manual_seed(s)\n torch.cuda.manual_seed(s)\n torch.backends.cudnn.deterministic = True\n if torch.cuda.is_available():\n torch.cuda.manual_seed_all(seed)\n \n seed_numpy_and_pytorch(1234)",
"471029": "mpware It will, at least `torch.backends.cudnn.deterministic = True`. It was tested by several teams in Quora competition. CuDNN is the same for Keras/TF and PyTorch.\n\n@areveillon make sure that you passes seeds to all initializers and data generators.",
"471267": "Interesting, do you know if there is combination of Pytorch, Keras, Tensorflow to make work? I tested it and got similar results as Antoine with:\n\n- Keras 2.2.4\n- TensorFlow 1.13.0-rc1\n- Pytorch 1.0",
"471733": "Antonie, `PYTHONHASHSEED` should be set before you start python interpreter, from within the script it does nothing.",
"472342": "An interesting comment from the Quora comp winner: \n\nEmbrace the randomness\nAs it was necessary to utilize CUDNN Layers in this competition, there was some randomness involved that could be quite frustrating from time to time. I saw many people trying to fix seeds etc. and some claiming they could completely remove the randomness by using Pytorch (I still don’t believe this BTW as CUDNN has atomic operations). However, as mentioned before, a well working strategy in this competition was to combine multiple models and to end up with a good ensemble, those models should be a bit different to each other. So having different random initializations etc. can be helpful. Seeing people setting the seed as a hyperparameter is weird.", "472343": "Thanks Marcin !" }, "source": "meta" }