I just notice that for cudnn, its backward computation for convolutional layers is stochastic and not reproducible. Cudnn has been used in winning solutions in the past, so it should be OK to use it in this competition?
", "messages": [ { "id": "106860", "postDate": "02/04/2016 14:06:33", "content": "I just notice that for cudnn, its backward computation for convolutional layers is stochastic and not reproducible. Cudnn has been used in winning solutions in the past, so it should be OK to use it in this competition?
", "rawMarkdown": "I just notice that for cudnn, its backward computation for convolutional layers is stochastic and not reproducible. Cudnn has been used in winning solutions in the past, so it should be OK to use it in this competition?", "votes": null }, { "id": "106879", "postDate": "02/04/2016 16:53:06", "content": "I noticed that. CUDNN LIBRARY User Guide / 2.5. Reproducibility (determinism) is useful for this topic.
", "rawMarkdown": "I noticed that. *CUDNN LIBRARY User Guide / 2.5. Reproducibility (determinism)* is useful for this topic.", "votes": null }, { "id": "107074", "postDate": "02/06/2016 13:10:00", "content": "Hi. I work for NVIDIA, so I am checking that this is the case up through the most recent cuDNN versions and will post back here.
\n\nIndependently of cuDNN the issue also applies if you are using dropout for regularization or a non-deterministic method for initializing weights/biases - so we need competition admin clarification on those issues.
", "rawMarkdown": "Hi. I work for NVIDIA, so I am checking that this is the case up through the most recent cuDNN versions and will post back here.\r\n\r\nIndependently of cuDNN the issue also applies if you are using dropout for regularization or a non-deterministic method for initializing weights/biases - so we need competition admin clarification on those issues.", "votes": null }, { "id": "107075", "postDate": "02/06/2016 13:39:45", "content": "We will be able to reproduce dropout or initial weights using fixed RNG seed. But cuDNN's non-deterministic method depends on the hardware and runtime conditions. It is not reproducible.
", "rawMarkdown": "We will be able to reproduce dropout or initial weights using fixed RNG seed. But cuDNN's non-deterministic method depends on the hardware and runtime conditions. It is not reproducible.", "votes": null }, { "id": "107234", "postDate": "02/08/2016 17:11:19", "content": "Having just re-read the rules it says "In order to be eligible for prizes, you are required to upload your model prior to the release of the test dataset. This model will be used to verify that the test set was scored in a completely automated manner." I interpret this as meaning that you need to upload a pre-trained model that can be used to perform inference on the test set by the competition admins. Inference using cuDNN is deterministic so there should be no problem, the non-deterministic methods are only used on the backward pass during model training.
", "rawMarkdown": "Having just re-read the rules it says \"In order to be eligible for prizes, you are required to upload your model prior to the release of the test dataset. This model will be used to verify that the test set was scored in a completely automated manner.\" I interpret this as meaning that you need to upload a pre-trained model that can be used to perform inference on the test set by the competition admins. Inference using cuDNN is deterministic so there should be no problem, the non-deterministic methods are only used on the backward pass during model training.", "votes": null }, { "id": "107524", "postDate": "02/10/2016 17:47:07", "content": "Is model loaded in json format? Here is my concern:\nI have used both mxnet and keras examples posted here. Many Thanks for them, they are extremely helpful for learning and get going in this contest. But it seems loading models using json is failing in both. can't able to load mxnet model once I save it in json format and posted in mxnet github in a similar open ticket. For Keras it is more flexible since you can load model and weights separately (maybe there is a way for mxnet however I couldn't figure out). Attempt to load a saved json model for Keras also fails however, I could load model from directly original code and weighs separately from hdf5. So I am not sure how Admin could load these models for inspection? You could save models in json but it seems loading back is failing for these complex models.
", "rawMarkdown": "Is model loaded in json format? Here is my concern:\r\nI have used both mxnet and keras examples posted here. Many Thanks for them, they are extremely helpful for learning and get going in this contest. But it seems loading models using json is failing in both. can't able to load mxnet model once I save it in json format and posted in mxnet github in a similar open ticket. For Keras it is more flexible since you can load model and weights separately (maybe there is a way for mxnet however I couldn't figure out). Attempt to load a saved json model for Keras also fails however, I could load model from directly original code and weighs separately from hdf5. So I am not sure how Admin could load these models for inspection? You could save models in json but it seems loading back is failing for these complex models.", "votes": null }, { "id": "109720", "postDate": "02/29/2016 21:19:40", "content": "I'm also having issues with reproducible results.
\n\nI'm using dev versions of Theano and Keras, and have Cuda 7 and Cudnn 4 at the back.
\n\nI've tried numerous configs enabling / disabling stuff, and it seems Cudnn is the culprit.
\n\n[quote=Senecaur;107234]
\n\nI interpret this as meaning that you need to upload a pre-trained model that can be used to perform inference on the test set by the competition admins. Inference using cuDNN is deterministic so there should be no problem, the non-deterministic methods are only used on the backward pass during model training.
\n\n[/quote]
\n\nAs William pointed out here, I think we're rather expected to upload code as models, not pre-trained serialized objects. So I think this might be problematic for anyone who's using Cudnn.
\n\nIs there some way to force Cudnn to work in a deterministic fashion?
\n\nOr should we drop Cudnn from our stack?
", "rawMarkdown": "I'm also having issues with reproducible results.\r\n\r\nI'm using dev versions of Theano and Keras, and have Cuda 7 and Cudnn 4 at the back.\r\n\r\nI've tried numerous configs enabling / disabling stuff, and it seems Cudnn is the culprit.\r\n\r\n[quote=Senecaur;107234]\r\n\r\nI interpret this as meaning that you need to upload a pre-trained model that can be used to perform inference on the test set by the competition admins. Inference using cuDNN is deterministic so there should be no problem, the non-deterministic methods are only used on the backward pass during model training. \r\n\r\n[/quote]\r\n\r\nAs William pointed out [here][1], I think we're rather expected to upload code as models, not pre-trained serialized objects. So I think this might be problematic for anyone who's using Cudnn.\r\n\r\nIs there some way to force Cudnn to work in a deterministic fashion?\r\n\r\nOr should we drop Cudnn from our stack?\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n [1]: https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/19148/1st-deadline-model-upload/109368#post109368", "votes": null }, { "id": "109832", "postDate": "03/01/2016 04:54:56", "content": "From my understanding, cuDNN v4's max-pooling provides only non-deterministic method. I suppose we cannot fix this issue with cuDNN v4.
\n\nI am going to remove cuDNN from my last submission.
", "rawMarkdown": "From my understanding, cuDNN v4's max-pooling provides only non-deterministic method. I suppose we cannot fix this issue with cuDNN v4.\r\n\r\nI am going to remove cuDNN from my last submission.", "votes": null } ], "comments": [ { "id": 106879, "author_name": "nagadomi", "author_url": "", "post_date": "02/04/2016 16:53:06", "content": "I noticed that. CUDNN LIBRARY User Guide / 2.5. Reproducibility (determinism) is useful for this topic.
", "votes": null, "replies": [] }, { "id": 107074, "author_name": "senecaur", "author_url": "", "post_date": "02/06/2016 13:10:00", "content": "Hi. I work for NVIDIA, so I am checking that this is the case up through the most recent cuDNN versions and will post back here.
\n\nIndependently of cuDNN the issue also applies if you are using dropout for regularization or a non-deterministic method for initializing weights/biases - so we need competition admin clarification on those issues.
", "votes": null, "replies": [] }, { "id": 107075, "author_name": "nagadomi", "author_url": "", "post_date": "02/06/2016 13:39:45", "content": "We will be able to reproduce dropout or initial weights using fixed RNG seed. But cuDNN's non-deterministic method depends on the hardware and runtime conditions. It is not reproducible.
", "votes": null, "replies": [] }, { "id": 107234, "author_name": "senecaur", "author_url": "", "post_date": "02/08/2016 17:11:19", "content": "Having just re-read the rules it says "In order to be eligible for prizes, you are required to upload your model prior to the release of the test dataset. This model will be used to verify that the test set was scored in a completely automated manner." I interpret this as meaning that you need to upload a pre-trained model that can be used to perform inference on the test set by the competition admins. Inference using cuDNN is deterministic so there should be no problem, the non-deterministic methods are only used on the backward pass during model training.
", "votes": null, "replies": [] }, { "id": 107524, "author_name": "esorar", "author_url": "", "post_date": "02/10/2016 17:47:07", "content": "Is model loaded in json format? Here is my concern:\nI have used both mxnet and keras examples posted here. Many Thanks for them, they are extremely helpful for learning and get going in this contest. But it seems loading models using json is failing in both. can't able to load mxnet model once I save it in json format and posted in mxnet github in a similar open ticket. For Keras it is more flexible since you can load model and weights separately (maybe there is a way for mxnet however I couldn't figure out). Attempt to load a saved json model for Keras also fails however, I could load model from directly original code and weighs separately from hdf5. So I am not sure how Admin could load these models for inspection? You could save models in json but it seems loading back is failing for these complex models.
", "votes": null, "replies": [] }, { "id": 109720, "author_name": "barisumog", "author_url": "", "post_date": "02/29/2016 21:19:40", "content": "I'm also having issues with reproducible results.
\n\nI'm using dev versions of Theano and Keras, and have Cuda 7 and Cudnn 4 at the back.
\n\nI've tried numerous configs enabling / disabling stuff, and it seems Cudnn is the culprit.
\n\n[quote=Senecaur;107234]
\n\nI interpret this as meaning that you need to upload a pre-trained model that can be used to perform inference on the test set by the competition admins. Inference using cuDNN is deterministic so there should be no problem, the non-deterministic methods are only used on the backward pass during model training.
\n\n[/quote]
\n\nAs William pointed out here, I think we're rather expected to upload code as models, not pre-trained serialized objects. So I think this might be problematic for anyone who's using Cudnn.
\n\nIs there some way to force Cudnn to work in a deterministic fashion?
\n\nOr should we drop Cudnn from our stack?
", "votes": null, "replies": [] }, { "id": 109832, "author_name": "nagadomi", "author_url": "", "post_date": "03/01/2016 04:54:56", "content": "From my understanding, cuDNN v4's max-pooling provides only non-deterministic method. I suppose we cannot fix this issue with cuDNN v4.
\n\nI am going to remove cuDNN from my last submission.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "106860": "I just notice that for cudnn, its backward computation for convolutional layers is stochastic and not reproducible. Cudnn has been used in winning solutions in the past, so it should be OK to use it in this competition?", "106879": "I noticed that. *CUDNN LIBRARY User Guide / 2.5. Reproducibility (determinism)* is useful for this topic.", "107074": "Hi. I work for NVIDIA, so I am checking that this is the case up through the most recent cuDNN versions and will post back here.\r\n\r\nIndependently of cuDNN the issue also applies if you are using dropout for regularization or a non-deterministic method for initializing weights/biases - so we need competition admin clarification on those issues.", "107075": "We will be able to reproduce dropout or initial weights using fixed RNG seed. But cuDNN's non-deterministic method depends on the hardware and runtime conditions. It is not reproducible.", "107234": "Having just re-read the rules it says \"In order to be eligible for prizes, you are required to upload your model prior to the release of the test dataset. This model will be used to verify that the test set was scored in a completely automated manner.\" I interpret this as meaning that you need to upload a pre-trained model that can be used to perform inference on the test set by the competition admins. Inference using cuDNN is deterministic so there should be no problem, the non-deterministic methods are only used on the backward pass during model training.", "107524": "Is model loaded in json format? Here is my concern:\r\nI have used both mxnet and keras examples posted here. Many Thanks for them, they are extremely helpful for learning and get going in this contest. But it seems loading models using json is failing in both. can't able to load mxnet model once I save it in json format and posted in mxnet github in a similar open ticket. For Keras it is more flexible since you can load model and weights separately (maybe there is a way for mxnet however I couldn't figure out). Attempt to load a saved json model for Keras also fails however, I could load model from directly original code and weighs separately from hdf5. So I am not sure how Admin could load these models for inspection? You could save models in json but it seems loading back is failing for these complex models.", "109720": "I'm also having issues with reproducible results.\r\n\r\nI'm using dev versions of Theano and Keras, and have Cuda 7 and Cudnn 4 at the back.\r\n\r\nI've tried numerous configs enabling / disabling stuff, and it seems Cudnn is the culprit.\r\n\r\n[quote=Senecaur;107234]\r\n\r\nI interpret this as meaning that you need to upload a pre-trained model that can be used to perform inference on the test set by the competition admins. Inference using cuDNN is deterministic so there should be no problem, the non-deterministic methods are only used on the backward pass during model training. \r\n\r\n[/quote]\r\n\r\nAs William pointed out [here][1], I think we're rather expected to upload code as models, not pre-trained serialized objects. So I think this might be problematic for anyone who's using Cudnn.\r\n\r\nIs there some way to force Cudnn to work in a deterministic fashion?\r\n\r\nOr should we drop Cudnn from our stack?\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n [1]: https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/19148/1st-deadline-model-upload/109368#post109368", "109832": "From my understanding, cuDNN v4's max-pooling provides only non-deterministic method. I suppose we cannot fix this issue with cuDNN v4.\r\n\r\nI am going to remove cuDNN from my last submission." }, "source": "meta" }