In case anyone else hits this issue ... I've found that even with setting random seeds, I was seeing different results on each run of my python script whenever I used a Convolutional layer in Keras.
\n\nAfter some digging, I found this, which seemed to solve the issue:
\n\n\n\nhttp://deeplearning.net/software/theano/library/sandbox/cuda/dnn.html
\n\nYou can use it from the .theanorc file, or via environment variable (Mac or linux) as follows:
\n\nexport THEANO_FLAGS=device=gpu,floatX=float32,dnn.conv.algo_bwd_filter=deterministic,dnn.conv.algo_bwd_data=deterministic;
", "messages": [ { "id": "120931", "postDate": "05/21/2016 18:49:29", "content": "In case anyone else hits this issue ... I've found that even with setting random seeds, I was seeing different results on each run of my python script whenever I used a Convolutional layer in Keras.
\n\nAfter some digging, I found this, which seemed to solve the issue:
\n\n\n\nhttp://deeplearning.net/software/theano/library/sandbox/cuda/dnn.html
\n\nYou can use it from the .theanorc file, or via environment variable (Mac or linux) as follows:
\n\nexport THEANO_FLAGS=device=gpu,floatX=float32,dnn.conv.algo_bwd_filter=deterministic,dnn.conv.algo_bwd_data=deterministic;
", "rawMarkdown": "In case anyone else hits this issue ... I've found that even with setting random seeds, I was seeing different results on each run of my python script whenever I used a Convolutional layer in Keras.\r\n\r\nAfter some digging, I found this, which seemed to solve the issue:\r\n\r\nhttps://www.bountysource.com/issues/31151094-do-we-have-a-way-to-initialize-the-random-seed-for-everything\r\n\r\nhttp://deeplearning.net/software/theano/library/sandbox/cuda/dnn.html\r\n\r\nYou can use it from the .theanorc file, or via environment variable (Mac or linux) as follows:\r\n\r\nexport THEANO_FLAGS=device=gpu,floatX=float32,dnn.conv.algo_bwd_filter=deterministic,dnn.conv.algo_bwd_data=deterministic;", "votes": null }, { "id": "123970", "postDate": "06/14/2016 18:16:39", "content": "Even with deterministic algorithm, I have problem with multi-gpus. I use cross-validation and each fold runs in different gpu. The problem is that when I run only one fold, I have deterministic result. But when they run in different gpus, I have non-deterministic result.\nAnybody has any solution or idea?
", "rawMarkdown": "Even with deterministic algorithm, I have problem with multi-gpus. I use cross-validation and each fold runs in different gpu. The problem is that when I run only one fold, I have deterministic result. But when they run in different gpus, I have non-deterministic result.\r\nAnybody has any solution or idea?", "votes": null }, { "id": "124032", "postDate": "06/15/2016 04:05:22", "content": "I'd check if the random seed is set exactly the same way every time. You have your code on your main thread that uses it's random seed. Then you have separate threads that run the folds on the different GPUs. Now you want to ensure that those threads get initialized with the same seed regardless of the order in which the jobs are launched.
\n\nAlso I'd check if the GPU's RNG has a separate seed. Moreover, every thread (or GPU) should have it's separate RNG. You don't want to have just one RNG and have the multiple threads asynchronously get a random number from it.
", "rawMarkdown": "I'd check if the random seed is set exactly the same way every time. You have your code on your main thread that uses it's random seed. Then you have separate threads that run the folds on the different GPUs. Now you want to ensure that those threads get initialized with the same seed regardless of the order in which the jobs are launched. \r\n\r\nAlso I'd check if the GPU's RNG has a separate seed. Moreover, every thread (or GPU) should have it's separate RNG. You don't want to have just one RNG and have the multiple threads asynchronously get a random number from it.", "votes": null } ], "comments": [ { "id": 123970, "author_name": "ehsanma", "author_url": "", "post_date": "06/14/2016 18:16:39", "content": "Even with deterministic algorithm, I have problem with multi-gpus. I use cross-validation and each fold runs in different gpu. The problem is that when I run only one fold, I have deterministic result. But when they run in different gpus, I have non-deterministic result.\nAnybody has any solution or idea?
", "votes": null, "replies": [] }, { "id": 124032, "author_name": "ilucian", "author_url": "", "post_date": "06/15/2016 04:05:22", "content": "I'd check if the random seed is set exactly the same way every time. You have your code on your main thread that uses it's random seed. Then you have separate threads that run the folds on the different GPUs. Now you want to ensure that those threads get initialized with the same seed regardless of the order in which the jobs are launched.
\n\nAlso I'd check if the GPU's RNG has a separate seed. Moreover, every thread (or GPU) should have it's separate RNG. You don't want to have just one RNG and have the multiple threads asynchronously get a random number from it.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "120931": "In case anyone else hits this issue ... I've found that even with setting random seeds, I was seeing different results on each run of my python script whenever I used a Convolutional layer in Keras.\r\n\r\nAfter some digging, I found this, which seemed to solve the issue:\r\n\r\nhttps://www.bountysource.com/issues/31151094-do-we-have-a-way-to-initialize-the-random-seed-for-everything\r\n\r\nhttp://deeplearning.net/software/theano/library/sandbox/cuda/dnn.html\r\n\r\nYou can use it from the .theanorc file, or via environment variable (Mac or linux) as follows:\r\n\r\nexport THEANO_FLAGS=device=gpu,floatX=float32,dnn.conv.algo_bwd_filter=deterministic,dnn.conv.algo_bwd_data=deterministic;", "123970": "Even with deterministic algorithm, I have problem with multi-gpus. I use cross-validation and each fold runs in different gpu. The problem is that when I run only one fold, I have deterministic result. But when they run in different gpus, I have non-deterministic result.\r\nAnybody has any solution or idea?", "124032": "I'd check if the random seed is set exactly the same way every time. You have your code on your main thread that uses it's random seed. Then you have separate threads that run the folds on the different GPUs. Now you want to ensure that those threads get initialized with the same seed regardless of the order in which the jobs are launched. \r\n\r\nAlso I'd check if the GPU's RNG has a separate seed. Moreover, every thread (or GPU) should have it's separate RNG. You don't want to have just one RNG and have the multiple threads asynchronously get a random number from it." }, "source": "meta" }