{ "id": 201552, "title": "OCD on Seeding: Deviations of results in different environments after Seeding.", "url": "/competitions/cassava-leaf-disease-classification/discussion/201552", "author_name": "", "post_date": "2020-12-05T14:53:37.344591800Z", "votes": 3, "comment_count": 1, "views": 0, "content": "

EDIT: Possible answer here

\n

Following this discussion I met my own problems. As I have quite a bit of OCD, I would like to seek clarification on the deviations of results I get here.

\n

Problem: In my inference notebook here, my predictions for the test image is

\n
    array([[0.00694168, 0.02380404, 0.19128187, 0.00819152, 0.7697809 ]],\n          dtype=float32)\n
\n
\n

This result stays deterministic every time I run the notebook in Kaggle. Good!

\n
\n

Now I like to document my codes on Jupyter Notebooks for book-keeping purposes, but I got a rude shock to see that the result I get this time is

\n
    array([[0.0066291 , 0.02009881, 0.21341121, 0.00788152, 0.75197935]],\n          dtype=float32)\n
\n
\n

This result stays deterministic every time I run the notebook in Jupyter Notebook. Good!

\n
\n

I then went to Google Colab and run the inference and nearly fainted as the result is as follows

\n
    array([[0.00662911, 0.02009881, 0.2134114 , 0.00788153, 0.7519792 ]],\n          dtype=float32)\n
\n
\n

This result stays deterministic every time I run the notebook in Google Colab. Good!

\n
\n

All I can say is that under these three different environments, every re-run will yield the same results - which is great since seed_all seems to work. But it still bothers me on why they give slightly different results when used in different environments. Any possible hypothesis to this as I feel my seed_all function is quite robust again randomness… My only guess is maybe on different GPU types there might be some small differences?

\n

PS: I also noticed that changing torch and torchvision versions do not affect the results.

\n

My seeding function is:

\n
def seed_all(seed: int):\n    if not seed:\n        seed = 10\n\n    print(\"[ Using Seed : \", seed, \" ]\")\n\n    os.environ['PYTHONHASHSEED'] = str(seed)  # set PYTHONHASHSEED env var at fixed value\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\n    torch.cuda.manual_seed(seed) # pytorch (both CPU and CUDA)\n    np.random.seed(seed) # for numpy pseudo-random generator\n    random.seed(seed) # set fixed value for python built-in pseudo-random generator\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = False\n    torch.backends.cudnn.enabled = False\n\ndef seed_worker(worker_id):\n    worker_seed = torch.initial_seed() % 2**32\n    np.random.seed(worker_seed)\n    random.seed(worker_seed)\n
\n

Yes, I even seed the workers.

", "messages": [ { "id": "1103000", "postDate": "12/05/2020 14:53:37", "content": "

EDIT: Possible answer here

\n

Following this discussion I met my own problems. As I have quite a bit of OCD, I would like to seek clarification on the deviations of results I get here.

\n

Problem: In my inference notebook here, my predictions for the test image is

\n
    array([[0.00694168, 0.02380404, 0.19128187, 0.00819152, 0.7697809 ]],\n          dtype=float32)\n
\n
\n

This result stays deterministic every time I run the notebook in Kaggle. Good!

\n
\n

Now I like to document my codes on Jupyter Notebooks for book-keeping purposes, but I got a rude shock to see that the result I get this time is

\n
    array([[0.0066291 , 0.02009881, 0.21341121, 0.00788152, 0.75197935]],\n          dtype=float32)\n
\n
\n

This result stays deterministic every time I run the notebook in Jupyter Notebook. Good!

\n
\n

I then went to Google Colab and run the inference and nearly fainted as the result is as follows

\n
    array([[0.00662911, 0.02009881, 0.2134114 , 0.00788153, 0.7519792 ]],\n          dtype=float32)\n
\n
\n

This result stays deterministic every time I run the notebook in Google Colab. Good!

\n
\n

All I can say is that under these three different environments, every re-run will yield the same results - which is great since seed_all seems to work. But it still bothers me on why they give slightly different results when used in different environments. Any possible hypothesis to this as I feel my seed_all function is quite robust again randomness… My only guess is maybe on different GPU types there might be some small differences?

\n

PS: I also noticed that changing torch and torchvision versions do not affect the results.

\n

My seeding function is:

\n
def seed_all(seed: int):\n    if not seed:\n        seed = 10\n\n    print(\"[ Using Seed : \", seed, \" ]\")\n\n    os.environ['PYTHONHASHSEED'] = str(seed)  # set PYTHONHASHSEED env var at fixed value\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\n    torch.cuda.manual_seed(seed) # pytorch (both CPU and CUDA)\n    np.random.seed(seed) # for numpy pseudo-random generator\n    random.seed(seed) # set fixed value for python built-in pseudo-random generator\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = False\n    torch.backends.cudnn.enabled = False\n\ndef seed_worker(worker_id):\n    worker_seed = torch.initial_seed() % 2**32\n    np.random.seed(worker_seed)\n    random.seed(worker_seed)\n
\n

Yes, I even seed the workers.

", "rawMarkdown": "EDIT: Possible answer [here](https://discuss.pytorch.org/t/reproducibility-over-different-machines/63047)\n\nFollowing [this discussion](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200182) I met my own problems. As I have quite a bit of OCD, I would like to seek clarification on the deviations of results I get here.\n\nProblem: In my [inference notebook here](https://www.kaggle.com/reighns/pytorch-a-complete-inference-guide), my predictions for the test image is \n\n```\n array([[0.00694168, 0.02380404, 0.19128187, 0.00819152, 0.7697809 ]],\n dtype=float32)\n```\n> This result stays deterministic every time I run the notebook in Kaggle. Good!\n\nNow I like to document my codes on Jupyter Notebooks for book-keeping purposes, but I got a rude shock to see that the result I get this time is \n\n```\n array([[0.0066291 , 0.02009881, 0.21341121, 0.00788152, 0.75197935]],\n dtype=float32)\n```\n\n> This result stays deterministic every time I run the notebook in Jupyter Notebook. Good!\n\nI then went to Google Colab and run the inference and nearly ***fainted*** as the result is as follows\n\n```\n array([[0.00662911, 0.02009881, 0.2134114 , 0.00788153, 0.7519792 ]],\n dtype=float32)\n```\n> This result stays deterministic every time I run the notebook in Google Colab. Good!\n\n\nAll I can say is that under these three different environments, every re-run will yield the same results - which is great since `seed_all` seems to work. But it still bothers me on why they give slightly different results when used in different environments. Any possible hypothesis to this as I feel my `seed_all` function is quite robust again randomness... My only guess is maybe on different GPU types there might be some small differences?\n\nPS: I also noticed that changing `torch` and `torchvision` versions do not affect the results.\n\nMy seeding function is:\n\n```\ndef seed_all(seed: int):\n if not seed:\n seed = 10\n\n print(\"[ Using Seed : \", seed, \" ]\")\n \n os.environ['PYTHONHASHSEED'] = str(seed) # set PYTHONHASHSEED env var at fixed value\n torch.manual_seed(seed)\n torch.cuda.manual_seed_all(seed)\n torch.cuda.manual_seed(seed) # pytorch (both CPU and CUDA)\n np.random.seed(seed) # for numpy pseudo-random generator\n random.seed(seed) # set fixed value for python built-in pseudo-random generator\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = False\n torch.backends.cudnn.enabled = False\n \ndef seed_worker(worker_id):\n worker_seed = torch.initial_seed() % 2**32\n np.random.seed(worker_seed)\n random.seed(worker_seed)\n```\n\nYes, I even seed the workers.", "votes": null }, { "id": "1104199", "postDate": "12/06/2020 17:43:52", "content": "

I guess you use mixed precision. Are all environments (GPUs) where you tested your pytorch code support mixed precision?

", "rawMarkdown": "I guess you use mixed precision. Are all environments (GPUs) where you tested your pytorch code support mixed precision?", "votes": null } ], "comments": [ { "id": 1104199, "author_name": "dunklerwald", "author_url": "", "post_date": "12/06/2020 17:43:52", "content": "

I guess you use mixed precision. Are all environments (GPUs) where you tested your pytorch code support mixed precision?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1103000": "EDIT: Possible answer [here](https://discuss.pytorch.org/t/reproducibility-over-different-machines/63047)\n\nFollowing [this discussion](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200182) I met my own problems. As I have quite a bit of OCD, I would like to seek clarification on the deviations of results I get here.\n\nProblem: In my [inference notebook here](https://www.kaggle.com/reighns/pytorch-a-complete-inference-guide), my predictions for the test image is \n\n```\n array([[0.00694168, 0.02380404, 0.19128187, 0.00819152, 0.7697809 ]],\n dtype=float32)\n```\n> This result stays deterministic every time I run the notebook in Kaggle. Good!\n\nNow I like to document my codes on Jupyter Notebooks for book-keeping purposes, but I got a rude shock to see that the result I get this time is \n\n```\n array([[0.0066291 , 0.02009881, 0.21341121, 0.00788152, 0.75197935]],\n dtype=float32)\n```\n\n> This result stays deterministic every time I run the notebook in Jupyter Notebook. Good!\n\nI then went to Google Colab and run the inference and nearly ***fainted*** as the result is as follows\n\n```\n array([[0.00662911, 0.02009881, 0.2134114 , 0.00788153, 0.7519792 ]],\n dtype=float32)\n```\n> This result stays deterministic every time I run the notebook in Google Colab. Good!\n\n\nAll I can say is that under these three different environments, every re-run will yield the same results - which is great since `seed_all` seems to work. But it still bothers me on why they give slightly different results when used in different environments. Any possible hypothesis to this as I feel my `seed_all` function is quite robust again randomness... My only guess is maybe on different GPU types there might be some small differences?\n\nPS: I also noticed that changing `torch` and `torchvision` versions do not affect the results.\n\nMy seeding function is:\n\n```\ndef seed_all(seed: int):\n if not seed:\n seed = 10\n\n print(\"[ Using Seed : \", seed, \" ]\")\n \n os.environ['PYTHONHASHSEED'] = str(seed) # set PYTHONHASHSEED env var at fixed value\n torch.manual_seed(seed)\n torch.cuda.manual_seed_all(seed)\n torch.cuda.manual_seed(seed) # pytorch (both CPU and CUDA)\n np.random.seed(seed) # for numpy pseudo-random generator\n random.seed(seed) # set fixed value for python built-in pseudo-random generator\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = False\n torch.backends.cudnn.enabled = False\n \ndef seed_worker(worker_id):\n worker_seed = torch.initial_seed() % 2**32\n np.random.seed(worker_seed)\n random.seed(worker_seed)\n```\n\nYes, I even seed the workers.", "1104199": "I guess you use mixed precision. Are all environments (GPUs) where you tested your pytorch code support mixed precision?" }, "source": "meta" }