If you are using Pytorch Dataset and have inside a function that uses random number generated please consider adding following modification to Dataloader (it has shown that random generated is not random and the number always repeat)
def worker_init_fn(worker_id): \n np.random.seed(np.random.get_state()[1][0] + worker_id)\n\ndataset = RandomDataset()\ndataloader = DataLoader(dataset, batch_size=2, num_workers=4, \n worker_init_fn=worker_init_fn)\n\nfor batch in dataloader:\n print(batch)\ncredits:
\nhttps://tanelp.github.io/posts/a-bug-that-plagues-thousands-of-open-source-ml-projects/
\nhttps://twitter.com/karpathy/status/1381121537164513288?s=20
If you are using Pytorch Dataset and have inside a function that uses random number generated please consider adding following modification to Dataloader (it has shown that random generated is not random and the number always repeat)
def worker_init_fn(worker_id): \n np.random.seed(np.random.get_state()[1][0] + worker_id)\n\ndataset = RandomDataset()\ndataloader = DataLoader(dataset, batch_size=2, num_workers=4, \n worker_init_fn=worker_init_fn)\n\nfor batch in dataloader:\n print(batch)\ncredits:
\nhttps://tanelp.github.io/posts/a-bug-that-plagues-thousands-of-open-source-ml-projects/
\nhttps://twitter.com/karpathy/status/1381121537164513288?s=20
i face this problem before.
\nit is always important to write check function.
it is actually not a bug. this is because np random function is not thread safe.
", "rawMarkdown": "i face this problem before.\nit is always important to write check function.\n\nit is actually not a bug. this is because np random function is not thread safe.", "votes": null }, { "id": "1270240", "postDate": "04/11/2021 13:11:49", "content": "I guess you are right its not a bug, but a feature which is probably not well-known and can cause a lot of issue and very hard to debug problems when overlooked.
", "rawMarkdown": "I guess you are right its not a bug, but a feature which is probably not well-known and can cause a lot of issue and very hard to debug problems when overlooked.", "votes": null }, { "id": "1270400", "postDate": "04/11/2021 16:01:58", "content": "I was amazed to see this today; so easy to miss it even if you do look at the data! Emphasizes how important it is to carefully inspect what goes into the model from one batch to another.
", "rawMarkdown": "I was amazed to see this today; so easy to miss it even if you do look at the data! Emphasizes how important it is to carefully inspect what goes into the model from one batch to another.", "votes": null }, { "id": "1271254", "postDate": "04/12/2021 12:37:07", "content": "here is more information
\nhttps://www.reddit.com/r/MachineLearning/comments/mocpgj/p_using_pytorch_numpy_a_bug_that_plagues/
Using NumPy’s random number generator with multi-process data loading in PyTorch causes identical augmentations unless you specifically set seeds using the worker_init_fn option in the DataLoader. I didn’t and this bug silently regressed my model’s accuracy.
\n\". Out of these, over 95% of the repositories are plagued by this problem. It’s inside PyTorch's official tutorial, OpenAI’s code, and NVIDIA’s projects. Even Karpathy admitted falling prey to it.\"
\nhttps://tanelp.github.io/posts/a-bug-that-plagues-thousands-of-open-source-ml-projects/
", "rawMarkdown": "here is more information\nhttps://www.reddit.com/r/MachineLearning/comments/mocpgj/p_using_pytorch_numpy_a_bug_that_plagues/\n\nUsing NumPy’s random number generator with multi-process data loading in PyTorch causes identical augmentations unless you specifically set seeds using the worker_init_fn option in the DataLoader. I didn’t and this bug silently regressed my model’s accuracy.\n\n\". Out of these, over 95% of the repositories are plagued by this problem. It’s inside PyTorch's official tutorial, OpenAI’s code, and NVIDIA’s projects. Even Karpathy admitted falling prey to it.\"\n\nhttps://tanelp.github.io/posts/a-bug-that-plagues-thousands-of-open-source-ml-projects/", "votes": null }, { "id": "1273535", "postDate": "04/14/2021 12:56:07", "content": "Wow, thanks for that..I only recently started using Pytorch and just yesterday implemented exactly that.
", "rawMarkdown": "Wow, thanks for that..I only recently started using Pytorch and just yesterday implemented exactly that.", "votes": null }, { "id": "1274498", "postDate": "04/15/2021 11:08:16", "content": "terrified*
", "rawMarkdown": "terrified*", "votes": null }, { "id": "1275822", "postDate": "04/16/2021 18:24:45", "content": "I read this on Reddit when they posted it, but it isn't clear to me yet if this happen even if you dont set the seed manually before instantiating the Dataloader.
", "rawMarkdown": "I read this on Reddit when they posted it, but it isn't clear to me yet if this happen even if you dont set the seed manually before instantiating the Dataloader.", "votes": null } ], "comments": [ { "id": 1270237, "author_name": "hengck23", "author_url": "", "post_date": "04/11/2021 13:05:38", "content": "i face this problem before.
\nit is always important to write check function.
it is actually not a bug. this is because np random function is not thread safe.
", "votes": null, "replies": [ { "id": 1270240, "author_name": "drhabib", "author_url": "", "post_date": "04/11/2021 13:11:49", "content": "I guess you are right its not a bug, but a feature which is probably not well-known and can cause a lot of issue and very hard to debug problems when overlooked.
", "votes": null, "replies": [] } ] }, { "id": 1270400, "author_name": "kozodoi", "author_url": "", "post_date": "04/11/2021 16:01:58", "content": "I was amazed to see this today; so easy to miss it even if you do look at the data! Emphasizes how important it is to carefully inspect what goes into the model from one batch to another.
", "votes": null, "replies": [ { "id": 1274498, "author_name": "nofreewill", "author_url": "", "post_date": "04/15/2021 11:08:16", "content": "terrified*
", "votes": null, "replies": [] } ] }, { "id": 1271254, "author_name": "hengck23", "author_url": "", "post_date": "04/12/2021 12:37:07", "content": "here is more information
\nhttps://www.reddit.com/r/MachineLearning/comments/mocpgj/p_using_pytorch_numpy_a_bug_that_plagues/
Using NumPy’s random number generator with multi-process data loading in PyTorch causes identical augmentations unless you specifically set seeds using the worker_init_fn option in the DataLoader. I didn’t and this bug silently regressed my model’s accuracy.
\n\". Out of these, over 95% of the repositories are plagued by this problem. It’s inside PyTorch's official tutorial, OpenAI’s code, and NVIDIA’s projects. Even Karpathy admitted falling prey to it.\"
\nhttps://tanelp.github.io/posts/a-bug-that-plagues-thousands-of-open-source-ml-projects/
", "votes": null, "replies": [] }, { "id": 1273535, "author_name": "kalfasyan", "author_url": "", "post_date": "04/14/2021 12:56:07", "content": "Wow, thanks for that..I only recently started using Pytorch and just yesterday implemented exactly that.
", "votes": null, "replies": [] }, { "id": 1275822, "author_name": "claverru", "author_url": "", "post_date": "04/16/2021 18:24:45", "content": "I read this on Reddit when they posted it, but it isn't clear to me yet if this happen even if you dont set the seed manually before instantiating the Dataloader.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1270234": "If you are using Pytorch `Dataset` and have inside a function that uses random number generated please consider adding following modification to `Dataloader` (it has shown that `random` generated is not random and the number always repeat) \n\n```\ndef worker_init_fn(worker_id): \n np.random.seed(np.random.get_state()[1][0] + worker_id)\n\ndataset = RandomDataset()\ndataloader = DataLoader(dataset, batch_size=2, num_workers=4, \n worker_init_fn=worker_init_fn)\n\nfor batch in dataloader:\n print(batch)\n```\ncredits:\nhttps://tanelp.github.io/posts/a-bug-that-plagues-thousands-of-open-source-ml-projects/\nhttps://twitter.com/karpathy/status/1381121537164513288?s=20", "1270237": "i face this problem before.\nit is always important to write check function.\n\nit is actually not a bug. this is because np random function is not thread safe.", "1270240": "I guess you are right its not a bug, but a feature which is probably not well-known and can cause a lot of issue and very hard to debug problems when overlooked.", "1270400": "I was amazed to see this today; so easy to miss it even if you do look at the data! Emphasizes how important it is to carefully inspect what goes into the model from one batch to another.", "1271254": "here is more information\nhttps://www.reddit.com/r/MachineLearning/comments/mocpgj/p_using_pytorch_numpy_a_bug_that_plagues/\n\nUsing NumPy’s random number generator with multi-process data loading in PyTorch causes identical augmentations unless you specifically set seeds using the worker_init_fn option in the DataLoader. I didn’t and this bug silently regressed my model’s accuracy.\n\n\". Out of these, over 95% of the repositories are plagued by this problem. It’s inside PyTorch's official tutorial, OpenAI’s code, and NVIDIA’s projects. Even Karpathy admitted falling prey to it.\"\n\nhttps://tanelp.github.io/posts/a-bug-that-plagues-thousands-of-open-source-ml-projects/", "1273535": "Wow, thanks for that..I only recently started using Pytorch and just yesterday implemented exactly that.", "1274498": "terrified*", "1275822": "I read this on Reddit when they posted it, but it isn't clear to me yet if this happen even if you dont set the seed manually before instantiating the Dataloader." }, "source": "meta" }