I didn't really manage to set seed so I can get reproducible results when I train on TPUs. I am using keras and for each submit I get slightly different scores even after having seed random functions...
\n\nMaybe I do it the wrong way but if someone has an existing tip to correct that, I would really appreciate.
", "messages": [ { "id": 883836, "postDate": "2020-06-13T01:29:21.933Z", "content": "I didn't really manage to set seed so I can get reproducible results when I train on TPUs. I am using keras and for each submit I get slightly different scores even after having seed random functions...
\n\nMaybe I do it the wrong way but if someone has an existing tip to correct that, I would really appreciate.
", "rawMarkdown": "I didn't really manage to set seed so I can get reproducible results when I train on TPUs. I am using keras and for each submit I get slightly different scores even after having seed random functions...\n\nMaybe I do it the wrong way but if someone has an existing tip to correct that, I would really appreciate.", "votes": 2 }, { "id": 883839, "postDate": "2020-06-13T01:31:11.290Z", "content": "well, regarding this topic, it looks like there is no way we can get reproducible result using TPUs with keras... \nThe topic is a little outdated tho (few months), so if you have any clue...
", "rawMarkdown": "well, regarding this topic, it looks like there is no way we can get reproducible result using TPUs with keras... \nThe topic is a little outdated tho (few months), so if you have any clue...", "replies": [ { "id": 884941, "postDate": "2020-06-13T18:29:06.653Z", "content": "Hi, there is an excellent blog post talking about reproducibility in ML. She stated some of the important factor in having reproducible ML experiment. One of them is this:
\n\n\n\n\n** Rounding precision, under-flows & overflows**\n Floating point arithmetic is ubiquitous in ML. The complexity and intensity of floating point operations (FLOPS) are increasing everyday with current needs easily meeting Giga-Flops order of computations. To achieve the efficiency in terms of speed despite complexity, mixed precision floating point operations have also been proposed. As discussed Part 1, accelerated hardware such as (GPGPU), tensor processing unit (TPU) etc. due to their architecture and asynchronous computing do not guarantee reproducibility. In addition, when dealing with floating points, the issues related to overflow and underflow are expected. This just adds to the complexity.
\n
So, I don't think that it's possible just yet :)
", "rawMarkdown": "Hi, there is an excellent [blog post talking about reproducibility in ML](https://suneeta-mall.github.io/2019/12/22/Reproducible-ml-tensorflow.html). She stated some of the important factor in having reproducible ML experiment. One of them is this:\n\n> ** Rounding precision, under-flows & overflows**\nFloating point arithmetic is ubiquitous in ML. The complexity and intensity of floating point operations (FLOPS) are increasing everyday with current needs easily meeting Giga-Flops order of computations. To achieve the efficiency in terms of speed despite complexity, mixed precision floating point operations have also been proposed. As discussed Part 1, accelerated hardware such as (GPGPU), tensor processing unit (TPU) etc. due to their architecture and asynchronous computing do not guarantee reproducibility. In addition, when dealing with floating points, the issues related to overflow and underflow are expected. This just adds to the complexity.\n\nSo, I don't think that it's possible just yet :)", "votes": 1 } ] }, { "id": 885742, "postDate": "2020-06-14T12:52:00.920Z", "content": "I got the same problem in split, I used train_test_split + seed_everything, but not working that I got overfitting (high CV, not high LB). I changed to StratifiedKSplit training then, feel better.
", "rawMarkdown": "I got the same problem in split, I used train\\_test\\_split + seed\\_everything, but not working that I got overfitting (high CV, not high LB). I changed to StratifiedKSplit training then, feel better. ", "isDeleted": true } ], "comments": [ { "id": 883839, "author_name": "BryanB", "author_url": "", "post_date": "2020-06-13T01:31:11.290000", "content": "well, regarding this topic, it looks like there is no way we can get reproducible result using TPUs with keras... \nThe topic is a little outdated tho (few months), so if you have any clue...
", "votes": 0, "replies": [ { "id": 884941, "author_name": "Ilham Firdausi Putra", "author_url": "", "post_date": "2020-06-13T18:29:06.653000", "content": "Hi, there is an excellent blog post talking about reproducibility in ML. She stated some of the important factor in having reproducible ML experiment. One of them is this:
\n\n\n\n\n** Rounding precision, under-flows & overflows**\n Floating point arithmetic is ubiquitous in ML. The complexity and intensity of floating point operations (FLOPS) are increasing everyday with current needs easily meeting Giga-Flops order of computations. To achieve the efficiency in terms of speed despite complexity, mixed precision floating point operations have also been proposed. As discussed Part 1, accelerated hardware such as (GPGPU), tensor processing unit (TPU) etc. due to their architecture and asynchronous computing do not guarantee reproducibility. In addition, when dealing with floating points, the issues related to overflow and underflow are expected. This just adds to the complexity.
\n
So, I don't think that it's possible just yet :)
", "votes": 1, "replies": [] } ] }, { "id": 885742, "author_name": "", "author_url": "", "post_date": "2020-06-14T12:52:00.920000", "content": "I got the same problem in split, I used train_test_split + seed_everything, but not working that I got overfitting (high CV, not high LB). I changed to StratifiedKSplit training then, feel better.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "883836": "I didn't really manage to set seed so I can get reproducible results when I train on TPUs. I am using keras and for each submit I get slightly different scores even after having seed random functions...\n\nMaybe I do it the wrong way but if someone has an existing tip to correct that, I would really appreciate.", "883839": "well, regarding this topic, it looks like there is no way we can get reproducible result using TPUs with keras... \nThe topic is a little outdated tho (few months), so if you have any clue...", "885742": "I got the same problem in split, I used train\\_test\\_split + seed\\_everything, but not working that I got overfitting (high CV, not high LB). I changed to StratifiedKSplit training then, feel better. " } }