Hi,
\n\nI tried to run @xhlulu 's tensorflow kernel. But after running the kernel many times, I always got the different or random results for every time running.
My kernel: https://www.kaggle.com/qinhui1999/jigsaw-tpu-xlm-roberta?scriptVersionId=31385845
\nIn pytorch, if I fix the random seeds, then I always get the same result for every time running. Hence I tried to fix the random seeds like pytorch as follow:
SEED=2020
\nimport random
\ndef set_seed(seed):
\n random.seed(seed)
\n np.random.seed(seed)
\n torch.manual_seed(seed)
\n torch.cuda.manual_seed(seed)
\nset_seed(SEED)
To my dispointed, it does not work.
\nIs it anyway to make TPU get the deterministic result?
--------updated 04-03----\nFor tf 2.1 version, I use the following config and get the deterministic result in GPU. But it can not work in TPU.
\n\nSEED=2020
\nos.environ['PYTHONHASHSEED']=str(SEED)
\nrandom.seed(SEED)
\nnp.random.seed(SEED)
\ntf.random.set_seed(SEED)
os.environ['TF_DETERMINISTIC_OPS'] = '1'
---------- other reference--------\nNvidia TensorFlow Determinism ---- https://github.com/NVIDIA/tensorflow-determinism
", "messages": [ { "id": "795702", "postDate": "04/03/2020 00:55:45", "content": "Hi,
\n\nI tried to run @xhlulu 's tensorflow kernel. But after running the kernel many times, I always got the different or random results for every time running.
My kernel: https://www.kaggle.com/qinhui1999/jigsaw-tpu-xlm-roberta?scriptVersionId=31385845
\nIn pytorch, if I fix the random seeds, then I always get the same result for every time running. Hence I tried to fix the random seeds like pytorch as follow:
SEED=2020
\nimport random
\ndef set_seed(seed):
\n random.seed(seed)
\n np.random.seed(seed)
\n torch.manual_seed(seed)
\n torch.cuda.manual_seed(seed)
\nset_seed(SEED)
To my dispointed, it does not work.
\nIs it anyway to make TPU get the deterministic result?
--------updated 04-03----\nFor tf 2.1 version, I use the following config and get the deterministic result in GPU. But it can not work in TPU.
\n\nSEED=2020
\nos.environ['PYTHONHASHSEED']=str(SEED)
\nrandom.seed(SEED)
\nnp.random.seed(SEED)
\ntf.random.set_seed(SEED)
os.environ['TF_DETERMINISTIC_OPS'] = '1'
---------- other reference--------\nNvidia TensorFlow Determinism ---- https://github.com/NVIDIA/tensorflow-determinism
", "rawMarkdown": "Hi,\n\n\n\n I tried to run @xhlulu 's tensorflow kernel. But after running the kernel many times, I always got the different or random results for every time running.I thought random results is a tensorflow problem, TPU or not?
", "rawMarkdown": "I thought random results is a tensorflow problem, TPU or not?", "votes": null }, { "id": "795850", "postDate": "04/03/2020 04:51:32", "content": "@suicaokhoailang Yeah. But I still think tensorflow can do the same thing like pytorch.
", "rawMarkdown": "suicaokhoailang Yeah. But I still think tensorflow can do the same thing like pytorch.", "votes": null }, { "id": "796028", "postDate": "04/03/2020 08:22:26", "content": "It's tensorflow features/bugs(it's hard to say what is it really). You can obtain the same results in two independent running only if would not be used parallelism (used 1 core of cpu only), but and this not guaranteed the same results. P.s. This video can help you understand why it's happening.
", "rawMarkdown": "It's tensorflow features/bugs(it's hard to say what is it really). You can obtain the same results in two independent running only if would not be used parallelism (used 1 core of cpu only), but and this not guaranteed the same results. P.s. This [video ](https://www.youtube.com/watch?v=Ys8ofBeR2kA&feature=youtu.be)can help you understand why it's happening.", "votes": null }, { "id": "796790", "postDate": "04/03/2020 23:14:49", "content": "\n\nIn my custom version, something like the following helps, but not completely deterministic (results are close in different run, but still having some small difference).
\n\nIn tensorflow, there are global seed and operation seed. You have to make sure both of them are set.
\n\n\n self.dense_layer = tf.keras.layers.Dense(1, activation='sigmoid', kernel_initializer=tf.keras.initializers.GlorotUniform(seed=OP_SEED), bias_initializer='zeros')\n\nalong with
```\n NO_RANDOM = True
\n\nGLOBAL_SEED = None\nOP_SEED = None\n\nif NO_RANDOM:\n GLOBAL_SEED = 127\n OP_SEED = 127\n tf.random.set_seed(seed=GLOBAL_SEED)\nI also saw
\n\nos.environ['TF_DETERMINISTIC_OPS'] = '1'\nfrom
\n\nhttps://www.kaggle.com/dimitreoliveira/jigsaw-tpu-optimized-training-loops
", "rawMarkdown": "I also saw \n\n os.environ['TF_DETERMINISTIC_OPS'] = '1'\n\nfrom \n\n[https://www.kaggle.com/dimitreoliveira/jigsaw-tpu-optimized-training-loops](https://www.kaggle.com/dimitreoliveira/jigsaw-tpu-optimized-training-loops)", "votes": null }, { "id": "796852", "postDate": "04/04/2020 02:06:00", "content": "@miklgr500 @yihdarshieh Thanks for your help. I tested it again, now GPU is ok ,but TPU is not ok. I still got random results. : (
", "rawMarkdown": "miklgr500 @yihdarshieh Thanks for your help. I tested it again, now GPU is ok ,but TPU is not ok. I still got random results. : (", "votes": null }, { "id": "802012", "postDate": "04/09/2020 03:32:27", "content": "https://medium.com/datadriveninvestor/getting-reproducible-results-in-tensorflow-3705536aa185, you can always check this out if this helps.
", "rawMarkdown": "https://medium.com/datadriveninvestor/getting-reproducible-results-in-tensorflow-3705536aa185, you can always check this out if this helps.", "votes": null }, { "id": "805264", "postDate": "04/12/2020 14:13:36", "content": "Check out this help section about Deterministic Training on TPUs: https://cloud.google.com/tpu/docs/troubleshooting#deterministic-training
\n\nMain takeaway: Single core TPU training can be deterministic, multi-core TPU training may lead to varying results (Due to data sharding)
", "rawMarkdown": "Check out this help section about Deterministic Training on TPUs: https://cloud.google.com/tpu/docs/troubleshooting#deterministic-training\n\nMain takeaway: Single core TPU training can be deterministic, multi-core TPU training may lead to varying results (Due to data sharding)", "votes": null }, { "id": "805305", "postDate": "04/12/2020 14:56:17", "content": "@jerryqu Thanks for your help. So we always get the random results for multi-core TPU. : (
", "rawMarkdown": "jerryqu Thanks for your help. So we always get the random results for multi-core TPU. : (", "votes": null }, { "id": "805309", "postDate": "04/12/2020 15:01:38", "content": "@harshitsheoran Thanks for your help. But it can not work in multi-core TPU.
", "rawMarkdown": "harshitsheoran Thanks for your help. But it can not work in multi-core TPU.", "votes": null }, { "id": "806798", "postDate": "04/14/2020 04:39:22", "content": "@qinhui1999 In my understanding, if we turn on os.environ['TF_DETERMINISTIC_OPS'] = '1', training speed is way slower. Have you experienced that?
", "rawMarkdown": "qinhui1999 In my understanding, if we turn on os.environ['TF_DETERMINISTIC_OPS'] = '1', training speed is way slower. Have you experienced that?", "votes": null }, { "id": "806805", "postDate": "04/14/2020 04:52:33", "content": "@bamps53 No, I do not test the speed. When we use the deterministic function, we just want to check if the new features or changes can bring some improvements in kpi.
", "rawMarkdown": "bamps53 No, I do not test the speed. When we use the deterministic function, we just want to check if the new features or changes can bring some improvements in kpi.", "votes": null }, { "id": "807257", "postDate": "04/14/2020 14:15:33", "content": "@qinhui1999 I know what you want to do, but I saw someone reported if you turn on TFDETERMINISTICOPS, it makes training speed very slower. (he said more than 1.5 times) So I gave up to do even I eagerly want to deterministic result!
", "rawMarkdown": "qinhui1999 I know what you want to do, but I saw someone reported if you turn on TFDETERMINISTICOPS, it makes training speed very slower. (he said more than 1.5 times) So I gave up to do even I eagerly want to deterministic result!", "votes": null }, { "id": "808226", "postDate": "04/15/2020 08:20:06", "content": "yes, this is really cruel. Did you find any solution? @qinhui1999
", "rawMarkdown": "yes, this is really cruel. Did you find any solution? @qinhui1999", "votes": null }, { "id": "808381", "postDate": "04/15/2020 11:24:16", "content": "@shahules No solution. You can check this page: https://cloud.google.com/tpu/docs/troubleshooting#deterministic-training
", "rawMarkdown": "shahules No solution. You can check this page: https://cloud.google.com/tpu/docs/troubleshooting#deterministic-training", "votes": null }, { "id": "808417", "postDate": "04/15/2020 12:13:52", "content": "@qinhui1999 Thanks for your reply, I was getting good performece when Running But it decreases whenever I commit.Any hacks for this issue ?
", "rawMarkdown": "qinhui1999 Thanks for your reply, I was getting good performece when Running But it decreases whenever I commit.Any hacks for this issue ?", "votes": null }, { "id": "1106434", "postDate": "12/08/2020 20:44:59", "content": "While we can solve the variation in GPU setup easily by setting the random seeds… Therefore it's more seen in TPU setup as the model is trained on multi core. The sharding and loss calculation techniques leads to different results each time and also different from a version that is trained on GPU.
\nFollowing documentation explains the reasons behind the variations very well along with debugging methods.
\nhttps://cloud.google.com/tpu/docs/troubleshooting#model-accuracy
I thought random results is a tensorflow problem, TPU or not?
", "votes": null, "replies": [ { "id": 1106434, "author_name": "akagarw", "author_url": "", "post_date": "12/08/2020 20:44:59", "content": "While we can solve the variation in GPU setup easily by setting the random seeds… Therefore it's more seen in TPU setup as the model is trained on multi core. The sharding and loss calculation techniques leads to different results each time and also different from a version that is trained on GPU.
\nFollowing documentation explains the reasons behind the variations very well along with debugging methods.
\nhttps://cloud.google.com/tpu/docs/troubleshooting#model-accuracy
@suicaokhoailang Yeah. But I still think tensorflow can do the same thing like pytorch.
", "votes": null, "replies": [] }, { "id": 796028, "author_name": "miklgr500", "author_url": "", "post_date": "04/03/2020 08:22:26", "content": "It's tensorflow features/bugs(it's hard to say what is it really). You can obtain the same results in two independent running only if would not be used parallelism (used 1 core of cpu only), but and this not guaranteed the same results. P.s. This video can help you understand why it's happening.
", "votes": null, "replies": [] }, { "id": 796790, "author_name": "yihdarshieh", "author_url": "", "post_date": "04/03/2020 23:14:49", "content": "\n\nIn my custom version, something like the following helps, but not completely deterministic (results are close in different run, but still having some small difference).
\n\nIn tensorflow, there are global seed and operation seed. You have to make sure both of them are set.
\n\n\n self.dense_layer = tf.keras.layers.Dense(1, activation='sigmoid', kernel_initializer=tf.keras.initializers.GlorotUniform(seed=OP_SEED), bias_initializer='zeros')\n\nalong with
```\n NO_RANDOM = True
\n\nGLOBAL_SEED = None\nOP_SEED = None\n\nif NO_RANDOM:\n GLOBAL_SEED = 127\n OP_SEED = 127\n tf.random.set_seed(seed=GLOBAL_SEED)\nI also saw
\n\nos.environ['TF_DETERMINISTIC_OPS'] = '1'\nfrom
\n\nhttps://www.kaggle.com/dimitreoliveira/jigsaw-tpu-optimized-training-loops
", "votes": null, "replies": [] }, { "id": 796852, "author_name": "qinhui1999", "author_url": "", "post_date": "04/04/2020 02:06:00", "content": "@miklgr500 @yihdarshieh Thanks for your help. I tested it again, now GPU is ok ,but TPU is not ok. I still got random results. : (
", "votes": null, "replies": [] }, { "id": 802012, "author_name": "harshitsheoran", "author_url": "", "post_date": "04/09/2020 03:32:27", "content": "https://medium.com/datadriveninvestor/getting-reproducible-results-in-tensorflow-3705536aa185, you can always check this out if this helps.
", "votes": null, "replies": [ { "id": 805309, "author_name": "qinhui1999", "author_url": "", "post_date": "04/12/2020 15:01:38", "content": "@harshitsheoran Thanks for your help. But it can not work in multi-core TPU.
", "votes": null, "replies": [] } ] }, { "id": 805264, "author_name": "jerryqu", "author_url": "", "post_date": "04/12/2020 14:13:36", "content": "Check out this help section about Deterministic Training on TPUs: https://cloud.google.com/tpu/docs/troubleshooting#deterministic-training
\n\nMain takeaway: Single core TPU training can be deterministic, multi-core TPU training may lead to varying results (Due to data sharding)
", "votes": null, "replies": [ { "id": 805305, "author_name": "qinhui1999", "author_url": "", "post_date": "04/12/2020 14:56:17", "content": "@jerryqu Thanks for your help. So we always get the random results for multi-core TPU. : (
", "votes": null, "replies": [] } ] }, { "id": 806798, "author_name": "bamps53", "author_url": "", "post_date": "04/14/2020 04:39:22", "content": "@qinhui1999 In my understanding, if we turn on os.environ['TF_DETERMINISTIC_OPS'] = '1', training speed is way slower. Have you experienced that?
", "votes": null, "replies": [ { "id": 806805, "author_name": "qinhui1999", "author_url": "", "post_date": "04/14/2020 04:52:33", "content": "@bamps53 No, I do not test the speed. When we use the deterministic function, we just want to check if the new features or changes can bring some improvements in kpi.
", "votes": null, "replies": [] }, { "id": 807257, "author_name": "bamps53", "author_url": "", "post_date": "04/14/2020 14:15:33", "content": "@qinhui1999 I know what you want to do, but I saw someone reported if you turn on TFDETERMINISTICOPS, it makes training speed very slower. (he said more than 1.5 times) So I gave up to do even I eagerly want to deterministic result!
", "votes": null, "replies": [] } ] }, { "id": 808226, "author_name": "shahules", "author_url": "", "post_date": "04/15/2020 08:20:06", "content": "yes, this is really cruel. Did you find any solution? @qinhui1999
", "votes": null, "replies": [ { "id": 808381, "author_name": "qinhui1999", "author_url": "", "post_date": "04/15/2020 11:24:16", "content": "@shahules No solution. You can check this page: https://cloud.google.com/tpu/docs/troubleshooting#deterministic-training
", "votes": null, "replies": [] }, { "id": 808417, "author_name": "shahules", "author_url": "", "post_date": "04/15/2020 12:13:52", "content": "@qinhui1999 Thanks for your reply, I was getting good performece when Running But it decreases whenever I commit.Any hacks for this issue ?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "795702": "Hi,\n\n\n\n I tried to run @xhlulu 's tensorflow kernel. But after running the kernel many times, I always got the different or random results for every time running.