I just shared work I did for Bengali for scoring on TPU. Given TPU wasn't allowed for scoring I didn't test it. Let me share code for training, I'll work and share the code for scoring ASAP.
\n\nThis uses a public dataset that contains what is needed to install Pytorch.
\n\nDataset: https://www.kaggle.com/cpmpml/torchxla
\n\nTest notebook: https://www.kaggle.com/cpmpml/tpu-test
\n\nA more realistic code: https://www.kaggle.com/cpmpml/tpu-sub-bs256-full-train
\n\nJust look where xm or device is used to see where you code may need to be modified.
The above uses only one TPU core. To use the 8 cores you need parallel code. I'll share an example but you can look at this one already: https://www.kaggle.com/dhananjay3/pytorch-xla-for-tpu-with-multiprocessing
\n\nPlease upvote the dataset if you use it ;)
", "messages": [ { "id": "784587", "postDate": "03/24/2020 11:00:29", "content": "I just shared work I did for Bengali for scoring on TPU. Given TPU wasn't allowed for scoring I didn't test it. Let me share code for training, I'll work and share the code for scoring ASAP.
\n\nThis uses a public dataset that contains what is needed to install Pytorch.
\n\nDataset: https://www.kaggle.com/cpmpml/torchxla
\n\nTest notebook: https://www.kaggle.com/cpmpml/tpu-test
\n\nA more realistic code: https://www.kaggle.com/cpmpml/tpu-sub-bs256-full-train
\n\nJust look where xm or device is used to see where you code may need to be modified.
The above uses only one TPU core. To use the 8 cores you need parallel code. I'll share an example but you can look at this one already: https://www.kaggle.com/dhananjay3/pytorch-xla-for-tpu-with-multiprocessing
\n\nPlease upvote the dataset if you use it ;)
", "rawMarkdown": "I just shared work I did for Bengali for scoring on TPU. Given TPU wasn't allowed for scoring I didn't test it. Let me share code for training, I'll work and share the code for scoring ASAP.\n\nThis uses a public dataset that contains what is needed to install Pytorch.\n\nDataset: https://www.kaggle.com/cpmpml/torchxla\n\nTest notebook: https://www.kaggle.com/cpmpml/tpu-test\n\nA more realistic code: https://www.kaggle.com/cpmpml/tpu-sub-bs256-full-train\n\nJust look where `xm` or `device` is used to see where you code may need to be modified.\n\nThe above uses only one TPU core. To use the 8 cores you need parallel code. I'll share an example but you can look at this one already: https://www.kaggle.com/dhananjay3/pytorch-xla-for-tpu-with-multiprocessing\n\nPlease upvote the dataset if you use it ;)", "votes": null }, { "id": "784683", "postDate": "03/24/2020 12:37:57", "content": "Thanks for sharing. I should follow all of your notebooks...
", "rawMarkdown": "Thanks for sharing. I should follow all of your notebooks...", "votes": null }, { "id": "784827", "postDate": "03/24/2020 14:29:27", "content": "Thanks, I just made them public, they were private until today.
", "rawMarkdown": "Thanks, I just made them public, they were private until today.", "votes": null }, { "id": "785018", "postDate": "03/24/2020 17:52:49", "content": "How does training speed on kaggle kernels compare with TF?
", "rawMarkdown": "How does training speed on kaggle kernels compare with TF?", "votes": null }, { "id": "785100", "postDate": "03/24/2020 19:22:32", "content": "@abhishek used a similar approcah to train on TPUs, check it out here
", "rawMarkdown": "abhishek used a similar approcah to train on TPUs, check it out [here](https://www.kaggle.com/abhishek/bert-multi-lingual-tpu-training)", "votes": null }, { "id": "785301", "postDate": "03/25/2020 00:34:26", "content": "@dhananjay3 posted a 3x faster version of his notebook here: https://www.kaggle.com/dhananjay3/fast-pytorch-xla-for-tpu-with-multiprocessing/notebook
", "rawMarkdown": "dhananjay3 posted a 3x faster version of his notebook here: https://www.kaggle.com/dhananjay3/fast-pytorch-xla-for-tpu-with-multiprocessing/notebook", "votes": null }, { "id": "785310", "postDate": "03/25/2020 00:37:30", "content": "His kernel downloads pytorch each time it runs. The rest is similar indeed as we all use torch-xla api.
", "rawMarkdown": "His kernel downloads pytorch each time it runs. The rest is similar indeed as we all use torch-xla api.", "votes": null }, { "id": "785311", "postDate": "03/25/2020 00:37:48", "content": "training speed of what?
", "rawMarkdown": "training speed of what?", "votes": null }, { "id": "785312", "postDate": "03/25/2020 00:38:05", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": null }, { "id": "785473", "postDate": "03/25/2020 05:06:54", "content": "Here's the trouble, to use TPU's efficiently wrt PyTorch-XLA, you need more RAM on kernels;
", "rawMarkdown": "Here's the trouble, to use TPU's efficiently wrt PyTorch-XLA, you need more RAM on kernels;", "votes": null }, { "id": "785485", "postDate": "03/25/2020 05:28:43", "content": "There's a simpler way:
\n\nimport tensorflow as torch
Ya sure! That's hell of an import :)
", "rawMarkdown": "Ya sure! That's hell of an import :)", "votes": null }, { "id": "786282", "postDate": "03/25/2020 18:50:16", "content": "Any way to make Pytorch deterministic on TPU?
\n\n\nimport tensorflow as torch
\n
It would be great indeed if it worked with pytorch api.
", "rawMarkdown": "> import tensorflow as torch\n\nIt would be great indeed if it worked with pytorch api.", "votes": null }, { "id": "787334", "postDate": "03/26/2020 18:04:54", "content": "ROFL
", "rawMarkdown": "ROFL", "votes": null }, { "id": "794810", "postDate": "04/02/2020 05:34:54", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": null }, { "id": "849830", "postDate": "05/16/2020 05:29:21", "content": "HERE To make TPU training deterministic\npython\ndef seed_everything(seed=396):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n
Thanks for sharing. This seems to be the same code snippet for GPU-Pytorch. Is there anything specific to TPUs? 👀
", "rawMarkdown": "Thanks for sharing. This seems to be the same code snippet for GPU-Pytorch. Is there anything specific to TPUs? 👀", "votes": null }, { "id": "893755", "postDate": "06/19/2020 21:28:16", "content": "It seems so. I guess being careful with dataset creation and gc.collect() often are key then.
No :) same script for GPU and TPU
Thanks for sharing. I should follow all of your notebooks...
", "votes": null, "replies": [ { "id": 784827, "author_name": "cpmpml", "author_url": "", "post_date": "03/24/2020 14:29:27", "content": "Thanks, I just made them public, they were private until today.
", "votes": null, "replies": [] } ] }, { "id": 785018, "author_name": "lopuhin", "author_url": "", "post_date": "03/24/2020 17:52:49", "content": "How does training speed on kaggle kernels compare with TF?
", "votes": null, "replies": [ { "id": 785311, "author_name": "cpmpml", "author_url": "", "post_date": "03/25/2020 00:37:48", "content": "training speed of what?
", "votes": null, "replies": [] } ] }, { "id": 785100, "author_name": "rimijoker", "author_url": "", "post_date": "03/24/2020 19:22:32", "content": "@abhishek used a similar approcah to train on TPUs, check it out here
", "votes": null, "replies": [ { "id": 785310, "author_name": "cpmpml", "author_url": "", "post_date": "03/25/2020 00:37:30", "content": "His kernel downloads pytorch each time it runs. The rest is similar indeed as we all use torch-xla api.
", "votes": null, "replies": [] } ] }, { "id": 785301, "author_name": "mgorner", "author_url": "", "post_date": "03/25/2020 00:34:26", "content": "@dhananjay3 posted a 3x faster version of his notebook here: https://www.kaggle.com/dhananjay3/fast-pytorch-xla-for-tpu-with-multiprocessing/notebook
", "votes": null, "replies": [ { "id": 785312, "author_name": "cpmpml", "author_url": "", "post_date": "03/25/2020 00:38:05", "content": "Thanks for sharing.
", "votes": null, "replies": [] } ] }, { "id": 785473, "author_name": "adityaecdrid", "author_url": "", "post_date": "03/25/2020 05:06:54", "content": "Here's the trouble, to use TPU's efficiently wrt PyTorch-XLA, you need more RAM on kernels;
", "votes": null, "replies": [ { "id": 893755, "author_name": "yassinealouini", "author_url": "", "post_date": "06/19/2020 21:28:16", "content": "It seems so. I guess being careful with dataset creation and gc.collect() often are key then.
There's a simpler way:
\n\nimport tensorflow as torch
Ya sure! That's hell of an import :)
", "votes": null, "replies": [] }, { "id": 787138, "author_name": "cpmpml", "author_url": "", "post_date": "03/26/2020 14:47:52", "content": "\n\n\nimport tensorflow as torch
\n
It would be great indeed if it worked with pytorch api.
", "votes": null, "replies": [] }, { "id": 787334, "author_name": "mgorner", "author_url": "", "post_date": "03/26/2020 18:04:54", "content": "ROFL
", "votes": null, "replies": [] } ] }, { "id": 786282, "author_name": "coolcoder22", "author_url": "", "post_date": "03/25/2020 18:50:16", "content": "Any way to make Pytorch deterministic on TPU?
HERE To make TPU training deterministic\npython\ndef seed_everything(seed=396):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n
Thanks for sharing. This seems to be the same code snippet for GPU-Pytorch. Is there anything specific to TPUs? 👀
", "votes": null, "replies": [] }, { "id": 934397, "author_name": "coolcoder22", "author_url": "", "post_date": "07/18/2020 12:09:57", "content": "No :) same script for GPU and TPU
Thanks for sharing.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "784587": "I just shared work I did for Bengali for scoring on TPU. Given TPU wasn't allowed for scoring I didn't test it. Let me share code for training, I'll work and share the code for scoring ASAP.\n\nThis uses a public dataset that contains what is needed to install Pytorch.\n\nDataset: https://www.kaggle.com/cpmpml/torchxla\n\nTest notebook: https://www.kaggle.com/cpmpml/tpu-test\n\nA more realistic code: https://www.kaggle.com/cpmpml/tpu-sub-bs256-full-train\n\nJust look where `xm` or `device` is used to see where you code may need to be modified.\n\nThe above uses only one TPU core. To use the 8 cores you need parallel code. I'll share an example but you can look at this one already: https://www.kaggle.com/dhananjay3/pytorch-xla-for-tpu-with-multiprocessing\n\nPlease upvote the dataset if you use it ;)", "784683": "Thanks for sharing. I should follow all of your notebooks...", "784827": "Thanks, I just made them public, they were private until today.", "785018": "How does training speed on kaggle kernels compare with TF?", "785100": "abhishek used a similar approcah to train on TPUs, check it out [here](https://www.kaggle.com/abhishek/bert-multi-lingual-tpu-training)", "785301": "dhananjay3 posted a 3x faster version of his notebook here: https://www.kaggle.com/dhananjay3/fast-pytorch-xla-for-tpu-with-multiprocessing/notebook", "785310": "His kernel downloads pytorch each time it runs. The rest is similar indeed as we all use torch-xla api.", "785311": "training speed of what?", "785312": "Thanks for sharing.", "785473": "Here's the trouble, to use TPU's efficiently wrt PyTorch-XLA, you need more RAM on kernels;", "785485": "There's a simpler way:\n\n```import tensorflow as torch```", "785498": "Ya sure! That's hell of an import :)", "786282": "Any way to make Pytorch `deterministic` on TPU?", "787138": "> import tensorflow as torch\n\nIt would be great indeed if it worked with pytorch api.", "787334": "ROFL", "794810": "Thanks for sharing.", "849830": "`HERE` To make `TPU` training `deterministic`\n```python\ndef seed_everything(seed=396):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n```", "893754": "Thanks for sharing. This seems to be the same code snippet for GPU-Pytorch. Is there anything specific to TPUs? 👀", "893755": "It seems so. I guess being careful with dataset creation and `gc.collect()` often are key then.", "934397": "No :) same script for `GPU` and `TPU`" }, "source": "meta" }