I've seen many notebooks utilizing TPUs for Tensorflow but I was trying to explore materials for TPU training with PyTorch.
\nI found hugging face accelerate, which allows running your code on Multi-GPUs or TPUs with just 4 lines of additional code.
\nFrom the docs:
\n\n\n🤗 Accelerate provides an easy API to make your scripts run with mixed precision and on any kind of distributed setting (multi-GPUs, TPUs etc.) while still letting you write your own training loop. The same code can then runs seamlessly on your local machine for debugging or your training environment.
\n🤗 Accelerate also provides a CLI tool that allows you to quickly configure and test your training environment then launch the scripts.
\n
Some resources:
\n\nLet's bring PyTorch to TPUs 🚀 I'll be playing around with the public kernels and will keep everyone posted of my experimentations with TPUs.
", "messages": [ { "id": "1698238", "postDate": "02/20/2022 08:23:54", "content": "I've seen many notebooks utilizing TPUs for Tensorflow but I was trying to explore materials for TPU training with PyTorch.
\nI found hugging face accelerate, which allows running your code on Multi-GPUs or TPUs with just 4 lines of additional code.
\nFrom the docs:
\n\n\n🤗 Accelerate provides an easy API to make your scripts run with mixed precision and on any kind of distributed setting (multi-GPUs, TPUs etc.) while still letting you write your own training loop. The same code can then runs seamlessly on your local machine for debugging or your training environment.
\n🤗 Accelerate also provides a CLI tool that allows you to quickly configure and test your training environment then launch the scripts.
\n
Some resources:
\n\nLet's bring PyTorch to TPUs 🚀 I'll be playing around with the public kernels and will keep everyone posted of my experimentations with TPUs.
", "rawMarkdown": "I've seen many notebooks utilizing TPUs for Tensorflow but I was trying to explore materials for TPU training with PyTorch.\n\nI found hugging face accelerate, which allows running your code on Multi-GPUs or TPUs with just 4 lines of additional code.\n\nFrom the [docs](https://huggingface.co/docs/accelerate/):\n\n> 🤗 Accelerate provides an easy API to make your scripts run with mixed precision and on any kind of distributed setting (multi-GPUs, TPUs etc.) while still letting you write your own training loop. The same code can then runs seamlessly on your local machine for debugging or your training environment.\n\n\n> 🤗 Accelerate also provides a CLI tool that allows you to quickly configure and test your training environment then launch the scripts.\n\nSome resources: \n- [Video Walkthrough by Sylvain Gugger](https://www.youtube.com/watch?v=A7lnu-ZsFZs)\n- [Shorter video](https://www.youtube.com/watch?v=s7dy8QRgjJ0)\n- [Github](https://github.com/huggingface/accelerate)\n\nLet's bring PyTorch to TPUs 🚀 I'll be playing around with the public kernels and will keep everyone posted of my experimentations with TPUs.", "votes": null }, { "id": "1699195", "postDate": "02/21/2022 02:20:18", "content": "great!! Thanks!!!!👍👍👍👍👍
", "rawMarkdown": "great!! Thanks!!!!👍👍👍👍👍", "votes": null }, { "id": "1699278", "postDate": "02/21/2022 04:43:04", "content": "This seems really good! I am a big fan of TensorFlow but it lacks libraries than pytorch.
", "rawMarkdown": "This seems really good! I am a big fan of TensorFlow but it lacks libraries than pytorch.", "votes": null }, { "id": "1700206", "postDate": "02/21/2022 18:31:13", "content": "+1, I think the growing ecosystem makes PyTorch a bit more fun with ease of experimentation.
", "rawMarkdown": "1, I think the growing ecosystem makes PyTorch a bit more fun with ease of experimentation.", "votes": null } ], "comments": [ { "id": 1699195, "author_name": "kalelpark", "author_url": "", "post_date": "02/21/2022 02:20:18", "content": "great!! Thanks!!!!👍👍👍👍👍
", "votes": null, "replies": [] }, { "id": 1699278, "author_name": "deepkim", "author_url": "", "post_date": "02/21/2022 04:43:04", "content": "This seems really good! I am a big fan of TensorFlow but it lacks libraries than pytorch.
", "votes": null, "replies": [ { "id": 1700206, "author_name": "init27", "author_url": "", "post_date": "02/21/2022 18:31:13", "content": "+1, I think the growing ecosystem makes PyTorch a bit more fun with ease of experimentation.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1698238": "I've seen many notebooks utilizing TPUs for Tensorflow but I was trying to explore materials for TPU training with PyTorch.\n\nI found hugging face accelerate, which allows running your code on Multi-GPUs or TPUs with just 4 lines of additional code.\n\nFrom the [docs](https://huggingface.co/docs/accelerate/):\n\n> 🤗 Accelerate provides an easy API to make your scripts run with mixed precision and on any kind of distributed setting (multi-GPUs, TPUs etc.) while still letting you write your own training loop. The same code can then runs seamlessly on your local machine for debugging or your training environment.\n\n\n> 🤗 Accelerate also provides a CLI tool that allows you to quickly configure and test your training environment then launch the scripts.\n\nSome resources: \n- [Video Walkthrough by Sylvain Gugger](https://www.youtube.com/watch?v=A7lnu-ZsFZs)\n- [Shorter video](https://www.youtube.com/watch?v=s7dy8QRgjJ0)\n- [Github](https://github.com/huggingface/accelerate)\n\nLet's bring PyTorch to TPUs 🚀 I'll be playing around with the public kernels and will keep everyone posted of my experimentations with TPUs.", "1699195": "great!! Thanks!!!!👍👍👍👍👍", "1699278": "This seems really good! I am a big fan of TensorFlow but it lacks libraries than pytorch.", "1700206": "1, I think the growing ecosystem makes PyTorch a bit more fun with ease of experimentation." }, "source": "meta" }