Hello, everyone. I don’t know if you have ever encountered such a situation. In kaggle environment, the tpu code written by tensorflow can run perfectly, but the tpu code written by pytorch xla with the same configuration (batch size, model, learning rate, input, etc.) will raise Out of Memory error or model gradient explosion.
My pytorch tpu code has meeted above phenomenon, because keeping the same settings with tf tpu implementation will cause Out of Memory error or model gradient explosion, so the configuration of my version 3 pytorch implementation is not completely implemented in accordance with the tf code.
I don’t think there is an error in my pytorch xla implementation, so is the above phenomenon a normal phenomenon?
", "messages": [ { "id": "1380298", "postDate": "07/08/2021 02:19:06", "content": "Hello, everyone. I don’t know if you have ever encountered such a situation. In kaggle environment, the tpu code written by tensorflow can run perfectly, but the tpu code written by pytorch xla with the same configuration (batch size, model, learning rate, input, etc.) will raise Out of Memory error or model gradient explosion.
My pytorch tpu code has meeted above phenomenon, because keeping the same settings with tf tpu implementation will cause Out of Memory error or model gradient explosion, so the configuration of my version 3 pytorch implementation is not completely implemented in accordance with the tf code.
I don’t think there is an error in my pytorch xla implementation, so is the above phenomenon a normal phenomenon?
", "rawMarkdown": "Hello, everyone. I don’t know if you have ever encountered such a situation. In kaggle environment, the tpu code written by tensorflow can run perfectly, but the tpu code written by pytorch xla with the same configuration (batch size, model, learning rate, input, etc.) will raise `Out of Memory` error or `model gradient explosion`.\n\n[My pytorch tpu code](https://www.kaggle.com/shangweichen/siim-efnb7-train-pytorch-xla-tpu/comments) has meeted above phenomenon, because keeping the same settings with [tf tpu implementation](https://www.kaggle.com/h053473666/siim-covid19-efnb7-train-study/data) will cause `Out of Memory` error or `model gradient explosion`, so the configuration of [my version 3 pytorch implementation](https://www.kaggle.com/shangweichen/siim-efnb7-train-pytorch-xla-tpu/comments) is not completely implemented in accordance with the [tf code](https://www.kaggle.com/h053473666/siim-covid19-efnb7-train-study/data).\n\n- If the image size in my version3 implementation becomes 600x600(version3: 400x400), learning rate becomes 1e-3/8(version3: 1e-4/8), and batch size becomes 128(version3: 64), to be consistent with above tf implementation, there will be an oom error which doesn't occur in above tf implementation. (Note: 8 is the num of tpu cores, final learning rate need to multiply with num of tpu cores, so final lr is equal to 1e-3/8*8)\n- If I only change the learning rate in my version3 implementation to 1e-3/8(version3: 1e-4/8), there will be a gradient explosion which doesn't occur in above tf implementation.\n\nI don’t think there is an error in my pytorch xla implementation, so is the above phenomenon a normal phenomenon?", "votes": null }, { "id": "1400339", "postDate": "07/26/2021 08:12:40", "content": "Hi, have you found a solution to this? Is the fix to try different learning rates?
", "rawMarkdown": "Hi, have you found a solution to this? Is the fix to try different learning rates?", "votes": null }, { "id": "1400383", "postDate": "07/26/2021 08:43:21", "content": "I have not yet find a solution to this, may be this is the implementation diff between pytorch and tensorflow.
", "rawMarkdown": "I have not yet find a solution to this, may be this is the implementation diff between pytorch and tensorflow.", "votes": null }, { "id": "1400542", "postDate": "07/26/2021 11:32:31", "content": "Tensorflow TPU setup
\ngradiant accumulate
\nhttps://www.kaggle.com/dschettler8845/3-71-cv-bms-efficientnetv2-transformer-e2e
tpu augment
\nmixup: https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu
\nrotation: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96
Thank you for your reply!
", "rawMarkdown": "Thank you for your reply!", "votes": null }, { "id": "1402245", "postDate": "07/28/2021 03:32:44", "content": "I have the same problem. Tensorflow + TPU works much better than PyTorch + TPU. If you want to use TPU with PyTorch, try training with a lower batch size. You can also use PyTorch lightning to write hardware-agnostic code and try comparing gpu and tpu performance
", "rawMarkdown": "I have the same problem. Tensorflow + TPU works much better than PyTorch + TPU. If you want to use TPU with PyTorch, try training with a lower batch size. You can also use PyTorch lightning to write hardware-agnostic code and try comparing gpu and tpu performance", "votes": null }, { "id": "1402253", "postDate": "07/28/2021 03:41:55", "content": "I'll give it a try, thank you for your suggestion~
", "rawMarkdown": "I'll give it a try, thank you for your suggestion~", "votes": null } ], "comments": [ { "id": 1400339, "author_name": "novice03", "author_url": "", "post_date": "07/26/2021 08:12:40", "content": "Hi, have you found a solution to this? Is the fix to try different learning rates?
", "votes": null, "replies": [ { "id": 1400383, "author_name": "shangweichen", "author_url": "", "post_date": "07/26/2021 08:43:21", "content": "I have not yet find a solution to this, may be this is the implementation diff between pytorch and tensorflow.
", "votes": null, "replies": [] } ] }, { "id": 1400542, "author_name": "drzhuzhe", "author_url": "", "post_date": "07/26/2021 11:32:31", "content": "Tensorflow TPU setup
\ngradiant accumulate
\nhttps://www.kaggle.com/dschettler8845/3-71-cv-bms-efficientnetv2-transformer-e2e
tpu augment
\nmixup: https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu
\nrotation: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96
Thank you for your reply!
", "votes": null, "replies": [] } ] }, { "id": 1402245, "author_name": "readoc", "author_url": "", "post_date": "07/28/2021 03:32:44", "content": "I have the same problem. Tensorflow + TPU works much better than PyTorch + TPU. If you want to use TPU with PyTorch, try training with a lower batch size. You can also use PyTorch lightning to write hardware-agnostic code and try comparing gpu and tpu performance
", "votes": null, "replies": [ { "id": 1402253, "author_name": "shangweichen", "author_url": "", "post_date": "07/28/2021 03:41:55", "content": "I'll give it a try, thank you for your suggestion~
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1380298": "Hello, everyone. I don’t know if you have ever encountered such a situation. In kaggle environment, the tpu code written by tensorflow can run perfectly, but the tpu code written by pytorch xla with the same configuration (batch size, model, learning rate, input, etc.) will raise `Out of Memory` error or `model gradient explosion`.\n\n[My pytorch tpu code](https://www.kaggle.com/shangweichen/siim-efnb7-train-pytorch-xla-tpu/comments) has meeted above phenomenon, because keeping the same settings with [tf tpu implementation](https://www.kaggle.com/h053473666/siim-covid19-efnb7-train-study/data) will cause `Out of Memory` error or `model gradient explosion`, so the configuration of [my version 3 pytorch implementation](https://www.kaggle.com/shangweichen/siim-efnb7-train-pytorch-xla-tpu/comments) is not completely implemented in accordance with the [tf code](https://www.kaggle.com/h053473666/siim-covid19-efnb7-train-study/data).\n\n- If the image size in my version3 implementation becomes 600x600(version3: 400x400), learning rate becomes 1e-3/8(version3: 1e-4/8), and batch size becomes 128(version3: 64), to be consistent with above tf implementation, there will be an oom error which doesn't occur in above tf implementation. (Note: 8 is the num of tpu cores, final learning rate need to multiply with num of tpu cores, so final lr is equal to 1e-3/8*8)\n- If I only change the learning rate in my version3 implementation to 1e-3/8(version3: 1e-4/8), there will be a gradient explosion which doesn't occur in above tf implementation.\n\nI don’t think there is an error in my pytorch xla implementation, so is the above phenomenon a normal phenomenon?", "1400339": "Hi, have you found a solution to this? Is the fix to try different learning rates?", "1400383": "I have not yet find a solution to this, may be this is the implementation diff between pytorch and tensorflow.", "1400542": "Tensorflow TPU setup\n\n1. gradiant accumulate\nhttps://www.kaggle.com/dschettler8845/3-71-cv-bms-efficientnetv2-transformer-e2e\n\n2. tpu augment\nmixup: https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu\nrotation: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96", "1400792": "Thank you for your reply!", "1402245": "I have the same problem. Tensorflow + TPU works much better than PyTorch + TPU. If you want to use TPU with PyTorch, try training with a lower batch size. You can also use PyTorch lightning to write hardware-agnostic code and try comparing gpu and tpu performance", "1402253": "I'll give it a try, thank you for your suggestion~" }, "source": "meta" }