We have free TPU time for blazing fast model training, does anyone use the resource with pytorch? I have invested a couple of days trying to understand how to do it, but I feel I didn't get much closer.
\n\nMy understanding is that pytorch/xla must be used. But I was not able to configure it with Google Cloud Shell.
\n\nAny leads and your experience are highly appreciated, please share.
\n\nUPDATE: I wrote a short article on my experience using TPU with pytorch, please take a look.
", "messages": [ { "id": "590085", "postDate": "08/01/2019 19:46:55", "content": "We have free TPU time for blazing fast model training, does anyone use the resource with pytorch? I have invested a couple of days trying to understand how to do it, but I feel I didn't get much closer.
\n\nMy understanding is that pytorch/xla must be used. But I was not able to configure it with Google Cloud Shell.
\n\nAny leads and your experience are highly appreciated, please share.
\n\nUPDATE: I wrote a short article on my experience using TPU with pytorch, please take a look.
", "rawMarkdown": "We have free TPU time for blazing fast model training, does anyone use the resource with pytorch? I have invested a couple of days trying to understand how to do it, but I feel I didn't get much closer. \n\nMy understanding is that [pytorch/xla](https://github.com/pytorch/xla) must be used. But I was not able to configure it with Google Cloud Shell.\n\nAny leads and your experience are highly appreciated, please share.\n\nUPDATE: I wrote a short article on [my experience using TPU with pytorch](https://medium.com/@zaharchikishev/running-pytorch-on-tpu-a-bag-of-tricks-b6d0130bddd4?source=friends_link&sk=695f9605874c7306837645ff814e7fc9), please take a look.", "votes": null }, { "id": "590226", "postDate": "08/01/2019 22:38:09", "content": "Use preinstalled vertion from google cloud. I learn tf ^^ , pythorch not good with tpu
", "rawMarkdown": "Use preinstalled vertion from google cloud. I learn tf ^^ , pythorch not good with tpu", "votes": null }, { "id": "590727", "postDate": "08/02/2019 14:15:35", "content": "Just to confirm, the tpu quota is unlimited. It is not capped at the 300eur, is this correct?
", "rawMarkdown": "Just to confirm, the tpu quota is unlimited. It is not capped at the 300eur, is this correct?", "votes": null }, { "id": "590739", "postDate": "08/02/2019 14:28:27", "content": "This is my understanding as well, it is unlimited for 3 months, but you can only use one of the following options:
\n\n5 on-demand Cloud TPU v2-8 device(s) in zone us-central1-b\n20 preemptible Cloud TPU v2-8 device(s) in zone us-central1-b\n2 on-demand Cloud TPU v3-8 device(s) in zone europe-west4-a
\n\nBy the way, it is 300 usd for me, not 300 eur.
", "rawMarkdown": "This is my understanding as well, it is unlimited for 3 months, but you can only use one of the following options: \n\n5 on-demand Cloud TPU v2-8 device(s) in zone us-central1-b\n20 preemptible Cloud TPU v2-8 device(s) in zone us-central1-b\n2 on-demand Cloud TPU v3-8 device(s) in zone europe-west4-a\n\nBy the way, it is 300 usd for me, not 300 eur.", "votes": null }, { "id": "590897", "postDate": "08/02/2019 19:14:06", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": null }, { "id": "592875", "postDate": "08/05/2019 23:54:38", "content": "TPU support is very preliminary and doesn't seem ready for individuals to immediately use easily, based on what I have read. I would recommend switching to Keras/TF if you want to use TPU
", "rawMarkdown": "TPU support is very preliminary and doesn't seem ready for individuals to immediately use easily, based on what I have read. I would recommend switching to Keras/TF if you want to use TPU", "votes": null }, { "id": "592883", "postDate": "08/06/2019 00:08:31", "content": "I still decided to pick this fight, and it almost works for me now. When I will have a stable setup I will be happy to share the instructions! Can be interesting to compare the speeds.
", "rawMarkdown": "I still decided to pick this fight, and it almost works for me now. When I will have a stable setup I will be happy to share the instructions! Can be interesting to compare the speeds.", "votes": null }, { "id": "592890", "postDate": "08/06/2019 00:23:27", "content": "Sounds good. If you do get it working, please let me know!
\n\nIt would definitely be helpful to use PyTorch for TPUs but it seemed like it was very preliminary, and the instructions weren't entirely clear. So if you get it working, I think the community would greatly appreciate some well laid-out instructions on using Pytorch-XLA.
", "rawMarkdown": "Sounds good. If you do get it working, please let me know! \n\nIt would definitely be helpful to use PyTorch for TPUs but it seemed like it was very preliminary, and the instructions weren't entirely clear. So if you get it working, I think the community would greatly appreciate some well laid-out instructions on using Pytorch-XLA.", "votes": null }, { "id": "592908", "postDate": "08/06/2019 01:04:50", "content": "Hi everyone - I'm one of the leaders of Google's Cloud TPU team! We are very excited about enabling additional ML frameworks and tools to benefit from Cloud TPU acceleration. The PyTorch / XLA integration is still experimental, so we can't officially support it yet, but my understanding is that if you follow the instructions in the \"Consume Prebuilt Docker Images\" section at https://github.com/pytorch/xla exactly, it should be possible to train simple PyTorch computer vision models on Cloud TPUs successfully.
\n\nI don't know how challenging it might be to port the Recursion sample code to PyTorch or whether that code is likely to run smoothly via the current PyTorch / XLA integration, but I would love to hear how things go if any of you decide to explore this path further. If you successfully get a baseline PyTorch model up and running on the Recursion dataset and are willing to publish your code with a permissive open-source license such as Apache 2.0 or MIT as a starting point for others, that would be amazing. Please also make sure we see it by emailing the link to pytorch-tpu at googlegroups.com.
\n\nBest of luck with the competition!
", "rawMarkdown": "Hi everyone - I'm one of the leaders of Google's Cloud TPU team! We are very excited about enabling additional ML frameworks and tools to benefit from Cloud TPU acceleration. The PyTorch / XLA integration is still experimental, so we can't officially support it yet, but my understanding is that if you follow the instructions in the \"Consume Prebuilt Docker Images\" section at https://github.com/pytorch/xla exactly, it should be possible to train simple PyTorch computer vision models on Cloud TPUs successfully.\n\nI don't know how challenging it might be to port the Recursion sample code to PyTorch or whether that code is likely to run smoothly via the current PyTorch / XLA integration, but I would love to hear how things go if any of you decide to explore this path further. If you successfully get a baseline PyTorch model up and running on the Recursion dataset and are willing to publish your code with a permissive open-source license such as Apache 2.0 or MIT as a starting point for others, that would be amazing. Please also make sure we see it by emailing the link to pytorch-tpu at googlegroups.com.\n\nBest of luck with the competition!", "votes": null }, { "id": "592910", "postDate": "08/06/2019 01:07:07", "content": "We would love to see your instructions if you get this to work! Please send a link to pytorch-tpu at googlegroups.com. I admire your courage. = )
\n\nI also just posted another comment with more information about the PyTorch / XLA integration.
", "rawMarkdown": "We would love to see your instructions if you get this to work! Please send a link to pytorch-tpu at googlegroups.com. I admire your courage. = )\n\nI also just posted another comment with more information about the PyTorch / XLA integration.", "votes": null }, { "id": "592916", "postDate": "08/06/2019 01:13:40", "content": "I should also note that Keras support for TPU is not fully ready; we still recommend using TPUEstimator with TF 1.14.
", "rawMarkdown": "I should also note that Keras support for TPU is not fully ready; we still recommend using TPUEstimator with TF 1.14.", "votes": null }, { "id": "597413", "postDate": "08/12/2019 10:00:02", "content": "For similar environments I would be very surprised if TF is faster than PyTorch from speed and model performance pov.
", "rawMarkdown": "For similar environments I would be very surprised if TF is faster than PyTorch from speed and model performance pov.", "votes": null }, { "id": "603445", "postDate": "08/20/2019 09:19:58", "content": "Any updates?
", "rawMarkdown": "Any updates?", "votes": null }, { "id": "603469", "postDate": "08/20/2019 10:05:24", "content": "Hi @christofhenkel , I am training my model for this competition on GCP TPU with torch/xla, after some trial & error I found a stable configuration. I will write about some tricks I learnt later today-tomorrow. What is your experience?
", "rawMarkdown": "Hi @christofhenkel , I am training my model for this competition on GCP TPU with torch/xla, after some trial & error I found a stable configuration. I will write about some tricks I learnt later today-tomorrow. What is your experience?", "votes": null }, { "id": "603483", "postDate": "08/20/2019 10:40:25", "content": "I was able to set everything up, (instance, tpu node etc)\nI tried some things but failed implementing a simple model. Even the mnist test case provided on xla GitHub did not work correctly, so I gave up some time ago and focused on another competition. But now I would like to try again.
", "rawMarkdown": "I was able to set everything up, (instance, tpu node etc)\nI tried some things but failed implementing a simple model. Even the mnist test case provided on xla GitHub did not work correctly, so I gave up some time ago and focused on another competition. But now I would like to try again.", "votes": null }, { "id": "603819", "postDate": "08/20/2019 17:34:07", "content": "@zaharch I'm really interested in seeing this as well. I am curious to try TPUs sometime and would be interested in summary of your experience. Also curious if you were able to realize a performance speedup when using TPUs.
", "rawMarkdown": "zaharch I'm really interested in seeing this as well. I am curious to try TPUs sometime and would be interested in summary of your experience. Also curious if you were able to realize a performance speedup when using TPUs.", "votes": null }, { "id": "606050", "postDate": "08/23/2019 06:11:29", "content": "Really enjoyed reading this post! We will keep improving the user experience:\nhttps://medium.com/@zaharchikishev/running-pytorch-on-tpu-a-bag-of-tricks-b6d0130bddd4
", "rawMarkdown": "Really enjoyed reading this post! We will keep improving the user experience:\nhttps://medium.com/@zaharchikishev/running-pytorch-on-tpu-a-bag-of-tricks-b6d0130bddd4", "votes": null }, { "id": "606262", "postDate": "08/23/2019 11:30:13", "content": "This is a summary of me working with TPU on pytorch so far. There is a performance comparison at the end, with images per second in training loop as a metric. Very interesting what speed others are getting, please share this number for comparison if you can.
", "rawMarkdown": "[This is a summary](https://medium.com/@zaharchikishev/running-pytorch-on-tpu-a-bag-of-tricks-b6d0130bddd4?source=friends_link&sk=695f9605874c7306837645ff814e7fc9) of me working with TPU on pytorch so far. There is a performance comparison at the end, with images per second in training loop as a metric. Very interesting what speed others are getting, please share this number for comparison if you can.", "votes": null }, { "id": "606272", "postDate": "08/23/2019 11:38:01", "content": "thank you
", "rawMarkdown": "thank you", "votes": null }, { "id": "606489", "postDate": "08/23/2019 16:55:06", "content": "Nice! Thanks for sharing.
", "rawMarkdown": "Nice! Thanks for sharing.", "votes": null }, { "id": "617706", "postDate": "09/04/2019 12:02:14", "content": "I was excited about the TPU quota but then was not able to work on this problem till this week. I have had some success (more like beginner's luck) training on TPU, but using the github repo provided by recursion. So I have used tensorflow, not pytorch. The repo (almost as is) training got me to a LB score of 70%, though I did do some post processing using this trick from the double strand team.
\n\nThe 70% LB score was based on training with image recognition only. No statistical features have been included yet but I will do this as soon as I can work out how to in TF. It doesn't seem that straightforward.
\n\nTPUs are very fast. It processed about 1000 examples per second of [512,512,6] images, so each epoch took just over a minute. That was using a single TPU v3-8 on google cloud.
", "rawMarkdown": "I was excited about the TPU quota but then was not able to work on this problem till this week. I have had some success (more like beginner's luck) training on TPU, but using the github repo provided by recursion. So I have used tensorflow, not pytorch. The repo (almost as is) training got me to a LB score of 70%, though I did do some post processing using [this trick](https://www.kaggle.com/zaharch/keras-model-boosted-with-plates-leak) from the double strand team.\n\nThe 70% LB score was based on training with image recognition only. No statistical features have been included yet but I will do this as soon as I can work out how to in TF. It doesn't seem that straightforward.\n\nTPUs are very fast. It processed about 1000 examples per second of [512,512,6] images, so each epoch took just over a minute. That was using a single TPU v3-8 on google cloud.", "votes": null }, { "id": "617963", "postDate": "09/04/2019 17:04:23", "content": "Thank you for your post!\nIt seems like the highest image resolution improves the LB score significantly.\nI am currently using 300x300 but it seems I need to step up (and use gradient accumulation).\nI wonder if this is the cause why a lot of people seem to struggle to achieve 0.50 on the LB?
", "rawMarkdown": "Thank you for your post!\nIt seems like the highest image resolution improves the LB score significantly.\nI am currently using 300x300 but it seems I need to step up (and use gradient accumulation).\nI wonder if this is the cause why a lot of people seem to struggle to achieve 0.50 on the LB?", "votes": null } ], "comments": [ { "id": 590226, "author_name": "leighplt", "author_url": "", "post_date": "08/01/2019 22:38:09", "content": "Use preinstalled vertion from google cloud. I learn tf ^^ , pythorch not good with tpu
", "votes": null, "replies": [] }, { "id": 590727, "author_name": "darraghdog", "author_url": "", "post_date": "08/02/2019 14:15:35", "content": "Just to confirm, the tpu quota is unlimited. It is not capped at the 300eur, is this correct?
", "votes": null, "replies": [ { "id": 590739, "author_name": "zaharch", "author_url": "", "post_date": "08/02/2019 14:28:27", "content": "This is my understanding as well, it is unlimited for 3 months, but you can only use one of the following options:
\n\n5 on-demand Cloud TPU v2-8 device(s) in zone us-central1-b\n20 preemptible Cloud TPU v2-8 device(s) in zone us-central1-b\n2 on-demand Cloud TPU v3-8 device(s) in zone europe-west4-a
\n\nBy the way, it is 300 usd for me, not 300 eur.
", "votes": null, "replies": [] }, { "id": 590897, "author_name": "darraghdog", "author_url": "", "post_date": "08/02/2019 19:14:06", "content": "Thanks!
", "votes": null, "replies": [] } ] }, { "id": 592875, "author_name": "tanlikesmath", "author_url": "", "post_date": "08/05/2019 23:54:38", "content": "TPU support is very preliminary and doesn't seem ready for individuals to immediately use easily, based on what I have read. I would recommend switching to Keras/TF if you want to use TPU
", "votes": null, "replies": [ { "id": 592883, "author_name": "zaharch", "author_url": "", "post_date": "08/06/2019 00:08:31", "content": "I still decided to pick this fight, and it almost works for me now. When I will have a stable setup I will be happy to share the instructions! Can be interesting to compare the speeds.
", "votes": null, "replies": [] }, { "id": 592890, "author_name": "tanlikesmath", "author_url": "", "post_date": "08/06/2019 00:23:27", "content": "Sounds good. If you do get it working, please let me know!
\n\nIt would definitely be helpful to use PyTorch for TPUs but it seemed like it was very preliminary, and the instructions weren't entirely clear. So if you get it working, I think the community would greatly appreciate some well laid-out instructions on using Pytorch-XLA.
", "votes": null, "replies": [] }, { "id": 592910, "author_name": "zakstone", "author_url": "", "post_date": "08/06/2019 01:07:07", "content": "We would love to see your instructions if you get this to work! Please send a link to pytorch-tpu at googlegroups.com. I admire your courage. = )
\n\nI also just posted another comment with more information about the PyTorch / XLA integration.
", "votes": null, "replies": [] }, { "id": 592916, "author_name": "zakstone", "author_url": "", "post_date": "08/06/2019 01:13:40", "content": "I should also note that Keras support for TPU is not fully ready; we still recommend using TPUEstimator with TF 1.14.
", "votes": null, "replies": [] }, { "id": 597413, "author_name": "cyberia", "author_url": "", "post_date": "08/12/2019 10:00:02", "content": "For similar environments I would be very surprised if TF is faster than PyTorch from speed and model performance pov.
", "votes": null, "replies": [] } ] }, { "id": 592908, "author_name": "zakstone", "author_url": "", "post_date": "08/06/2019 01:04:50", "content": "Hi everyone - I'm one of the leaders of Google's Cloud TPU team! We are very excited about enabling additional ML frameworks and tools to benefit from Cloud TPU acceleration. The PyTorch / XLA integration is still experimental, so we can't officially support it yet, but my understanding is that if you follow the instructions in the \"Consume Prebuilt Docker Images\" section at https://github.com/pytorch/xla exactly, it should be possible to train simple PyTorch computer vision models on Cloud TPUs successfully.
\n\nI don't know how challenging it might be to port the Recursion sample code to PyTorch or whether that code is likely to run smoothly via the current PyTorch / XLA integration, but I would love to hear how things go if any of you decide to explore this path further. If you successfully get a baseline PyTorch model up and running on the Recursion dataset and are willing to publish your code with a permissive open-source license such as Apache 2.0 or MIT as a starting point for others, that would be amazing. Please also make sure we see it by emailing the link to pytorch-tpu at googlegroups.com.
\n\nBest of luck with the competition!
", "votes": null, "replies": [] }, { "id": 603445, "author_name": "christofhenkel", "author_url": "", "post_date": "08/20/2019 09:19:58", "content": "Any updates?
", "votes": null, "replies": [ { "id": 603469, "author_name": "zaharch", "author_url": "", "post_date": "08/20/2019 10:05:24", "content": "Hi @christofhenkel , I am training my model for this competition on GCP TPU with torch/xla, after some trial & error I found a stable configuration. I will write about some tricks I learnt later today-tomorrow. What is your experience?
", "votes": null, "replies": [] }, { "id": 603483, "author_name": "christofhenkel", "author_url": "", "post_date": "08/20/2019 10:40:25", "content": "I was able to set everything up, (instance, tpu node etc)\nI tried some things but failed implementing a simple model. Even the mnist test case provided on xla GitHub did not work correctly, so I gave up some time ago and focused on another competition. But now I would like to try again.
", "votes": null, "replies": [] }, { "id": 603819, "author_name": "antgoldbloom", "author_url": "", "post_date": "08/20/2019 17:34:07", "content": "@zaharch I'm really interested in seeing this as well. I am curious to try TPUs sometime and would be interested in summary of your experience. Also curious if you were able to realize a performance speedup when using TPUs.
", "votes": null, "replies": [] }, { "id": 606050, "author_name": "zakstone", "author_url": "", "post_date": "08/23/2019 06:11:29", "content": "Really enjoyed reading this post! We will keep improving the user experience:\nhttps://medium.com/@zaharchikishev/running-pytorch-on-tpu-a-bag-of-tricks-b6d0130bddd4
", "votes": null, "replies": [] } ] }, { "id": 606262, "author_name": "zaharch", "author_url": "", "post_date": "08/23/2019 11:30:13", "content": "This is a summary of me working with TPU on pytorch so far. There is a performance comparison at the end, with images per second in training loop as a metric. Very interesting what speed others are getting, please share this number for comparison if you can.
", "votes": null, "replies": [ { "id": 606272, "author_name": "christofhenkel", "author_url": "", "post_date": "08/23/2019 11:38:01", "content": "thank you
", "votes": null, "replies": [] }, { "id": 606489, "author_name": "antgoldbloom", "author_url": "", "post_date": "08/23/2019 16:55:06", "content": "Nice! Thanks for sharing.
", "votes": null, "replies": [] } ] }, { "id": 617706, "author_name": "kenkrige", "author_url": "", "post_date": "09/04/2019 12:02:14", "content": "I was excited about the TPU quota but then was not able to work on this problem till this week. I have had some success (more like beginner's luck) training on TPU, but using the github repo provided by recursion. So I have used tensorflow, not pytorch. The repo (almost as is) training got me to a LB score of 70%, though I did do some post processing using this trick from the double strand team.
\n\nThe 70% LB score was based on training with image recognition only. No statistical features have been included yet but I will do this as soon as I can work out how to in TF. It doesn't seem that straightforward.
\n\nTPUs are very fast. It processed about 1000 examples per second of [512,512,6] images, so each epoch took just over a minute. That was using a single TPU v3-8 on google cloud.
", "votes": null, "replies": [ { "id": 617963, "author_name": "micpie", "author_url": "", "post_date": "09/04/2019 17:04:23", "content": "Thank you for your post!\nIt seems like the highest image resolution improves the LB score significantly.\nI am currently using 300x300 but it seems I need to step up (and use gradient accumulation).\nI wonder if this is the cause why a lot of people seem to struggle to achieve 0.50 on the LB?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "590085": "We have free TPU time for blazing fast model training, does anyone use the resource with pytorch? I have invested a couple of days trying to understand how to do it, but I feel I didn't get much closer. \n\nMy understanding is that [pytorch/xla](https://github.com/pytorch/xla) must be used. But I was not able to configure it with Google Cloud Shell.\n\nAny leads and your experience are highly appreciated, please share.\n\nUPDATE: I wrote a short article on [my experience using TPU with pytorch](https://medium.com/@zaharchikishev/running-pytorch-on-tpu-a-bag-of-tricks-b6d0130bddd4?source=friends_link&sk=695f9605874c7306837645ff814e7fc9), please take a look.", "590226": "Use preinstalled vertion from google cloud. I learn tf ^^ , pythorch not good with tpu", "590727": "Just to confirm, the tpu quota is unlimited. It is not capped at the 300eur, is this correct?", "590739": "This is my understanding as well, it is unlimited for 3 months, but you can only use one of the following options: \n\n5 on-demand Cloud TPU v2-8 device(s) in zone us-central1-b\n20 preemptible Cloud TPU v2-8 device(s) in zone us-central1-b\n2 on-demand Cloud TPU v3-8 device(s) in zone europe-west4-a\n\nBy the way, it is 300 usd for me, not 300 eur.", "590897": "Thanks!", "592875": "TPU support is very preliminary and doesn't seem ready for individuals to immediately use easily, based on what I have read. I would recommend switching to Keras/TF if you want to use TPU", "592883": "I still decided to pick this fight, and it almost works for me now. When I will have a stable setup I will be happy to share the instructions! Can be interesting to compare the speeds.", "592890": "Sounds good. If you do get it working, please let me know! \n\nIt would definitely be helpful to use PyTorch for TPUs but it seemed like it was very preliminary, and the instructions weren't entirely clear. So if you get it working, I think the community would greatly appreciate some well laid-out instructions on using Pytorch-XLA.", "592908": "Hi everyone - I'm one of the leaders of Google's Cloud TPU team! We are very excited about enabling additional ML frameworks and tools to benefit from Cloud TPU acceleration. The PyTorch / XLA integration is still experimental, so we can't officially support it yet, but my understanding is that if you follow the instructions in the \"Consume Prebuilt Docker Images\" section at https://github.com/pytorch/xla exactly, it should be possible to train simple PyTorch computer vision models on Cloud TPUs successfully.\n\nI don't know how challenging it might be to port the Recursion sample code to PyTorch or whether that code is likely to run smoothly via the current PyTorch / XLA integration, but I would love to hear how things go if any of you decide to explore this path further. If you successfully get a baseline PyTorch model up and running on the Recursion dataset and are willing to publish your code with a permissive open-source license such as Apache 2.0 or MIT as a starting point for others, that would be amazing. Please also make sure we see it by emailing the link to pytorch-tpu at googlegroups.com.\n\nBest of luck with the competition!", "592910": "We would love to see your instructions if you get this to work! Please send a link to pytorch-tpu at googlegroups.com. I admire your courage. = )\n\nI also just posted another comment with more information about the PyTorch / XLA integration.", "592916": "I should also note that Keras support for TPU is not fully ready; we still recommend using TPUEstimator with TF 1.14.", "597413": "For similar environments I would be very surprised if TF is faster than PyTorch from speed and model performance pov.", "603445": "Any updates?", "603469": "Hi @christofhenkel , I am training my model for this competition on GCP TPU with torch/xla, after some trial & error I found a stable configuration. I will write about some tricks I learnt later today-tomorrow. What is your experience?", "603483": "I was able to set everything up, (instance, tpu node etc)\nI tried some things but failed implementing a simple model. Even the mnist test case provided on xla GitHub did not work correctly, so I gave up some time ago and focused on another competition. But now I would like to try again.", "603819": "zaharch I'm really interested in seeing this as well. I am curious to try TPUs sometime and would be interested in summary of your experience. Also curious if you were able to realize a performance speedup when using TPUs.", "606050": "Really enjoyed reading this post! We will keep improving the user experience:\nhttps://medium.com/@zaharchikishev/running-pytorch-on-tpu-a-bag-of-tricks-b6d0130bddd4", "606262": "[This is a summary](https://medium.com/@zaharchikishev/running-pytorch-on-tpu-a-bag-of-tricks-b6d0130bddd4?source=friends_link&sk=695f9605874c7306837645ff814e7fc9) of me working with TPU on pytorch so far. There is a performance comparison at the end, with images per second in training loop as a metric. Very interesting what speed others are getting, please share this number for comparison if you can.", "606272": "thank you", "606489": "Nice! Thanks for sharing.", "617706": "I was excited about the TPU quota but then was not able to work on this problem till this week. I have had some success (more like beginner's luck) training on TPU, but using the github repo provided by recursion. So I have used tensorflow, not pytorch. The repo (almost as is) training got me to a LB score of 70%, though I did do some post processing using [this trick](https://www.kaggle.com/zaharch/keras-model-boosted-with-plates-leak) from the double strand team.\n\nThe 70% LB score was based on training with image recognition only. No statistical features have been included yet but I will do this as soon as I can work out how to in TF. It doesn't seem that straightforward.\n\nTPUs are very fast. It processed about 1000 examples per second of [512,512,6] images, so each epoch took just over a minute. That was using a single TPU v3-8 on google cloud.", "617963": "Thank you for your post!\nIt seems like the highest image resolution improves the LB score significantly.\nI am currently using 300x300 but it seems I need to step up (and use gradient accumulation).\nI wonder if this is the cause why a lot of people seem to struggle to achieve 0.50 on the LB?" }, "source": "meta" }