Hi everyone between the previous two TPU competitions (Flowers and Jigsaw) I have learned a lot and done many experiments, thanks to all experts and the community. So here I would like to share some tips so that TPU beginners can have an easier time using them.
\n\nbatch size large to take more advantage of the TPU resources, but keep in mind that you may need to adjust your learning rate.Tensorflow try to use tf.data to load and manipulate your data, it will make things faster!flip images with tf.data API with tf.image.random_flip_left_right(image).tf.tpu.experimental.initialize_tpu_system(tpu) to clear TPU memory before loading the next model here is an example.![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4003597%2Ff70fc3744bd57681ff109f3f135ed323%2FTPU%20idle%20time2.png?generation=1583185609964304&alt=media\")
Your insights are really valuable @dimitreoliveira
", "rawMarkdown": "Your insights are really valuable @dimitreoliveira ", "votes": 3 }, { "id": 892529, "postDate": "2020-06-19T01:39:04.587Z", "content": "Hi everyone between the previous two TPU competitions (Flowers and Jigsaw) I have learned a lot and done many experiments, thanks to all experts and the community. So here I would like to share some tips so that TPU beginners can have an easier time using them.
\n\nbatch size large to take more advantage of the TPU resources, but keep in mind that you may need to adjust your learning rate.Tensorflow try to use tf.data to load and manipulate your data, it will make things faster!flip images with tf.data API with tf.image.random_flip_left_right(image).tf.tpu.experimental.initialize_tpu_system(tpu) to clear TPU memory before loading the next model here is an example.Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": 1, "replies": [ { "id": 1261318, "postDate": "2021-04-02T23:52:41.683Z", "content": "You're welcome @saukha
", "rawMarkdown": "You're welcome @saukha " } ] }, { "id": 2244238, "postDate": "2023-05-03T14:27:42.737Z", "content": "Thank you very much for sharing, it helped me to apply in other competitions.
", "rawMarkdown": "Thank you very much for sharing, it helped me to apply in other competitions." }, { "id": 2080990, "postDate": "2022-12-30T17:42:48.263Z", "content": "@dimitreoliveira Hii,
\nI have been following notebooks on Kaggle to train models on TPU. This and This where the former notebook shows to make TF Records and the latter one to train models on TPU with these TF Records.
\nI am strictly following these two notebooks but still encountering with strange error. I am using Skin Cancer HAM10000 dataset from Kaggle.
\nThe error looks like this -
OutOfRangeError: root error(s) found.\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_2}}]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[tpu_compile_succeeded_assert/_1935868936045555324/_5/_143]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[TPUReplicate/_/_17327685160909203060/_4/_108]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[Pad_7/_86]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[Pad_15/_150]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[strided_slice_30/_120]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[TPUReplicate/_/_17327685160909203060/_4/_44]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[IteratorGetNext_5/_24]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n successful operations.\n derived errors ignored.\nI have been searching over the internet for these issues and changing some Hyperparameters but could not able to resolve it. My notebook is here
\nPlease have a look into it and I would be grateful to get a reply back.
\nThank you.
I found the solution . Here it is stated.
", "rawMarkdown": "I found the solution . [Here ](https://www.kaggle.com/discussions/questions-and-answers/375165)it is stated. " } ] }, { "id": 1261416, "postDate": "2021-04-03T04:19:06.137Z", "content": "After your TPU time runs out, how do you figure out when the30 hours will be available again?
", "rawMarkdown": "After your TPU time runs out, how do you figure out when the30 hours will be available again?", "replies": [ { "id": 1261779, "postDate": "2021-04-03T12:35:35.937Z", "content": "@saukha The TPU and GPU quota resets every Friday.
", "rawMarkdown": "@saukha The TPU and GPU quota resets every Friday.", "votes": 1 } ] }, { "id": 1268579, "postDate": "2021-04-09T14:53:43.800Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1649577, "postDate": "2022-01-14T12:01:59.947Z", "content": "This is very useful, thank you!
", "rawMarkdown": "This is very useful, thank you!", "votes": 1 }, { "id": 1345797, "postDate": "2021-06-11T21:28:05.347Z", "content": "Wow! been looking for this. Thank you.
", "rawMarkdown": "Wow! been looking for this. Thank you.", "votes": 1 } ], "comments": [ { "id": 2185615, "author_name": "Yeakub Sadlil", "author_url": "", "post_date": "2023-03-17T07:33:29.760000", "content": "Your insights are really valuable @dimitreoliveira
", "votes": 3, "replies": [] }, { "id": 1260116, "author_name": "Sau Kha", "author_url": "", "post_date": "2021-04-01T21:39:25.307000", "content": "Thanks for sharing!
", "votes": 1, "replies": [ { "id": 1261318, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2021-04-02T23:52:41.683000", "content": "You're welcome @saukha
", "votes": 0, "replies": [] } ] }, { "id": 2244238, "author_name": "Mario Herlein", "author_url": "", "post_date": "2023-05-03T14:27:42.737000", "content": "Thank you very much for sharing, it helped me to apply in other competitions.
", "votes": 0, "replies": [] }, { "id": 2080990, "author_name": "Nishchay Singh", "author_url": "", "post_date": "2022-12-30T17:42:48.263000", "content": "@dimitreoliveira Hii,
\nI have been following notebooks on Kaggle to train models on TPU. This and This where the former notebook shows to make TF Records and the latter one to train models on TPU with these TF Records.
\nI am strictly following these two notebooks but still encountering with strange error. I am using Skin Cancer HAM10000 dataset from Kaggle.
\nThe error looks like this -
OutOfRangeError: root error(s) found.\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_2}}]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[tpu_compile_succeeded_assert/_1935868936045555324/_5/_143]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[TPUReplicate/_/_17327685160909203060/_4/_108]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[Pad_7/_86]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[Pad_15/_150]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[strided_slice_30/_120]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[TPUReplicate/_/_17327685160909203060/_4/_44]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[IteratorGetNext_5/_24]]\n () Out of : {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n successful operations.\n derived errors ignored.\nI have been searching over the internet for these issues and changing some Hyperparameters but could not able to resolve it. My notebook is here
\nPlease have a look into it and I would be grateful to get a reply back.
\nThank you.
I found the solution . Here it is stated.
", "votes": 0, "replies": [] } ] }, { "id": 1261416, "author_name": "Sau Kha", "author_url": "", "post_date": "2021-04-03T04:19:06.137000", "content": "After your TPU time runs out, how do you figure out when the30 hours will be available again?
", "votes": 0, "replies": [ { "id": 1261779, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2021-04-03T12:35:35.937000", "content": "@saukha The TPU and GPU quota resets every Friday.
", "votes": 1, "replies": [] } ] }, { "id": 1268579, "author_name": "", "author_url": "", "post_date": "2021-04-09T14:53:43.800000", "content": "", "votes": 1, "replies": [] }, { "id": 1649577, "author_name": "Henryruddick", "author_url": "", "post_date": "2022-01-14T12:01:59.947000", "content": "This is very useful, thank you!
", "votes": 1, "replies": [] }, { "id": 1345797, "author_name": "Lawrence Jay Young", "author_url": "", "post_date": "2021-06-11T21:28:05.347000", "content": "Wow! been looking for this. Thank you.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2185615": "Your insights are really valuable @dimitreoliveira ", "892529": "Hi everyone between the previous two TPU competitions ([Flowers](https://www.kaggle.com/c/flower-classification-with-tpus) and [Jigsaw](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification)) I have learned a lot and done many experiments, thanks to all experts and the community. So here I would like to share some tips so that TPU beginners can have an easier time using them.\n\n### Some tips\n- With TPU your labels need to be of the same type (in case you are using more than 1 data source).\n- Beware of memory issues, I got many of them and some are not easy to spot just looking at error messages.\n- Keep your `batch size` large to take more advantage of the TPU resources, but keep in mind that you may need to adjust your `learning rate`.\n- If you are using `Tensorflow` try to use `tf.data` to load and manipulate your data, it will make things faster!\n- Try to make any data manipulation inside the TPU whenever you can, or at least accelerate your CPU data processing, for example, if you are doing data augmentation you can `flip` images with `tf.data` API with `tf.image.random_flip_left_right(image)`.\n- If you are training more than one model on the same kernel make sure to use `tf.tpu.experimental.initialize_tpu_system(tpu)` to clear TPU memory before loading the next model [here is an example](https://www.kaggle.com/dimitreoliveira/flower-with-tpus-k-fold-optimized-training-loop/notebook#Optimized-training-loop).\n- Make sure you end your kernel after committing or finish editing it, if you don't it will run some minutes after automatic shut down (I think 15 minutes).\n- Watch your TPU idle time to make sure you are using your TPU as much as you should, as [pointed here](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/133447) by Martin.\n", "1260116": "Thanks for sharing!", "2244238": "Thank you very much for sharing, it helped me to apply in other competitions.", "2080990": "@dimitreoliveira Hii, \nI have been following notebooks on Kaggle to train models on TPU. [This](https://www.kaggle.com/code/cdeotte/how-to-create-tfrecords) and [This](https://www.kaggle.com/code/philculliton/a-simple-tf-2-1-notebook/notebook) where the former notebook shows to make TF Records and the latter one to train models on TPU with these TF Records.\nI am strictly following these two notebooks but still encountering with strange error. I am using Skin Cancer HAM10000 dataset from Kaggle.\nThe error looks like this -\n```python\nOutOfRangeError: 9 root error(s) found.\n (0) Out of range: {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_2}}]]\n (1) Out of range: {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[tpu_compile_succeeded_assert/_1935868936045555324/_5/_143]]\n (2) Out of range: {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[TPUReplicate/_compile/_17327685160909203060/_4/_108]]\n (3) Out of range: {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[Pad_7/_86]]\n (4) Out of range: {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[Pad_15/_150]]\n (5) Out of range: {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[strided_slice_30/_120]]\n (6) Out of range: {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[TPUReplicate/_compile/_17327685160909203060/_4/_44]]\n (7) Out of range: {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n [[IteratorGetNext_5/_24]]\n (8) Out of range: {{function_node __inference_train_function_24967}} End of sequence\n [[{{node IteratorGetNext_1}}]]\n0 successful operations.\n0 derived errors ignored.\n```\n\nI have been searching over the internet for these issues and changing some Hyperparameters but could not able to resolve it. My notebook is [here](https://www.kaggle.com/code/nishchay331/skincancertpu2)\nPlease have a look into it and I would be grateful to get a reply back.\nThank you.", "1261416": "After your TPU time runs out, how do you figure out when the30 hours will be available again?", "1268579": "", "1649577": "This is very useful, thank you!", "1345797": "Wow! been looking for this. Thank you." } }