I trained multiple models using the TPU lately and I wonder if there is any scientific way to determine optimal batch size.
\nI was able to train models with following batch sizes using simple EFnetB5 model :
Then when I do predictions I can evaluate model using model.predict for approx 4 times bigger batch size.
What is your experience with batch size while training on TPU?
", "messages": [ { "id": 962749, "postDate": "2020-08-08T12:04:36.023Z", "content": "I trained multiple models using the TPU lately and I wonder if there is any scientific way to determine optimal batch size.
\nI was able to train models with following batch sizes using simple EFnetB5 model :
Then when I do predictions I can evaluate model using model.predict for approx 4 times bigger batch size.
What is your experience with batch size while training on TPU?
", "rawMarkdown": "I trained multiple models using the TPU lately and I wonder if there is any scientific way to determine optimal batch size. \nI was able to train models with following batch sizes using simple EFnetB5 model : \n\n- 256x256 - batch_size = 128 * 8(replicas)\n- 384x384 - batch_size = 64 * 8(replicas)\n- 512x512 - batch_size = 32* 8(replicas)\n- 768x768 - batch_size = 16 * 8(replicas)\n- 1024x1024 - batch_size = 8 * 8 (replicas) \n\nThen when I do predictions I can evaluate model using `model.predict` for approx 4 times bigger batch size. \n\nWhat is your experience with batch size while training on TPU?", "votes": 5 }, { "id": 963069, "postDate": "2020-08-08T16:32:09.183Z", "content": "I've gotten away with 24 * 8 for 768x768
", "rawMarkdown": "I've gotten away with 24 * 8 for 768x768", "votes": 1, "replies": [ { "id": 964632, "postDate": "2020-08-10T04:14:51.343Z", "content": "What is the model size?
", "rawMarkdown": "What is the model size? " }, { "id": 965884, "postDate": "2020-08-10T23:48:30.657Z", "content": "B7 using Chris's triple stratified kernel
", "rawMarkdown": "B7 using Chris's triple stratified kernel" }, { "id": 967698, "postDate": "2020-08-12T12:44:11.910Z", "content": "This is wired, I tried to train B4 B5 noisy-student and B4 B5 image net (4 models at once) and I can fit only batch size of 3 or 2
", "rawMarkdown": "This is wired, I tried to train B4 B5 noisy-student and B4 B5 image net (4 models at once) and I can fit only batch size of 3 or 2\n" } ] }, { "id": 969458, "postDate": "2020-08-13T17:42:58.603Z", "content": "Important information here:
\n Kaggle's TPU are v3 and Colab are v2, so you may notice that on Kaggle, you may use larger batches and it will also run the epochs faster!
Today I tried to train model on 768 b4 and b5 (two of each, one with imagenet and one with noisy student) weights in the model together with metadata and ** it failed for batch size 16**. Now trying to re run for batch size of 8.
\n(here is sample of notebook that I used for training)
", "rawMarkdown": "Today I tried to train model on **768** b4 and b5 (two of each, one with imagenet and one with noisy student) weights in the model together **with metadata** and ** it failed for batch size 16**. Now trying to re run for batch size of 8. \n\n ([here is sample of notebook that I used for training](https://www.kaggle.com/janidziak/crazy-incredible-tpu-noisy-and-imgn-192-metadata)) \n" }, { "id": 962779, "postDate": "2020-08-08T12:40:36.380Z", "content": "Are your numbers for Tensorflow or Pytorch? Because as I posted yesterday I had big problems with memory and 384x384
", "rawMarkdown": "Are your numbers for Tensorflow or Pytorch? Because as I posted yesterday I had big problems with memory and 384x384 ", "replies": [ { "id": 963066, "postDate": "2020-08-08T16:30:00.507Z", "rawMarkdown": "", "isDeleted": true }, { "id": 963096, "postDate": "2020-08-08T17:02:34.200Z", "content": "I suppose it does, Abhishek has created this notebook: https://www.kaggle.com/abhishek/accelerator-power-hour-pytorch-tpu
\nNever tried yet, let me know how it works
I only tried tensorflow for now. What were your problems with 384?
", "rawMarkdown": "I only tried tensorflow for now. What were your problems with 384? " }, { "id": 963254, "postDate": "2020-08-08T19:51:29.660Z", "content": "he uses 224 not 384
", "rawMarkdown": "he uses 224 not 384" }, { "id": 963571, "postDate": "2020-08-09T06:12:16.473Z", "content": "@jacekpoplawski can you post the link to your discussion? Are you using tensorflow or pytorch yourself?
", "rawMarkdown": "@jacekpoplawski can you post the link to your discussion? Are you using tensorflow or pytorch yourself? " }, { "id": 963818, "postDate": "2020-08-09T10:20:23.597Z", "content": "I have problem with pytorch, TPU and 384x384 on b3. Will try to debug on colab today.
\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/173117
I've gotten away with 24 * 8 for 768x768
", "votes": 1, "replies": [ { "id": 964632, "author_name": "Janek Idziak", "author_url": "", "post_date": "2020-08-10T04:14:51.343000", "content": "What is the model size?
", "votes": 0, "replies": [] }, { "id": 965884, "author_name": "Tim Yee", "author_url": "", "post_date": "2020-08-10T23:48:30.657000", "content": "B7 using Chris's triple stratified kernel
", "votes": 0, "replies": [] }, { "id": 967698, "author_name": "Janek Idziak", "author_url": "", "post_date": "2020-08-12T12:44:11.910000", "content": "This is wired, I tried to train B4 B5 noisy-student and B4 B5 image net (4 models at once) and I can fit only batch size of 3 or 2
", "votes": 0, "replies": [] } ] }, { "id": 969458, "author_name": "Janek Idziak", "author_url": "", "post_date": "2020-08-13T17:42:58.603000", "content": "Important information here:
\n Kaggle's TPU are v3 and Colab are v2, so you may notice that on Kaggle, you may use larger batches and it will also run the epochs faster!
Today I tried to train model on 768 b4 and b5 (two of each, one with imagenet and one with noisy student) weights in the model together with metadata and ** it failed for batch size 16**. Now trying to re run for batch size of 8.
\n(here is sample of notebook that I used for training)
", "votes": 0, "replies": [] }, { "id": 962779, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2020-08-08T12:40:36.380000", "content": "Are your numbers for Tensorflow or Pytorch? Because as I posted yesterday I had big problems with memory and 384x384
", "votes": 0, "replies": [ { "id": 963066, "author_name": "", "author_url": "", "post_date": "2020-08-08T16:30:00.507000", "content": "", "votes": 0, "replies": [] }, { "id": 963096, "author_name": "Janek Idziak", "author_url": "", "post_date": "2020-08-08T17:02:34.200000", "content": "I suppose it does, Abhishek has created this notebook: https://www.kaggle.com/abhishek/accelerator-power-hour-pytorch-tpu
\nNever tried yet, let me know how it works
I only tried tensorflow for now. What were your problems with 384?
", "votes": 0, "replies": [] }, { "id": 963254, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2020-08-08T19:51:29.660000", "content": "he uses 224 not 384
", "votes": 0, "replies": [] }, { "id": 963571, "author_name": "Janek Idziak", "author_url": "", "post_date": "2020-08-09T06:12:16.473000", "content": "@jacekpoplawski can you post the link to your discussion? Are you using tensorflow or pytorch yourself?
", "votes": 0, "replies": [] }, { "id": 963818, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2020-08-09T10:20:23.597000", "content": "I have problem with pytorch, TPU and 384x384 on b3. Will try to debug on colab today.
\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/173117