I use colab TPU to train xhlulu notebook, but colab uses TPU v2, batch size can only be set to 64. My val auc performance is not as good as tpu v3. I would like to ask if I use a better tpu and set the batch size to 256 or 512, is it possible to get better score? I don't know if EfficientNet has a chance to exceed the score of ResNet-200 with a single model.
\nBut TPU is so expensive. And I cannot rent TPU in my area.
", "messages": [ { "id": "1182022", "postDate": "02/02/2021 09:01:00", "content": "I use colab TPU to train xhlulu notebook, but colab uses TPU v2, batch size can only be set to 64. My val auc performance is not as good as tpu v3. I would like to ask if I use a better tpu and set the batch size to 256 or 512, is it possible to get better score? I don't know if EfficientNet has a chance to exceed the score of ResNet-200 with a single model.
\nBut TPU is so expensive. And I cannot rent TPU in my area.
", "rawMarkdown": "I use colab TPU to train [xhlulu notebook](https://www.kaggle.com/xhlulu/ranzcr-efficientnet-tpu-training), but colab uses TPU v2, batch size can only be set to 64. My val auc performance is not as good as tpu v3. I would like to ask if I use a better tpu and set the batch size to 256 or 512, is it possible to get better score? I don't know if EfficientNet has a chance to exceed the score of ResNet-200 with a single model.\n\nBut TPU is so expensive. And I cannot rent TPU in my area.", "votes": null }, { "id": "1182163", "postDate": "02/02/2021 10:37:57", "content": "from my experience, you dont really need batchsize that big. even 32 is enough for classify 1000 imagenet categories
", "rawMarkdown": "from my experience, you dont really need batchsize that big. even 32 is enough for classify 1000 imagenet categories", "votes": null }, { "id": "1182692", "postDate": "02/02/2021 14:52:48", "content": "There are several studies of studying the effect of batch_size on training quality:
\nhttps://medium.com/deep-learning-experiments/effect-of-batch-size-on-neural-net-training-c5ae8516e57
\nhttps://medium.com/mini-distill/effect-of-batch-size-on-training-dynamics-21c14f7a716e
\nhttps://arxiv.org/pdf/1705.08741.pdf
According to the above, both batch size = 32 or 64 should be fine. If you set that to 256 or 512, you may experience a decrease in the learning performance per the studies. And please share with us if that is the case for your experiment.
", "rawMarkdown": "There are several studies of studying the effect of batch_size on training quality:\n\nhttps://medium.com/deep-learning-experiments/effect-of-batch-size-on-neural-net-training-c5ae8516e57\nhttps://medium.com/mini-distill/effect-of-batch-size-on-training-dynamics-21c14f7a716e\nhttps://arxiv.org/pdf/1705.08741.pdf\n\nAccording to the above, both batch size = 32 or 64 should be fine. If you set that to 256 or 512, you may experience a decrease in the learning performance per the studies. And please share with us if that is the case for your experiment.", "votes": null }, { "id": "1182747", "postDate": "02/02/2021 15:18:13", "content": "https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/204950#1118206
\nI know that the batch size does not usually need to be large. However, GPU and TPU performance have changed significantly, and TPU V3 is better than TPU V2. So I think if I use a better TPU and then increase the batch size, whether it can increase EfficientNet's performance.
", "rawMarkdown": "[https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/204950#1118206](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/204950#1118206)\n\nI know that the batch size does not usually need to be large. However, GPU and TPU performance have changed significantly, and TPU V3 is better than TPU V2. So I think if I use a better TPU and then increase the batch size, whether it can increase EfficientNet's performance.", "votes": null }, { "id": "1182767", "postDate": "02/02/2021 15:28:32", "content": "Thanks for letting me know. Good point on the use of TPU.
", "rawMarkdown": "Thanks for letting me know. Good point on the use of TPU.", "votes": null } ], "comments": [ { "id": 1182163, "author_name": "moewie94", "author_url": "", "post_date": "02/02/2021 10:37:57", "content": "from my experience, you dont really need batchsize that big. even 32 is enough for classify 1000 imagenet categories
", "votes": null, "replies": [] }, { "id": 1182692, "author_name": "pukkinming", "author_url": "", "post_date": "02/02/2021 14:52:48", "content": "There are several studies of studying the effect of batch_size on training quality:
\nhttps://medium.com/deep-learning-experiments/effect-of-batch-size-on-neural-net-training-c5ae8516e57
\nhttps://medium.com/mini-distill/effect-of-batch-size-on-training-dynamics-21c14f7a716e
\nhttps://arxiv.org/pdf/1705.08741.pdf
According to the above, both batch size = 32 or 64 should be fine. If you set that to 256 or 512, you may experience a decrease in the learning performance per the studies. And please share with us if that is the case for your experiment.
", "votes": null, "replies": [] }, { "id": 1182747, "author_name": "h053473666", "author_url": "", "post_date": "02/02/2021 15:18:13", "content": "https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/204950#1118206
\nI know that the batch size does not usually need to be large. However, GPU and TPU performance have changed significantly, and TPU V3 is better than TPU V2. So I think if I use a better TPU and then increase the batch size, whether it can increase EfficientNet's performance.
", "votes": null, "replies": [ { "id": 1182767, "author_name": "pukkinming", "author_url": "", "post_date": "02/02/2021 15:28:32", "content": "Thanks for letting me know. Good point on the use of TPU.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1182022": "I use colab TPU to train [xhlulu notebook](https://www.kaggle.com/xhlulu/ranzcr-efficientnet-tpu-training), but colab uses TPU v2, batch size can only be set to 64. My val auc performance is not as good as tpu v3. I would like to ask if I use a better tpu and set the batch size to 256 or 512, is it possible to get better score? I don't know if EfficientNet has a chance to exceed the score of ResNet-200 with a single model.\n\nBut TPU is so expensive. And I cannot rent TPU in my area.", "1182163": "from my experience, you dont really need batchsize that big. even 32 is enough for classify 1000 imagenet categories", "1182692": "There are several studies of studying the effect of batch_size on training quality:\n\nhttps://medium.com/deep-learning-experiments/effect-of-batch-size-on-neural-net-training-c5ae8516e57\nhttps://medium.com/mini-distill/effect-of-batch-size-on-training-dynamics-21c14f7a716e\nhttps://arxiv.org/pdf/1705.08741.pdf\n\nAccording to the above, both batch size = 32 or 64 should be fine. If you set that to 256 or 512, you may experience a decrease in the learning performance per the studies. And please share with us if that is the case for your experiment.", "1182747": "[https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/204950#1118206](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/204950#1118206)\n\nI know that the batch size does not usually need to be large. However, GPU and TPU performance have changed significantly, and TPU V3 is better than TPU V2. So I think if I use a better TPU and then increase the batch size, whether it can increase EfficientNet's performance.", "1182767": "Thanks for letting me know. Good point on the use of TPU." }, "source": "meta" }