First of all, congratulations to the winners, all the teams in the gold zone, and to everyone who took part and learnt something!
\nDid you use gradient accumulation to train your models?
\nHow did it work? How did you deal with BN layers (when present)?
\nWhich batch size and how many accumulation steps did you use?
\nI didn't have time to try it, and it could have been important, given the high resolution input images
", "messages": [ { "id": "1243400", "postDate": "03/18/2021 07:30:49", "content": "First of all, congratulations to the winners, all the teams in the gold zone, and to everyone who took part and learnt something!
\nDid you use gradient accumulation to train your models?
\nHow did it work? How did you deal with BN layers (when present)?
\nWhich batch size and how many accumulation steps did you use?
\nI didn't have time to try it, and it could have been important, given the high resolution input images
", "rawMarkdown": "First of all, congratulations to the winners, all the teams in the gold zone, and to everyone who took part and learnt something!\n\nDid you use gradient accumulation to train your models?\n\nHow did it work? How did you deal with BN layers (when present)?\n\nWhich batch size and how many accumulation steps did you use?\n\nI didn't have time to try it, and it could have been important, given the high resolution input images", "votes": null }, { "id": "1243454", "postDate": "03/18/2021 08:17:10", "content": "Take this with a grain of salt since others managed to train these models better, but here's what I did:
\n.eval() on the layer) after this was mentioned here and it immediately improved my results.I'd love to hear whether other approaches worked for people/there's better ways of doing this. What you can do when working around hardware limitations is surely interesting to a lot of people.
", "rawMarkdown": "Take this with a grain of salt since others managed to train these models better, but here's what I did:\n* I [used gradient accumulation](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/226684) to train ResNet-200D with 640 by 640 images and SE-ResNet-152D with 704 by 704 on a GTX 1080 TI.\n* My batch size was 4 and I accumulated over 8 batches (effective batch size of 32 - I did not experiment around that, but just went by the [\"Friends don't let friends use batch sizes over 32\" quote](https://twitter.com/ylecun/status/989610208497360896?lang=en)). \n* I froze all BatchNorm layers (i.e. set `.eval()` on the layer) after this was mentioned [here](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/224085) and it immediately improved my results.\n* I had done some experimentation with smaller models, where more complex model heads worked pretty well (e.g. when I did not need to freeze BN layers, I used the head suggested by Jeremy Howard and Sylvain Gugger in [their book](https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527) - see also the [repository for the book](https://github.com/fastai/fastbook/blob/master/15_arch_details.ipynb): \nAdaptiveConcatPool2d - Flatten - BatchNorm1d - Dropout - FC - ReLU - BatchNorm1d - Dropout - FC to output). I just could not get that to work with small batch sizes. That was presumably because of the BN layers, so I tried to experiment with high momentum etc., but I did not find a way to get it to work well. In the end, I did just go straight to an output layer after pooling & flattening.\n\nI'd love to hear whether other approaches worked for people/there's better ways of doing this. What you can do when working around hardware limitations is surely interesting to a lot of people.", "votes": null }, { "id": "1243530", "postDate": "03/18/2021 09:42:17", "content": "I tried multiple times in the past and during this comp. It didn’t really work. It’s pretty hard to say if it works when the batchnorm presents.
", "rawMarkdown": "I tried multiple times in the past and during this comp. It didn’t really work. It’s pretty hard to say if it works when the batchnorm presents.", "votes": null }, { "id": "1244888", "postDate": "03/19/2021 09:29:43", "content": "I used gradient accumulation to train large model like Resnet200d. I have only 1080Ti GPU with 11GB RAM.
", "rawMarkdown": "I used gradient accumulation to train large model like Resnet200d. I have only 1080Ti GPU with 11GB RAM.", "votes": null }, { "id": "1245941", "postDate": "03/20/2021 11:18:21", "content": "@yoshitaka1105 which batch size and accumulation steps did you use? have you tested if it actually improved the results? did you replaced the BN layers by something, or freeze these layers or use a high momentum? thanks!
", "rawMarkdown": "yoshitaka1105 which batch size and accumulation steps did you use? have you tested if it actually improved the results? did you replaced the BN layers by something, or freeze these layers or use a high momentum? thanks!", "votes": null }, { "id": "1245958", "postDate": "03/20/2021 11:44:34", "content": "thanks @bjoernholzhauer @underwearfitting @yoshitaka1105 !
\n@bjoernholzhauer did you measured if it improved your results respect training without gradient accumulation and using the same simpler head?
\ndid you freeze BN layers to train models without gradient accumulation? I suspect that it could be the reason of the improvement, instead of the gradient accumulation
", "rawMarkdown": "thanks @bjoernholzhauer @underwearfitting @yoshitaka1105 !\n\n@bjoernholzhauer did you measured if it improved your results respect training without gradient accumulation and using the same simpler head?\n\ndid you freeze BN layers to train models without gradient accumulation? I suspect that it could be the reason of the improvement, instead of the gradient accumulation", "votes": null }, { "id": "1246009", "postDate": "03/20/2021 12:26:48", "content": "I didn't test without gradient accumulation, I suppose that might work fine with frozen BN layers (as long as one also reduces the learning rate accordingly). However, we know that with unfrozen BN layers largish batch sizes are good (for a start the models were originally trained like that), as long as your GPU(s) can handle it.
", "rawMarkdown": "I didn't test without gradient accumulation, I suppose that might work fine with frozen BN layers (as long as one also reduces the learning rate accordingly). However, we know that with unfrozen BN layers largish batch sizes are good (for a start the models were originally trained like that), as long as your GPU(s) can handle it.", "votes": null }, { "id": "1246192", "postDate": "03/20/2021 14:23:31", "content": "In my case, Batch size: 4 and Accumulation: 8 batches. I did not try other parameters, so I do not know how good my parameter is. And I froze all BN layers.
", "rawMarkdown": "In my case, Batch size: 4 and Accumulation: 8 batches. I did not try other parameters, so I do not know how good my parameter is. And I froze all BN layers.", "votes": null } ], "comments": [ { "id": 1243454, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "03/18/2021 08:17:10", "content": "Take this with a grain of salt since others managed to train these models better, but here's what I did:
\n.eval() on the layer) after this was mentioned here and it immediately improved my results.I'd love to hear whether other approaches worked for people/there's better ways of doing this. What you can do when working around hardware limitations is surely interesting to a lot of people.
", "votes": null, "replies": [] }, { "id": 1243530, "author_name": "underwearfitting", "author_url": "", "post_date": "03/18/2021 09:42:17", "content": "I tried multiple times in the past and during this comp. It didn’t really work. It’s pretty hard to say if it works when the batchnorm presents.
", "votes": null, "replies": [] }, { "id": 1244888, "author_name": "yoshitaka1105", "author_url": "", "post_date": "03/19/2021 09:29:43", "content": "I used gradient accumulation to train large model like Resnet200d. I have only 1080Ti GPU with 11GB RAM.
", "votes": null, "replies": [ { "id": 1245941, "author_name": "virilo", "author_url": "", "post_date": "03/20/2021 11:18:21", "content": "@yoshitaka1105 which batch size and accumulation steps did you use? have you tested if it actually improved the results? did you replaced the BN layers by something, or freeze these layers or use a high momentum? thanks!
", "votes": null, "replies": [] }, { "id": 1246192, "author_name": "yoshitaka1105", "author_url": "", "post_date": "03/20/2021 14:23:31", "content": "In my case, Batch size: 4 and Accumulation: 8 batches. I did not try other parameters, so I do not know how good my parameter is. And I froze all BN layers.
", "votes": null, "replies": [] } ] }, { "id": 1245958, "author_name": "virilo", "author_url": "", "post_date": "03/20/2021 11:44:34", "content": "thanks @bjoernholzhauer @underwearfitting @yoshitaka1105 !
\n@bjoernholzhauer did you measured if it improved your results respect training without gradient accumulation and using the same simpler head?
\ndid you freeze BN layers to train models without gradient accumulation? I suspect that it could be the reason of the improvement, instead of the gradient accumulation
", "votes": null, "replies": [ { "id": 1246009, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "03/20/2021 12:26:48", "content": "I didn't test without gradient accumulation, I suppose that might work fine with frozen BN layers (as long as one also reduces the learning rate accordingly). However, we know that with unfrozen BN layers largish batch sizes are good (for a start the models were originally trained like that), as long as your GPU(s) can handle it.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1243400": "First of all, congratulations to the winners, all the teams in the gold zone, and to everyone who took part and learnt something!\n\nDid you use gradient accumulation to train your models?\n\nHow did it work? How did you deal with BN layers (when present)?\n\nWhich batch size and how many accumulation steps did you use?\n\nI didn't have time to try it, and it could have been important, given the high resolution input images", "1243454": "Take this with a grain of salt since others managed to train these models better, but here's what I did:\n* I [used gradient accumulation](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/226684) to train ResNet-200D with 640 by 640 images and SE-ResNet-152D with 704 by 704 on a GTX 1080 TI.\n* My batch size was 4 and I accumulated over 8 batches (effective batch size of 32 - I did not experiment around that, but just went by the [\"Friends don't let friends use batch sizes over 32\" quote](https://twitter.com/ylecun/status/989610208497360896?lang=en)). \n* I froze all BatchNorm layers (i.e. set `.eval()` on the layer) after this was mentioned [here](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/224085) and it immediately improved my results.\n* I had done some experimentation with smaller models, where more complex model heads worked pretty well (e.g. when I did not need to freeze BN layers, I used the head suggested by Jeremy Howard and Sylvain Gugger in [their book](https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527) - see also the [repository for the book](https://github.com/fastai/fastbook/blob/master/15_arch_details.ipynb): \nAdaptiveConcatPool2d - Flatten - BatchNorm1d - Dropout - FC - ReLU - BatchNorm1d - Dropout - FC to output). I just could not get that to work with small batch sizes. That was presumably because of the BN layers, so I tried to experiment with high momentum etc., but I did not find a way to get it to work well. In the end, I did just go straight to an output layer after pooling & flattening.\n\nI'd love to hear whether other approaches worked for people/there's better ways of doing this. What you can do when working around hardware limitations is surely interesting to a lot of people.", "1243530": "I tried multiple times in the past and during this comp. It didn’t really work. It’s pretty hard to say if it works when the batchnorm presents.", "1244888": "I used gradient accumulation to train large model like Resnet200d. I have only 1080Ti GPU with 11GB RAM.", "1245941": "yoshitaka1105 which batch size and accumulation steps did you use? have you tested if it actually improved the results? did you replaced the BN layers by something, or freeze these layers or use a high momentum? thanks!", "1245958": "thanks @bjoernholzhauer @underwearfitting @yoshitaka1105 !\n\n@bjoernholzhauer did you measured if it improved your results respect training without gradient accumulation and using the same simpler head?\n\ndid you freeze BN layers to train models without gradient accumulation? I suspect that it could be the reason of the improvement, instead of the gradient accumulation", "1246009": "I didn't test without gradient accumulation, I suppose that might work fine with frozen BN layers (as long as one also reduces the learning rate accordingly). However, we know that with unfrozen BN layers largish batch sizes are good (for a start the models were originally trained like that), as long as your GPU(s) can handle it.", "1246192": "In my case, Batch size: 4 and Accumulation: 8 batches. I did not try other parameters, so I do not know how good my parameter is. And I froze all BN layers." }, "source": "meta" }