I see the AUC of best public weight is 0.9632 for the ResNet200d model, but when I train some kernels, I only get the score around 0.950, far away from the public one. What can I do to train the model well, or are there some great training pipelines you can recommend for me?
\nThanks!
", "messages": [ { "id": "1236833", "postDate": "03/13/2021 13:50:19", "content": "I see the AUC of best public weight is 0.9632 for the ResNet200d model, but when I train some kernels, I only get the score around 0.950, far away from the public one. What can I do to train the model well, or are there some great training pipelines you can recommend for me?
\nThanks!
", "rawMarkdown": "I see the AUC of best public weight is 0.9632 for the ResNet200d model, but when I train some kernels, I only get the score around 0.950, far away from the public one. What can I do to train the model well, or are there some great training pipelines you can recommend for me?\n\nThanks!", "votes": null }, { "id": "1237036", "postDate": "03/13/2021 17:09:40", "content": "I struggled quite a bit with training ResNet-200D, especially about using small batch sizes due to not having anything better than the GPU on Kaggle / a GTX 1080 Ti at home. In this discussion, freezing BatchNorm layers came up (in addition to using mixed precision and gradient accumulation). With those tricks, I finally got a CV score AuC of 0.9473 that I thought was worth trying on the public LB (where this scored 0.958 - my public LB scores are always higher than my CV). I did some experimenting with learning rates, numbers of epochs, classification heads and learning rate schedules, but that was obviously limited, because a single 5-fold CV takes well over a day on my personal machine. So, I suspect with more experimentation one can improve to some extent beyond that.
\nIt also seems that at some point, to improve further (I guess we'll find out, soon, how far anyone got without such things) one needs to do fancier things like teacher-student training or using unlabelled external data. I'm planning to try some of those things in the final days.
", "rawMarkdown": "I struggled quite a bit with training ResNet-200D, especially about using small batch sizes due to not having anything better than the GPU on Kaggle / a GTX 1080 Ti at home. In [this discussion](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/224085), freezing BatchNorm layers came up (in addition to using mixed precision and gradient accumulation). With those tricks, I finally got a CV score AuC of 0.9473 that I thought was worth trying on the public LB (where this scored 0.958 - my public LB scores are always higher than my CV). I did some experimenting with learning rates, numbers of epochs, classification heads and learning rate schedules, but that was obviously limited, because a single 5-fold CV takes well over a day on my personal machine. So, I suspect with more experimentation one can improve to some extent beyond that.\n\nIt also seems that at some point, to improve further (I guess we'll find out, soon, how far anyone got without such things) one needs to do fancier things like [teacher-student training](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207577) or using [unlabelled external data](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/225445). I'm planning to try some of those things in the final days.", "votes": null }, { "id": "1237063", "postDate": "03/13/2021 17:36:31", "content": "Simple models can go a long way, with some fancy augmentations, I was able to reach 96.5+ CV on a resnet200d. It scored 96.6~ lb.
", "rawMarkdown": "Simple models can go a long way, with some fancy augmentations, I was able to reach 96.5+ CV on a resnet200d. It scored 96.6~ lb.", "votes": null }, { "id": "1237293", "postDate": "03/14/2021 02:59:41", "content": "Could you give more details about augmentations? @andy1010 , I get lower AUC or LB score when I use simpler model, such as Resnet50d;
", "rawMarkdown": "Could you give more details about augmentations? @andy1010 , I get lower AUC or LB score when I use simpler model, such as Resnet50d;", "votes": null } ], "comments": [ { "id": 1237036, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "03/13/2021 17:09:40", "content": "I struggled quite a bit with training ResNet-200D, especially about using small batch sizes due to not having anything better than the GPU on Kaggle / a GTX 1080 Ti at home. In this discussion, freezing BatchNorm layers came up (in addition to using mixed precision and gradient accumulation). With those tricks, I finally got a CV score AuC of 0.9473 that I thought was worth trying on the public LB (where this scored 0.958 - my public LB scores are always higher than my CV). I did some experimenting with learning rates, numbers of epochs, classification heads and learning rate schedules, but that was obviously limited, because a single 5-fold CV takes well over a day on my personal machine. So, I suspect with more experimentation one can improve to some extent beyond that.
\nIt also seems that at some point, to improve further (I guess we'll find out, soon, how far anyone got without such things) one needs to do fancier things like teacher-student training or using unlabelled external data. I'm planning to try some of those things in the final days.
", "votes": null, "replies": [ { "id": 1237063, "author_name": "andy1010", "author_url": "", "post_date": "03/13/2021 17:36:31", "content": "Simple models can go a long way, with some fancy augmentations, I was able to reach 96.5+ CV on a resnet200d. It scored 96.6~ lb.
", "votes": null, "replies": [] }, { "id": 1237293, "author_name": "zaviersnowish", "author_url": "", "post_date": "03/14/2021 02:59:41", "content": "Could you give more details about augmentations? @andy1010 , I get lower AUC or LB score when I use simpler model, such as Resnet50d;
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1236833": "I see the AUC of best public weight is 0.9632 for the ResNet200d model, but when I train some kernels, I only get the score around 0.950, far away from the public one. What can I do to train the model well, or are there some great training pipelines you can recommend for me?\n\nThanks!", "1237036": "I struggled quite a bit with training ResNet-200D, especially about using small batch sizes due to not having anything better than the GPU on Kaggle / a GTX 1080 Ti at home. In [this discussion](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/224085), freezing BatchNorm layers came up (in addition to using mixed precision and gradient accumulation). With those tricks, I finally got a CV score AuC of 0.9473 that I thought was worth trying on the public LB (where this scored 0.958 - my public LB scores are always higher than my CV). I did some experimenting with learning rates, numbers of epochs, classification heads and learning rate schedules, but that was obviously limited, because a single 5-fold CV takes well over a day on my personal machine. So, I suspect with more experimentation one can improve to some extent beyond that.\n\nIt also seems that at some point, to improve further (I guess we'll find out, soon, how far anyone got without such things) one needs to do fancier things like [teacher-student training](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207577) or using [unlabelled external data](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/225445). I'm planning to try some of those things in the final days.", "1237063": "Simple models can go a long way, with some fancy augmentations, I was able to reach 96.5+ CV on a resnet200d. It scored 96.6~ lb.", "1237293": "Could you give more details about augmentations? @andy1010 , I get lower AUC or LB score when I use simpler model, such as Resnet50d;" }, "source": "meta" }