\n\nMany congratulations to all the winners in this competition. Huge thanks to all the discussion topics, public notebooks and datasets - most of my learning came from them.
\n
For the first time I am finishing so high in the final leaderboard grabbing a silver medal. I have failed miserably in past competitions, never had a satisfying finish but this time I really feel like I am learning from my mistakes, so yeah it feels great!
\nCompetition Data ONLY. No extra data.
\n5 Fold Stratified CV
\nImage Size: 512 for B7 and 384 for ViT
\nEpochs: 20
\nEarly Stopping: No (Saved Best Epoch)
\nLoss Function: Bi-tempered Logistic Loss with Label Smoothing
\nOptimizer: Good old \"Adam\"
\nLR Scheduler: Cosine Annealing with Warm Restarts
\nBatch Size: 32 (with gradient accumulation)
\nAugmentations: Standard (No CutMix, FMix or SnapMix)
- Transpose\n - HorizontalFlip\n - VerticalFlip\n - ShiftScaleRotate\n - HueSaturationValue\n - RandomBrightnessContrast\n - CoarseDropout\n - Cutout\n\n\nEdit: Both models didn't use the same augmentations, I did use mix augs for ViT.
\n
\n EfficientNet-B7: Standard Augs
\n ViT Base P16: Standard + FMix + CutMix
I experimented a few things here,
\nI noticed that my OOF predictions were more stable when I used Standard 4xTTA for both ViT and B7 so I went with that for the final submission.
\nI decided to evenly blend them at first and surprisingly that gave me a boost in OOF (CV) Score from 0.898 --> 0.901 and also a boost in Public LB 0.902 --> 0.905. However a weighted ensemble always seemed to be just short yet better than my single models. This gave me a feeling that an unweighted ensemble can cut it for me so I went with it.
\n| Model | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| EfficientNet-B7 NS | \n0.894 | \n0.900 | \n0.894 | \n
| ViT Base16 | \n0.888 | \n- | \n- | \n
| EfficientNet-B7 NS 4xTTA | \n0.896 | \n0.902 | \n0.897 | \n
| ViT Base16 4xTTA | \n0.891 | \n0.899 | \n0.896 | \n
| (0.35 * ViT) + (0.65 * B7) weighted ensemble with 4xTTA | \n0.898 | \n0.903 | \n0.899 | \n
| ViT + B7 unweighted ensemble with 4xTTA | \n0.901 | \n0.905 | \n0.900 | \n
Looking forward to competing more and learning more!
", "messages": [ { "id": "1209710", "postDate": "02/19/2021 02:42:58", "content": "\n\nMany congratulations to all the winners in this competition. Huge thanks to all the discussion topics, public notebooks and datasets - most of my learning came from them.
\n
For the first time I am finishing so high in the final leaderboard grabbing a silver medal. I have failed miserably in past competitions, never had a satisfying finish but this time I really feel like I am learning from my mistakes, so yeah it feels great!
\nCompetition Data ONLY. No extra data.
\n5 Fold Stratified CV
\nImage Size: 512 for B7 and 384 for ViT
\nEpochs: 20
\nEarly Stopping: No (Saved Best Epoch)
\nLoss Function: Bi-tempered Logistic Loss with Label Smoothing
\nOptimizer: Good old \"Adam\"
\nLR Scheduler: Cosine Annealing with Warm Restarts
\nBatch Size: 32 (with gradient accumulation)
\nAugmentations: Standard (No CutMix, FMix or SnapMix)
- Transpose\n - HorizontalFlip\n - VerticalFlip\n - ShiftScaleRotate\n - HueSaturationValue\n - RandomBrightnessContrast\n - CoarseDropout\n - Cutout\n\n\nEdit: Both models didn't use the same augmentations, I did use mix augs for ViT.
\n
\n EfficientNet-B7: Standard Augs
\n ViT Base P16: Standard + FMix + CutMix
I experimented a few things here,
\nI noticed that my OOF predictions were more stable when I used Standard 4xTTA for both ViT and B7 so I went with that for the final submission.
\nI decided to evenly blend them at first and surprisingly that gave me a boost in OOF (CV) Score from 0.898 --> 0.901 and also a boost in Public LB 0.902 --> 0.905. However a weighted ensemble always seemed to be just short yet better than my single models. This gave me a feeling that an unweighted ensemble can cut it for me so I went with it.
\n| Model | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| EfficientNet-B7 NS | \n0.894 | \n0.900 | \n0.894 | \n
| ViT Base16 | \n0.888 | \n- | \n- | \n
| EfficientNet-B7 NS 4xTTA | \n0.896 | \n0.902 | \n0.897 | \n
| ViT Base16 4xTTA | \n0.891 | \n0.899 | \n0.896 | \n
| (0.35 * ViT) + (0.65 * B7) weighted ensemble with 4xTTA | \n0.898 | \n0.903 | \n0.899 | \n
| ViT + B7 unweighted ensemble with 4xTTA | \n0.901 | \n0.905 | \n0.900 | \n
Looking forward to competing more and learning more!
", "rawMarkdown": "> **Many congratulations to all the winners in this competition. Huge thanks to all the discussion topics, public notebooks and datasets - most of my learning came from them.**\n\nFor the first time I am finishing so high in the final leaderboard grabbing a silver medal. I have failed miserably in past competitions, never had a satisfying finish but this time I really feel like I am learning from my mistakes, so yeah it feels great! \n\ngreat work, congratulations!
", "rawMarkdown": "great work, congratulations!", "votes": null }, { "id": "1209735", "postDate": "02/19/2021 02:57:16", "content": "Thanks @wonjunpark congrats to you too
", "rawMarkdown": "Thanks @wonjunpark congrats to you too", "votes": null }, { "id": "1209784", "postDate": "02/19/2021 03:33:01", "content": "Thank you so much for sharing and congratulations on the silver. Fantastic work!
", "rawMarkdown": "Thank you so much for sharing and congratulations on the silver. Fantastic work!", "votes": null }, { "id": "1209788", "postDate": "02/19/2021 03:34:40", "content": "Thanks @julianbakerphd
", "rawMarkdown": "Thanks @julianbakerphd", "votes": null }, { "id": "1209797", "postDate": "02/19/2021 03:40:46", "content": "Congrats on a solo silver medal. Great! @abhinand05
", "rawMarkdown": "Congrats on a solo silver medal. Great! @abhinand05", "votes": null }, { "id": "1209798", "postDate": "02/19/2021 03:42:01", "content": "Thanks, Abhinand for the detailed and useful post. It is encouraging to see that with a simple setup you were able to achieve a silver here. A quick question. Did you use your own hardware for training? Training 20 epochs/5 fold would have run easily for 12+ hours on a GPU, isn't it?.
", "rawMarkdown": "Thanks, Abhinand for the detailed and useful post. It is encouraging to see that with a simple setup you were able to achieve a silver here. A quick question. Did you use your own hardware for training? Training 20 epochs/5 fold would have run easily for 12+ hours on a GPU, isn't it?.", "votes": null }, { "id": "1209808", "postDate": "02/19/2021 03:48:41", "content": "My own hardware wasn't good enough I used it for the baseline b0 model though. I split the process into 10 epochs each every time and used Kaggle Hardware + Google Colab and waited long hours.
", "rawMarkdown": "My own hardware wasn't good enough I used it for the baseline b0 model though. I split the process into 10 epochs each every time and used Kaggle Hardware + Google Colab and waited long hours.", "votes": null }, { "id": "1209824", "postDate": "02/19/2021 03:59:50", "content": "Thank you for sharing these results! Wondering why you chose Efnet B7 first? What were the models you've tested at first? (If you did)
", "rawMarkdown": "Thank you for sharing these results! Wondering why you chose Efnet B7 first? What were the models you've tested at first? (If you did)", "votes": null }, { "id": "1209838", "postDate": "02/19/2021 04:14:56", "content": "Based on my CV score it was always slightly better than my B4 and gave a nice boost in LB and combined well with ViT. Maybe I could have still got similar results with smaller models but was short on time and resources so gave B7 a shot and luckily it paid off.
", "rawMarkdown": "Based on my CV score it was always slightly better than my B4 and gave a nice boost in LB and combined well with ViT. Maybe I could have still got similar results with smaller models but was short on time and resources so gave B7 a shot and luckily it paid off.", "votes": null }, { "id": "1209839", "postDate": "02/19/2021 04:16:37", "content": "Thanks @piantic congrats on your medal as well
", "rawMarkdown": "Thanks @piantic congrats on your medal as well", "votes": null }, { "id": "1209850", "postDate": "02/19/2021 04:28:43", "content": "Congrats on your silver @abhinand05 ! Could you please say what you mean by \"unweighted ensemble\" at the last ? And as serigne mentioned is this thread , many single models seems to outperform the ensemble of those good models . Could you share your thoughts on this ? Thanks for the write-up btw !
", "rawMarkdown": "Congrats on your silver @abhinand05 ! Could you please say what you mean by \"unweighted ensemble\" at the last ? And as *serigne* mentioned is [this](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/220590) thread , many single models seems to outperform the ensemble of those good models . Could you share your thoughts on this ? Thanks for the write-up btw !", "votes": null }, { "id": "1209876", "postDate": "02/19/2021 04:49:10", "content": "\n\nCould you please say what you mean by \"unweighted ensemble\" at the last
\n
Just the simple average over your predicted probabilities.
\n\n\nmany single models seems to outperform the ensemble of those good models . Could you share your thoughts on this ?
\n
The margins are tiny in this competition. Maybe it was just a stroke of luck, my best single model B7 has only scored 0.897.
", "rawMarkdown": "> Could you please say what you mean by \"unweighted ensemble\" at the last \n\nJust the simple average over your predicted probabilities. \n\n> many single models seems to outperform the ensemble of those good models . Could you share your thoughts on this ?\n\nThe margins are tiny in this competition. Maybe it was just a stroke of luck, my best single model B7 has only scored 0.897.", "votes": null }, { "id": "1209907", "postDate": "02/19/2021 05:24:17", "content": "Ah, I see thank you! For me, I we tested only a few models such as B4, resnet and se-resnext… Wished we could've tried more. So for your final solution, did you use light or standard TTA?
", "rawMarkdown": "Ah, I see thank you! For me, I we tested only a few models such as B4, resnet and se-resnext... Wished we could've tried more. So for your final solution, did you use light or standard TTA?", "votes": null }, { "id": "1209909", "postDate": "02/19/2021 05:24:55", "content": "Couldn't be done without your support in discussions and notebooks haha
", "rawMarkdown": "Couldn't be done without your support in discussions and notebooks haha", "votes": null }, { "id": "1209911", "postDate": "02/19/2021 05:27:15", "content": "Standard TTA.
", "rawMarkdown": "Standard TTA.", "votes": null }, { "id": "1210038", "postDate": "02/19/2021 07:03:17", "content": "Congratulations.
\n\n\nCompetition Data ONLY. No extra data.
\n
This is impressive!
\nMind to share a code?
", "rawMarkdown": "Congratulations.\n\n> Competition Data ONLY. No extra data.\n\nThis is impressive!\n\nMind to share a code?", "votes": null }, { "id": "1210070", "postDate": "02/19/2021 07:28:57", "content": "Thanks @nroman Will share code soon
", "rawMarkdown": "Thanks @nroman Will share code soon", "votes": null }, { "id": "1210177", "postDate": "02/19/2021 08:39:53", "content": "Congrats!
", "rawMarkdown": "Congrats!", "votes": null }, { "id": "1210244", "postDate": "02/19/2021 09:37:46", "content": "Nice to see different approaches. Congratulations!!
", "rawMarkdown": "Nice to see different approaches. Congratulations!!", "votes": null }, { "id": "1210445", "postDate": "02/19/2021 12:49:52", "content": "Congratulations!!!!!
\nAlso informative write-up…
", "rawMarkdown": "Congratulations!!!!!\n\nAlso informative write-up...", "votes": null }, { "id": "1211467", "postDate": "02/20/2021 09:00:49", "content": "Thanks for the write-up, Finally someone with great results with no extra data. That's impressive. Congratulations Man.
\nWill you be kind enough to share your training code of Vit and efficient models. Will be learning experience for beginners like me.
Thanks @abhinand05 for sharing your solution writeup.
", "rawMarkdown": "Thanks @abhinand05 for sharing your solution writeup.", "votes": null } ], "comments": [ { "id": 1209731, "author_name": "wonjunpark", "author_url": "", "post_date": "02/19/2021 02:53:51", "content": "great work, congratulations!
", "votes": null, "replies": [ { "id": 1209735, "author_name": "abhinand05", "author_url": "", "post_date": "02/19/2021 02:57:16", "content": "Thanks @wonjunpark congrats to you too
", "votes": null, "replies": [] } ] }, { "id": 1209784, "author_name": "julianbakerphd", "author_url": "", "post_date": "02/19/2021 03:33:01", "content": "Thank you so much for sharing and congratulations on the silver. Fantastic work!
", "votes": null, "replies": [ { "id": 1209788, "author_name": "abhinand05", "author_url": "", "post_date": "02/19/2021 03:34:40", "content": "Thanks @julianbakerphd
", "votes": null, "replies": [] } ] }, { "id": 1209797, "author_name": "piantic", "author_url": "", "post_date": "02/19/2021 03:40:46", "content": "Congrats on a solo silver medal. Great! @abhinand05
", "votes": null, "replies": [ { "id": 1209839, "author_name": "abhinand05", "author_url": "", "post_date": "02/19/2021 04:16:37", "content": "Thanks @piantic congrats on your medal as well
", "votes": null, "replies": [] }, { "id": 1209909, "author_name": "andyjianzhou", "author_url": "", "post_date": "02/19/2021 05:24:55", "content": "Couldn't be done without your support in discussions and notebooks haha
", "votes": null, "replies": [] } ] }, { "id": 1209798, "author_name": "krisho007", "author_url": "", "post_date": "02/19/2021 03:42:01", "content": "Thanks, Abhinand for the detailed and useful post. It is encouraging to see that with a simple setup you were able to achieve a silver here. A quick question. Did you use your own hardware for training? Training 20 epochs/5 fold would have run easily for 12+ hours on a GPU, isn't it?.
", "votes": null, "replies": [ { "id": 1209808, "author_name": "abhinand05", "author_url": "", "post_date": "02/19/2021 03:48:41", "content": "My own hardware wasn't good enough I used it for the baseline b0 model though. I split the process into 10 epochs each every time and used Kaggle Hardware + Google Colab and waited long hours.
", "votes": null, "replies": [] } ] }, { "id": 1209824, "author_name": "andyjianzhou", "author_url": "", "post_date": "02/19/2021 03:59:50", "content": "Thank you for sharing these results! Wondering why you chose Efnet B7 first? What were the models you've tested at first? (If you did)
", "votes": null, "replies": [ { "id": 1209838, "author_name": "abhinand05", "author_url": "", "post_date": "02/19/2021 04:14:56", "content": "Based on my CV score it was always slightly better than my B4 and gave a nice boost in LB and combined well with ViT. Maybe I could have still got similar results with smaller models but was short on time and resources so gave B7 a shot and luckily it paid off.
", "votes": null, "replies": [] }, { "id": 1209907, "author_name": "andyjianzhou", "author_url": "", "post_date": "02/19/2021 05:24:17", "content": "Ah, I see thank you! For me, I we tested only a few models such as B4, resnet and se-resnext… Wished we could've tried more. So for your final solution, did you use light or standard TTA?
", "votes": null, "replies": [] }, { "id": 1209911, "author_name": "abhinand05", "author_url": "", "post_date": "02/19/2021 05:27:15", "content": "Standard TTA.
", "votes": null, "replies": [] } ] }, { "id": 1209850, "author_name": "zarif98sjs", "author_url": "", "post_date": "02/19/2021 04:28:43", "content": "Congrats on your silver @abhinand05 ! Could you please say what you mean by \"unweighted ensemble\" at the last ? And as serigne mentioned is this thread , many single models seems to outperform the ensemble of those good models . Could you share your thoughts on this ? Thanks for the write-up btw !
", "votes": null, "replies": [ { "id": 1209876, "author_name": "abhinand05", "author_url": "", "post_date": "02/19/2021 04:49:10", "content": "\n\nCould you please say what you mean by \"unweighted ensemble\" at the last
\n
Just the simple average over your predicted probabilities.
\n\n\nmany single models seems to outperform the ensemble of those good models . Could you share your thoughts on this ?
\n
The margins are tiny in this competition. Maybe it was just a stroke of luck, my best single model B7 has only scored 0.897.
", "votes": null, "replies": [] } ] }, { "id": 1210038, "author_name": "nroman", "author_url": "", "post_date": "02/19/2021 07:03:17", "content": "Congratulations.
\n\n\nCompetition Data ONLY. No extra data.
\n
This is impressive!
\nMind to share a code?
", "votes": null, "replies": [ { "id": 1210070, "author_name": "abhinand05", "author_url": "", "post_date": "02/19/2021 07:28:57", "content": "Thanks @nroman Will share code soon
", "votes": null, "replies": [] } ] }, { "id": 1210177, "author_name": "elcaiseri", "author_url": "", "post_date": "02/19/2021 08:39:53", "content": "Congrats!
", "votes": null, "replies": [] }, { "id": 1210244, "author_name": "saurabh2mishra", "author_url": "", "post_date": "02/19/2021 09:37:46", "content": "Nice to see different approaches. Congratulations!!
", "votes": null, "replies": [] }, { "id": 1210445, "author_name": "subbuvolvosekar", "author_url": "", "post_date": "02/19/2021 12:49:52", "content": "Congratulations!!!!!
\nAlso informative write-up…
", "votes": null, "replies": [] }, { "id": 1211467, "author_name": "robertlangdonvinci", "author_url": "", "post_date": "02/20/2021 09:00:49", "content": "Thanks for the write-up, Finally someone with great results with no extra data. That's impressive. Congratulations Man.
\nWill you be kind enough to share your training code of Vit and efficient models. Will be learning experience for beginners like me.
Thanks @abhinand05 for sharing your solution writeup.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1209710": "> **Many congratulations to all the winners in this competition. Huge thanks to all the discussion topics, public notebooks and datasets - most of my learning came from them.**\n\nFor the first time I am finishing so high in the final leaderboard grabbing a silver medal. I have failed miserably in past competitions, never had a satisfying finish but this time I really feel like I am learning from my mistakes, so yeah it feels great! \n\n