{ "id": 220621, "title": "Things that worked in this competition", "url": "/competitions/cassava-leaf-disease-classification/discussion/220621", "author_name": "", "post_date": "2021-02-19T01:55:34.003936300Z", "votes": 9, "comment_count": 10, "views": 0, "content": "

I just had a quick look at all my submissions to understand what worked in this competition. The private LB is so tight (0.901 gold zone -> 0.898 last bronze medal) so we can't tell what really works, expect for the first place team with the amazing score of 0.913.

\n

Here are some observations I had from checking the private scores of all my experiments where the very basic and simple tricks could get you a 0.901 score in private LB:

\n

Things that worked to get + 0.9 on Private LB:

\n\n

The difference between my selected submission (private=0.8998) and my best submission (private=0.9019) is that my selected models were trained on 1 fold because they gave better public score than the average of 5 folds

\n
\n

Things that didn't work:

\n

Here is where I spent 90% of my time in this competition, working on things that didn't work! I entered this competition because of the noisy-labels, to learn how to deal with them since a lot of the real world data can be mislabeled, I don't regret the time I spent on this because it might be useful in the future.

\n

1- Manually denosing the data:

\n

I spent hours checking every image (especially in classes 0 and 4 where the model gets confused the most).
\n\"\"
\nAfter dropping around 300 images:

\n\n

2- Bigger architectures:

\n

EfficientNet B7 and se_resnext101 gave worse CV and LB scores compared to B3/B4 and se_resnext50.

\n

3- Heavy augmentations:

\n

CutMix, MixUp and the combination of both didn't work.

\n
\n

Final thought

\n

From the private score breach from 2 months ago, I realized that the public and private scores were so close, so I tried not to go for anything crazy in my final submissions selection, assuming that the distribution of both public and private sets is similar, otherwise I would have selected models trained with clean data.

", "messages": [ { "id": "1209652", "postDate": "02/19/2021 01:55:34", "content": "

I just had a quick look at all my submissions to understand what worked in this competition. The private LB is so tight (0.901 gold zone -> 0.898 last bronze medal) so we can't tell what really works, expect for the first place team with the amazing score of 0.913.

\n

Here are some observations I had from checking the private scores of all my experiments where the very basic and simple tricks could get you a 0.901 score in private LB:

\n

Things that worked to get + 0.9 on Private LB:

\n\n

The difference between my selected submission (private=0.8998) and my best submission (private=0.9019) is that my selected models were trained on 1 fold because they gave better public score than the average of 5 folds

\n
\n

Things that didn't work:

\n

Here is where I spent 90% of my time in this competition, working on things that didn't work! I entered this competition because of the noisy-labels, to learn how to deal with them since a lot of the real world data can be mislabeled, I don't regret the time I spent on this because it might be useful in the future.

\n

1- Manually denosing the data:

\n

I spent hours checking every image (especially in classes 0 and 4 where the model gets confused the most).
\n\"\"
\nAfter dropping around 300 images:

\n\n

2- Bigger architectures:

\n

EfficientNet B7 and se_resnext101 gave worse CV and LB scores compared to B3/B4 and se_resnext50.

\n

3- Heavy augmentations:

\n

CutMix, MixUp and the combination of both didn't work.

\n
\n

Final thought

\n

From the private score breach from 2 months ago, I realized that the public and private scores were so close, so I tried not to go for anything crazy in my final submissions selection, assuming that the distribution of both public and private sets is similar, otherwise I would have selected models trained with clean data.

", "rawMarkdown": "I just had a quick look at all my submissions to understand what worked in this competition. The private LB is so tight (0.901 gold zone -> 0.898 last bronze medal) so we can't tell what really works, expect for the first place team with the amazing score of 0.913.\n\nHere are some observations I had from checking the private scores of all my experiments where the very basic and simple tricks could get you a 0.901 score in private LB:\n\n## Things that worked to get + 0.9 on Private LB:\n* **Ensemble** of EfficientNet + se_resnext50 + ViT_base16 (Including ViT gives a significant boost)\n* **5-Fold models** score better in private LB. *(1-Fold models scored better in public LB).*\n* **Light augmentations** work the best: Rotation, flips, shift, cutout, CoarseDropout and pixel level augs.\n* TaylorCrossEntropyLoss and Bi-TemperedLoss loss functions work the best with noisy labels.\n* **Moderate TTA:** `N_TTA=12` gave the best score. Anything more than that degraded both CV and LB.\n* **Using external data.**\n* **Self Distillation:** I used the plant pathology's 1st place solution ([code available in their GitHub repo](https://github.com/alipay/cvpr2020-plant-pathology)) to do self distillation. You basically have to one hot encode the labels and blend them with the OOFs logits (labels * 0.7 + logits oofs * 0.3) and tweak the loss function to accept one hot encoded labels. I got a +0.003 boost with small models (from 0.887 -> 0.890). However, when I upgraded to bigger models, I didn't get any improvement.\n\n**The difference between my selected submission (private=0.8998) and my best submission (private=0.9019) is that my selected models were trained on 1 fold because they gave better public score than the average of 5 folds**\n***\n\n# Things that didn't work:\n**Here is where I spent 90% of my time in this competition, working on things that didn't work!** I entered this competition because of the noisy-labels, to learn how to deal with them since a lot of the real world data can be mislabeled, I don't regret the time I spent on this because it might be useful in the future.\n\n### 1- Manually denosing the data:\nI spent hours checking every image (especially in classes 0 and 4 where the model gets confused the most).\n![](https://i.ibb.co/PTryzF5/cm-original.png)\nAfter dropping around 300 images:\n* Validating on clean data: Improves CV but degrades LB.\n* Validing on noisy data: Degrades both CV and LB. So I decided not to continue further with it.\n\n\n### 2- Bigger architectures:\nEfficientNet B7 and se_resnext101 gave worse CV and LB scores compared to B3/B4 and se_resnext50.\n\n### 3- Heavy augmentations:\nCutMix, MixUp and the combination of both didn't work.\n\n***\n## Final thought\nFrom the private score breach from 2 months ago, I realized that the public and private scores were so close, so I tried not to go for anything crazy in my final submissions selection, assuming that the distribution of both public and private sets is similar, otherwise I would have selected models trained with clean data.", "votes": null }, { "id": "1209655", "postDate": "02/19/2021 01:59:00", "content": "

Congrats on your silver finish.

\n
\n

The private LB is so tight (0.91 gold zone -> 0.898 last bronze medal)

\n
\n

This was a very tight competition, many teams will have a tie score when the 4th and 5th digits are revealed.

", "rawMarkdown": "Congrats on your silver finish.\n>The private LB is so tight (0.91 gold zone -> 0.898 last bronze medal)\n \nThis was a very tight competition, many teams will have a tie score when the 4th and 5th digits are revealed.", "votes": null }, { "id": "1209687", "postDate": "02/19/2021 02:29:57", "content": "

Thank you @cdeotte :) Looking forward to read your write-up!

", "rawMarkdown": "Thank you @cdeotte :) Looking forward to read your write-up!", "votes": null }, { "id": "1209711", "postDate": "02/19/2021 02:43:35", "content": "

TTA was one of the key deciding factor in this competition. My best submission is 8 TTA * 3 Models (Not Selected). All my ensembles with at least 2 TTA scores 0.9+!! All my ensembles with no TTA doesn't cross 0.9!!

", "rawMarkdown": "TTA was one of the key deciding factor in this competition. My best submission is 8 TTA * 3 Models (Not Selected). All my ensembles with at least 2 TTA scores 0.9+!! All my ensembles with no TTA doesn't cross 0.9!!", "votes": null }, { "id": "1209718", "postDate": "02/19/2021 02:49:56", "content": "

Congratulations on your strong finish! What augmentations did you use in TTA?

", "rawMarkdown": "Congratulations on your strong finish! What augmentations did you use in TTA?", "votes": null }, { "id": "1209724", "postDate": "02/19/2021 02:52:08", "content": "

great work, congratulations!

", "rawMarkdown": "great work, congratulations!", "votes": null }, { "id": "1209727", "postDate": "02/19/2021 02:52:46", "content": "

2 LR Flips * 4 Rot90

", "rawMarkdown": "2 LR Flips * 4 Rot90", "votes": null }, { "id": "1209776", "postDate": "02/19/2021 03:26:24", "content": "

Thanks for sharing! @amiiiney

", "rawMarkdown": "Thanks for sharing! @amiiiney", "votes": null }, { "id": "1209794", "postDate": "02/19/2021 03:37:11", "content": "

Thank you @piantic! I used your public notebook as a starting point in this competition, it was very well structured. Thanks for everything you shared during the competition, I learned a lot from you!

", "rawMarkdown": "Thank you @piantic! I used your public notebook as a starting point in this competition, it was very well structured. Thanks for everything you shared during the competition, I learned a lot from you!", "votes": null }, { "id": "1212372", "postDate": "02/21/2021 07:01:15", "content": "

Good job and Congratulations!

", "rawMarkdown": "Good job and Congratulations!", "votes": null }, { "id": "1217914", "postDate": "02/25/2021 11:55:13", "content": "

Congrats and thanks for sharing !

", "rawMarkdown": "Congrats and thanks for sharing !", "votes": null } ], "comments": [ { "id": 1209655, "author_name": "cdeotte", "author_url": "", "post_date": "02/19/2021 01:59:00", "content": "

Congrats on your silver finish.

\n
\n

The private LB is so tight (0.91 gold zone -> 0.898 last bronze medal)

\n
\n

This was a very tight competition, many teams will have a tie score when the 4th and 5th digits are revealed.

", "votes": null, "replies": [ { "id": 1209687, "author_name": "amiiiney", "author_url": "", "post_date": "02/19/2021 02:29:57", "content": "

Thank you @cdeotte :) Looking forward to read your write-up!

", "votes": null, "replies": [] } ] }, { "id": 1209711, "author_name": "ks2019", "author_url": "", "post_date": "02/19/2021 02:43:35", "content": "

TTA was one of the key deciding factor in this competition. My best submission is 8 TTA * 3 Models (Not Selected). All my ensembles with at least 2 TTA scores 0.9+!! All my ensembles with no TTA doesn't cross 0.9!!

", "votes": null, "replies": [ { "id": 1209718, "author_name": "amiiiney", "author_url": "", "post_date": "02/19/2021 02:49:56", "content": "

Congratulations on your strong finish! What augmentations did you use in TTA?

", "votes": null, "replies": [] }, { "id": 1209727, "author_name": "ks2019", "author_url": "", "post_date": "02/19/2021 02:52:46", "content": "

2 LR Flips * 4 Rot90

", "votes": null, "replies": [] } ] }, { "id": 1209724, "author_name": "wonjunpark", "author_url": "", "post_date": "02/19/2021 02:52:08", "content": "

great work, congratulations!

", "votes": null, "replies": [] }, { "id": 1209776, "author_name": "piantic", "author_url": "", "post_date": "02/19/2021 03:26:24", "content": "

Thanks for sharing! @amiiiney

", "votes": null, "replies": [ { "id": 1209794, "author_name": "amiiiney", "author_url": "", "post_date": "02/19/2021 03:37:11", "content": "

Thank you @piantic! I used your public notebook as a starting point in this competition, it was very well structured. Thanks for everything you shared during the competition, I learned a lot from you!

", "votes": null, "replies": [] } ] }, { "id": 1212372, "author_name": "hungkhoi", "author_url": "", "post_date": "02/21/2021 07:01:15", "content": "

Good job and Congratulations!

", "votes": null, "replies": [] }, { "id": 1217914, "author_name": "arijzou", "author_url": "", "post_date": "02/25/2021 11:55:13", "content": "

Congrats and thanks for sharing !

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1209652": "I just had a quick look at all my submissions to understand what worked in this competition. The private LB is so tight (0.901 gold zone -> 0.898 last bronze medal) so we can't tell what really works, expect for the first place team with the amazing score of 0.913.\n\nHere are some observations I had from checking the private scores of all my experiments where the very basic and simple tricks could get you a 0.901 score in private LB:\n\n## Things that worked to get + 0.9 on Private LB:\n* **Ensemble** of EfficientNet + se_resnext50 + ViT_base16 (Including ViT gives a significant boost)\n* **5-Fold models** score better in private LB. *(1-Fold models scored better in public LB).*\n* **Light augmentations** work the best: Rotation, flips, shift, cutout, CoarseDropout and pixel level augs.\n* TaylorCrossEntropyLoss and Bi-TemperedLoss loss functions work the best with noisy labels.\n* **Moderate TTA:** `N_TTA=12` gave the best score. Anything more than that degraded both CV and LB.\n* **Using external data.**\n* **Self Distillation:** I used the plant pathology's 1st place solution ([code available in their GitHub repo](https://github.com/alipay/cvpr2020-plant-pathology)) to do self distillation. You basically have to one hot encode the labels and blend them with the OOFs logits (labels * 0.7 + logits oofs * 0.3) and tweak the loss function to accept one hot encoded labels. I got a +0.003 boost with small models (from 0.887 -> 0.890). However, when I upgraded to bigger models, I didn't get any improvement.\n\n**The difference between my selected submission (private=0.8998) and my best submission (private=0.9019) is that my selected models were trained on 1 fold because they gave better public score than the average of 5 folds**\n***\n\n# Things that didn't work:\n**Here is where I spent 90% of my time in this competition, working on things that didn't work!** I entered this competition because of the noisy-labels, to learn how to deal with them since a lot of the real world data can be mislabeled, I don't regret the time I spent on this because it might be useful in the future.\n\n### 1- Manually denosing the data:\nI spent hours checking every image (especially in classes 0 and 4 where the model gets confused the most).\n![](https://i.ibb.co/PTryzF5/cm-original.png)\nAfter dropping around 300 images:\n* Validating on clean data: Improves CV but degrades LB.\n* Validing on noisy data: Degrades both CV and LB. So I decided not to continue further with it.\n\n\n### 2- Bigger architectures:\nEfficientNet B7 and se_resnext101 gave worse CV and LB scores compared to B3/B4 and se_resnext50.\n\n### 3- Heavy augmentations:\nCutMix, MixUp and the combination of both didn't work.\n\n***\n## Final thought\nFrom the private score breach from 2 months ago, I realized that the public and private scores were so close, so I tried not to go for anything crazy in my final submissions selection, assuming that the distribution of both public and private sets is similar, otherwise I would have selected models trained with clean data.", "1209655": "Congrats on your silver finish.\n>The private LB is so tight (0.91 gold zone -> 0.898 last bronze medal)\n \nThis was a very tight competition, many teams will have a tie score when the 4th and 5th digits are revealed.", "1209687": "Thank you @cdeotte :) Looking forward to read your write-up!", "1209711": "TTA was one of the key deciding factor in this competition. My best submission is 8 TTA * 3 Models (Not Selected). All my ensembles with at least 2 TTA scores 0.9+!! All my ensembles with no TTA doesn't cross 0.9!!", "1209718": "Congratulations on your strong finish! What augmentations did you use in TTA?", "1209724": "great work, congratulations!", "1209727": "2 LR Flips * 4 Rot90", "1209776": "Thanks for sharing! @amiiiney", "1209794": "Thank you @piantic! I used your public notebook as a starting point in this competition, it was very well structured. Thanks for everything you shared during the competition, I learned a lot from you!", "1212372": "Good job and Congratulations!", "1217914": "Congrats and thanks for sharing !" }, "source": "meta" }