First and foremost, i would like to sincerely thank the Kaggle community for all the great tips and ideas throughout the competition.
\nPrior to the competition, i was struggling to understand the concept of averaging folds as a bagging technique, using OOF etc. Until i found @cdeotte discussion here (Thank you! @cdeotte) and together with some research, i felt i needed to put it into practice to ensure i had understood the concept. This competition is a good platform for me to apply it. Not only that, i didn't try ensembles before, and never really build a training pipeline for iterative experiments prior to this. I must say i have really learnt a lot from this competition :)
\nSpecial shoutout to @khyeh0719 and @reighns for their awesome notebooks from which i build my pipeline upon!
\nhttps://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-train-amp-aug
\nhttps://www.kaggle.com/reighns/complete-and-reusable-pytorch-pipeline
My solution is a weighted ensemble of 4 models: EfficientNetB4, Vit_base16, Resnet200d and NF-Resnet50. The models are trained on either with or without external data, with the images augmented in the form of Cutmix and Fmix alongside some basic albumentation augmentations. Inference was done with x4 TTA.
\nGithub: https://github.com/aqx95/cassava_kaggle
I would like to thank @tahsin for sharing with the community the merged 2019/2020 dataset. I used the merged data for training my NFNet. I did a stratified KFold only on the 2020 data source, and combine the 2019 data with the training folds to ensure that the 2019 data does not seep into my validation fold, which may affect my CV if there is a shift in input distribution between 2019 and 2020 data.
\nThe models i experimented with (5 folds):
\nI first got to know about VisionTransformers from this competition and i was immediately intrigued by its architecture; to solve vision tasks without convolution layers was just mind blowing to me! I would like to thank @mobassir for introducing the NFNet, which has quite a decent performance on my local CV. At the time of competing, timm package only has a few pre-trained weights available for transfer learning, and i used NF-Resnet50 (Thank you @rwightman)!
\n![\"corr\"](\"https://i.imgur.com/qOGMo90.png\")
\nCombining models with different strengths will usually give the best synergies. Thus, i generate a correlation matrix using out-of-fold predictions to experiment my way into ensembles. I find it interesting that the ViT model has the least correlation with all other models. Maybe because of the different underlying operation that creates the receptive field of the image (Convolution vs Attention) ??
EfficientNet B4 and NF-Resnet perform the strongest on my local CV without any TTA. However earlier on i was using EfficientNet B4 as a starting baseline since i only knew about NFNets few days before the deadline, and start building my ensemble from there. I cant seem to find any correlation between CV/Public/Private. Im eager to see what are some of the robust strategies that were used.
\n| Models | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| EffNet | \n0.897 | \n0.900 | \n0.898 | \n
| ViT | \n0.896 | \n0.898 | \n0.896 | \n
| NFNet* (no TTA) | \n0.896 | \n- | \n- | \n
| Resnet | \n0.895 | \n- | \n- | \n
| EffNet + ViT | \n0.898 | \n0.903 | \n0.900 | \n
| EffNet + ViT + Resnet + NFNet* | \n0.901 | \n0.904 | \n0.899 | \n
| EffNet + ViT + Resnet + NFNet* + Resnext (No TTA) | \n0.901 | \n0.902 | \n0.897 | \n
*denotes the use of external data (2019 dataset). All results are based on x4 TTA unless stated otherwise.
\nThe final model i used for submission consist of a weighted ensemble of 4 models, as shown in the 6th row, with x4 TTA. The funny thing is that a simple average of Effnet with ViT (5th row) gives a better private LB score.
\nAll inferences are done with x4 TTA as follows:
\nreturn Compose([\n RandomResizedCrop(CFG.size[CFG.model], CFG.size[CFG.model]), \n HorizontalFlip(p=0.5), \n VerticalFlip(p=0.5),\n Transpose(p=0.5),\n Normalize(\n mean=[0.485, 0.456, 0.406], \n std=[0.229, 0.224, 0.225], \n max_pixel_value=255.0, p=1.0),\n ToTensorV2(p=1.0),\n ])\nIt has been a fruitful journey to me, learning so much from the community. All the knowledge and hands-on experimenting will definitely be my greatest takeaway. I believe everything we learnt from here; whether is it useful or not in this competition, will definitely come to use in future :)
", "messages": [ { "id": "1209945", "postDate": "02/19/2021 05:52:16", "content": "First and foremost, i would like to sincerely thank the Kaggle community for all the great tips and ideas throughout the competition.
\nPrior to the competition, i was struggling to understand the concept of averaging folds as a bagging technique, using OOF etc. Until i found @cdeotte discussion here (Thank you! @cdeotte) and together with some research, i felt i needed to put it into practice to ensure i had understood the concept. This competition is a good platform for me to apply it. Not only that, i didn't try ensembles before, and never really build a training pipeline for iterative experiments prior to this. I must say i have really learnt a lot from this competition :)
\nSpecial shoutout to @khyeh0719 and @reighns for their awesome notebooks from which i build my pipeline upon!
\nhttps://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-train-amp-aug
\nhttps://www.kaggle.com/reighns/complete-and-reusable-pytorch-pipeline
My solution is a weighted ensemble of 4 models: EfficientNetB4, Vit_base16, Resnet200d and NF-Resnet50. The models are trained on either with or without external data, with the images augmented in the form of Cutmix and Fmix alongside some basic albumentation augmentations. Inference was done with x4 TTA.
\nGithub: https://github.com/aqx95/cassava_kaggle
I would like to thank @tahsin for sharing with the community the merged 2019/2020 dataset. I used the merged data for training my NFNet. I did a stratified KFold only on the 2020 data source, and combine the 2019 data with the training folds to ensure that the 2019 data does not seep into my validation fold, which may affect my CV if there is a shift in input distribution between 2019 and 2020 data.
\nThe models i experimented with (5 folds):
\nI first got to know about VisionTransformers from this competition and i was immediately intrigued by its architecture; to solve vision tasks without convolution layers was just mind blowing to me! I would like to thank @mobassir for introducing the NFNet, which has quite a decent performance on my local CV. At the time of competing, timm package only has a few pre-trained weights available for transfer learning, and i used NF-Resnet50 (Thank you @rwightman)!
\n
\nCombining models with different strengths will usually give the best synergies. Thus, i generate a correlation matrix using out-of-fold predictions to experiment my way into ensembles. I find it interesting that the ViT model has the least correlation with all other models. Maybe because of the different underlying operation that creates the receptive field of the image (Convolution vs Attention) ??
EfficientNet B4 and NF-Resnet perform the strongest on my local CV without any TTA. However earlier on i was using EfficientNet B4 as a starting baseline since i only knew about NFNets few days before the deadline, and start building my ensemble from there. I cant seem to find any correlation between CV/Public/Private. Im eager to see what are some of the robust strategies that were used.
\n| Models | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| EffNet | \n0.897 | \n0.900 | \n0.898 | \n
| ViT | \n0.896 | \n0.898 | \n0.896 | \n
| NFNet* (no TTA) | \n0.896 | \n- | \n- | \n
| Resnet | \n0.895 | \n- | \n- | \n
| EffNet + ViT | \n0.898 | \n0.903 | \n0.900 | \n
| EffNet + ViT + Resnet + NFNet* | \n0.901 | \n0.904 | \n0.899 | \n
| EffNet + ViT + Resnet + NFNet* + Resnext (No TTA) | \n0.901 | \n0.902 | \n0.897 | \n
*denotes the use of external data (2019 dataset). All results are based on x4 TTA unless stated otherwise.
\nThe final model i used for submission consist of a weighted ensemble of 4 models, as shown in the 6th row, with x4 TTA. The funny thing is that a simple average of Effnet with ViT (5th row) gives a better private LB score.
\nAll inferences are done with x4 TTA as follows:
\nreturn Compose([\n RandomResizedCrop(CFG.size[CFG.model], CFG.size[CFG.model]), \n HorizontalFlip(p=0.5), \n VerticalFlip(p=0.5),\n Transpose(p=0.5),\n Normalize(\n mean=[0.485, 0.456, 0.406], \n std=[0.229, 0.224, 0.225], \n max_pixel_value=255.0, p=1.0),\n ToTensorV2(p=1.0),\n ])\nIt has been a fruitful journey to me, learning so much from the community. All the knowledge and hands-on experimenting will definitely be my greatest takeaway. I believe everything we learnt from here; whether is it useful or not in this competition, will definitely come to use in future :)
", "rawMarkdown": "First and foremost, i would like to sincerely thank the Kaggle community for all the great tips and ideas throughout the competition. \n\nPrior to the competition, i was struggling to understand the concept of averaging folds as a bagging technique, using OOF etc. Until i found [@cdeotte](https://www.kaggle.com/cdeotte) discussion [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614) (Thank you! [@cdeotte](https://www.kaggle.com/cdeotte)) and together with some research, i felt i needed to put it into practice to ensure i had understood the concept. This competition is a good platform for me to apply it. Not only that, i didn't try ensembles before, and never really build a training pipeline for iterative experiments prior to this. I must say i have really learnt a lot from this competition :)\n\nSpecial shoutout to [@khyeh0719](https://www.kaggle.com/khyeh0719) and [@reighns](https://www.kaggle.com/reighns) for their awesome notebooks from which i build my pipeline upon!\nhttps://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-train-amp-aug\nhttps://www.kaggle.com/reighns/complete-and-reusable-pytorch-pipeline\n\n### TLDR;\nMy solution is a weighted ensemble of 4 models: **EfficientNetB4**, **Vit_base16**, **Resnet200d** and **NF-Resnet50**. The models are trained on either with or without external data, with the images augmented in the form of Cutmix and Fmix alongside some basic albumentation augmentations. Inference was done with x4 TTA.\nGithub: https://github.com/aqx95/cassava_kaggle\n\n### Dataset\nI would like to thank [@tahsin](https://www.kaggle.com/tahsin) for sharing with the community the merged 2019/2020 dataset. I used the merged data for training my NFNet. I did a stratified KFold only on the 2020 data source, and combine the 2019 data with the training folds to ensure that the 2019 data does not seep into my validation fold, which may affect my CV if there is a shift in input distribution between 2019 and 2020 data.\n\n### Models\nThe models i experimented with (5 folds):\n* EfficientNet B4 (512 x 512)\n* Vit base16 (384 x 384)\n* Resnet200d (512 x 512)\n* NF-Resnet50 (512 x 512)\n* Resnext50_32x4d (512 x 512)\n\nI first got to know about [VisionTransformers](https://arxiv.org/pdf/2010.11929.pdf) from this competition and i was immediately intrigued by its architecture; to solve vision tasks without convolution layers was just mind blowing to me! I would like to thank [@mobassir](https://www.kaggle.com/mobassir) for introducing the [NFNet](https://arxiv.org/abs/2102.06171), which has quite a decent performance on my local CV. At the time of competing, timm package only has a few pre-trained weights available for transfer learning, and i used NF-Resnet50 (Thank you [@rwightman](https://www.kaggle.com/rwightman))! \n\n####Model Correlation\n\nCombining models with different strengths will usually give the best synergies. Thus, i generate a correlation matrix using out-of-fold predictions to experiment my way into ensembles. I find it interesting that the **ViT model has the least correlation with all other models**. Maybe because of the different underlying operation that creates the receptive field of the image (Convolution vs Attention) ??\n\n\n#### Ensemble\nEfficientNet B4 and NF-Resnet perform the strongest on my local CV without any TTA. However earlier on i was using EfficientNet B4 as a starting baseline since i only knew about NFNets few days before the deadline, and start building my ensemble from there. I cant seem to find any correlation between CV/Public/Private. Im eager to see what are some of the robust strategies that were used.\n\n| Models | CV | Public LB | Private LB|\n|---|---|---|---|\n|EffNet|0.897|0.900|0.898|\n|ViT|0.896|0.898|0.896|\n|NFNet* (no TTA)|0.896|-|-|\n|Resnet|0.895|-|-|\n|EffNet + ViT|0.898|0.903|0.900|\n|EffNet + ViT + Resnet + NFNet*|0.901|0.904|0.899|\n|EffNet + ViT + Resnet + NFNet* + Resnext (No TTA)|0.901|0.902|0.897|\n\n*denotes the use of external data (2019 dataset). All results are based on x4 TTA unless stated otherwise.\n\n#### Final Submission\nThe final model i used for submission consist of a weighted ensemble of 4 models, as shown in the 6th row, with x4 TTA. The funny thing is that a simple average of Effnet with ViT (5th row) gives a better private LB score.\n\n\nAll inferences are done with x4 TTA as follows:\n```\nreturn Compose([\n RandomResizedCrop(CFG.size[CFG.model], CFG.size[CFG.model]), \n HorizontalFlip(p=0.5), \n VerticalFlip(p=0.5),\n Transpose(p=0.5),\n Normalize(\n mean=[0.485, 0.456, 0.406], \n std=[0.229, 0.224, 0.225], \n max_pixel_value=255.0, p=1.0),\n ToTensorV2(p=1.0),\n ])\n```\n\n### Training Settings\n* Validation: Stratified, 5 folds\n* Optimizer: Adam\n* Loss: LabelSmoothLoss (smooth=0.3)\n* Scheduler: CosineWarmRestarts\n* Augmentation: Cutmix + Fmix (25% prob each, independently) with Albumentation\n* Epoch: 15\n\nWell done. Have you done anything special to deal with the huge shakeup?
", "rawMarkdown": "Well done. Have you done anything special to deal with the huge shakeup?", "votes": null }, { "id": "1210052", "postDate": "02/19/2021 07:12:27", "content": "Thanks for sharing and congrats on your medal! :)
", "rawMarkdown": "Thanks for sharing and congrats on your medal! :)", "votes": null }, { "id": "1210081", "postDate": "02/19/2021 07:41:27", "content": "Thank you! Congratz to you too :)
", "rawMarkdown": "Thank you! Congratz to you too :)", "votes": null }, { "id": "1210082", "postDate": "02/19/2021 07:44:20", "content": "Unfortunately i did not have anything special. The only thing i tried to do in hope of minimising the impact is to choose 2 submissions based on highest CV / LB separately. I would love to have a look at the approach behind the 1st place solution. They are immune to the shakeup, remarkable!
", "rawMarkdown": "Unfortunately i did not have anything special. The only thing i tried to do in hope of minimising the impact is to choose 2 submissions based on highest CV / LB separately. I would love to have a look at the approach behind the 1st place solution. They are immune to the shakeup, remarkable!", "votes": null }, { "id": "1210278", "postDate": "02/19/2021 10:09:13", "content": "Great works!
", "rawMarkdown": "Great works!", "votes": null }, { "id": "1210313", "postDate": "02/19/2021 10:45:08", "content": "Thank you for sharing awesome notebooks with the community!
", "rawMarkdown": "Thank you for sharing awesome notebooks with the community!", "votes": null }, { "id": "1210702", "postDate": "02/19/2021 16:07:56", "content": "Yes, that's what I am looking for too. Most teams have suffered from some form of shakeup (either positively or negatively) except the top team.
", "rawMarkdown": "Yes, that's what I am looking for too. Most teams have suffered from some form of shakeup (either positively or negatively) except the top team.", "votes": null }, { "id": "1212300", "postDate": "02/21/2021 05:02:29", "content": "Thank you for sharing your notebook and others code
\nnow it'll be my source of learning using ensemble and tta 👍
Have fun kaggling and learning!
", "rawMarkdown": "Have fun kaggling and learning!", "votes": null } ], "comments": [ { "id": 1210020, "author_name": "yassinealouini", "author_url": "", "post_date": "02/19/2021 06:51:53", "content": "Well done. Have you done anything special to deal with the huge shakeup?
", "votes": null, "replies": [ { "id": 1210082, "author_name": "angqx95", "author_url": "", "post_date": "02/19/2021 07:44:20", "content": "Unfortunately i did not have anything special. The only thing i tried to do in hope of minimising the impact is to choose 2 submissions based on highest CV / LB separately. I would love to have a look at the approach behind the 1st place solution. They are immune to the shakeup, remarkable!
", "votes": null, "replies": [] }, { "id": 1210702, "author_name": "yassinealouini", "author_url": "", "post_date": "02/19/2021 16:07:56", "content": "Yes, that's what I am looking for too. Most teams have suffered from some form of shakeup (either positively or negatively) except the top team.
", "votes": null, "replies": [] } ] }, { "id": 1210052, "author_name": "piantic", "author_url": "", "post_date": "02/19/2021 07:12:27", "content": "Thanks for sharing and congrats on your medal! :)
", "votes": null, "replies": [ { "id": 1210081, "author_name": "angqx95", "author_url": "", "post_date": "02/19/2021 07:41:27", "content": "Thank you! Congratz to you too :)
", "votes": null, "replies": [] } ] }, { "id": 1210278, "author_name": "khyeh0719", "author_url": "", "post_date": "02/19/2021 10:09:13", "content": "Great works!
", "votes": null, "replies": [ { "id": 1210313, "author_name": "angqx95", "author_url": "", "post_date": "02/19/2021 10:45:08", "content": "Thank you for sharing awesome notebooks with the community!
", "votes": null, "replies": [] } ] }, { "id": 1212300, "author_name": "houoinkyoma", "author_url": "", "post_date": "02/21/2021 05:02:29", "content": "Thank you for sharing your notebook and others code
\nnow it'll be my source of learning using ensemble and tta 👍
Have fun kaggling and learning!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1209945": "First and foremost, i would like to sincerely thank the Kaggle community for all the great tips and ideas throughout the competition. \n\nPrior to the competition, i was struggling to understand the concept of averaging folds as a bagging technique, using OOF etc. Until i found [@cdeotte](https://www.kaggle.com/cdeotte) discussion [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614) (Thank you! [@cdeotte](https://www.kaggle.com/cdeotte)) and together with some research, i felt i needed to put it into practice to ensure i had understood the concept. This competition is a good platform for me to apply it. Not only that, i didn't try ensembles before, and never really build a training pipeline for iterative experiments prior to this. I must say i have really learnt a lot from this competition :)\n\nSpecial shoutout to [@khyeh0719](https://www.kaggle.com/khyeh0719) and [@reighns](https://www.kaggle.com/reighns) for their awesome notebooks from which i build my pipeline upon!\nhttps://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-train-amp-aug\nhttps://www.kaggle.com/reighns/complete-and-reusable-pytorch-pipeline\n\n### TLDR;\nMy solution is a weighted ensemble of 4 models: **EfficientNetB4**, **Vit_base16**, **Resnet200d** and **NF-Resnet50**. The models are trained on either with or without external data, with the images augmented in the form of Cutmix and Fmix alongside some basic albumentation augmentations. Inference was done with x4 TTA.\nGithub: https://github.com/aqx95/cassava_kaggle\n\n### Dataset\nI would like to thank [@tahsin](https://www.kaggle.com/tahsin) for sharing with the community the merged 2019/2020 dataset. I used the merged data for training my NFNet. I did a stratified KFold only on the 2020 data source, and combine the 2019 data with the training folds to ensure that the 2019 data does not seep into my validation fold, which may affect my CV if there is a shift in input distribution between 2019 and 2020 data.\n\n### Models\nThe models i experimented with (5 folds):\n* EfficientNet B4 (512 x 512)\n* Vit base16 (384 x 384)\n* Resnet200d (512 x 512)\n* NF-Resnet50 (512 x 512)\n* Resnext50_32x4d (512 x 512)\n\nI first got to know about [VisionTransformers](https://arxiv.org/pdf/2010.11929.pdf) from this competition and i was immediately intrigued by its architecture; to solve vision tasks without convolution layers was just mind blowing to me! I would like to thank [@mobassir](https://www.kaggle.com/mobassir) for introducing the [NFNet](https://arxiv.org/abs/2102.06171), which has quite a decent performance on my local CV. At the time of competing, timm package only has a few pre-trained weights available for transfer learning, and i used NF-Resnet50 (Thank you [@rwightman](https://www.kaggle.com/rwightman))! \n\n####Model Correlation\n\nCombining models with different strengths will usually give the best synergies. Thus, i generate a correlation matrix using out-of-fold predictions to experiment my way into ensembles. I find it interesting that the **ViT model has the least correlation with all other models**. Maybe because of the different underlying operation that creates the receptive field of the image (Convolution vs Attention) ??\n\n\n#### Ensemble\nEfficientNet B4 and NF-Resnet perform the strongest on my local CV without any TTA. However earlier on i was using EfficientNet B4 as a starting baseline since i only knew about NFNets few days before the deadline, and start building my ensemble from there. I cant seem to find any correlation between CV/Public/Private. Im eager to see what are some of the robust strategies that were used.\n\n| Models | CV | Public LB | Private LB|\n|---|---|---|---|\n|EffNet|0.897|0.900|0.898|\n|ViT|0.896|0.898|0.896|\n|NFNet* (no TTA)|0.896|-|-|\n|Resnet|0.895|-|-|\n|EffNet + ViT|0.898|0.903|0.900|\n|EffNet + ViT + Resnet + NFNet*|0.901|0.904|0.899|\n|EffNet + ViT + Resnet + NFNet* + Resnext (No TTA)|0.901|0.902|0.897|\n\n*denotes the use of external data (2019 dataset). All results are based on x4 TTA unless stated otherwise.\n\n#### Final Submission\nThe final model i used for submission consist of a weighted ensemble of 4 models, as shown in the 6th row, with x4 TTA. The funny thing is that a simple average of Effnet with ViT (5th row) gives a better private LB score.\n\n\nAll inferences are done with x4 TTA as follows:\n```\nreturn Compose([\n RandomResizedCrop(CFG.size[CFG.model], CFG.size[CFG.model]), \n HorizontalFlip(p=0.5), \n VerticalFlip(p=0.5),\n Transpose(p=0.5),\n Normalize(\n mean=[0.485, 0.456, 0.406], \n std=[0.229, 0.224, 0.225], \n max_pixel_value=255.0, p=1.0),\n ToTensorV2(p=1.0),\n ])\n```\n\n### Training Settings\n* Validation: Stratified, 5 folds\n* Optimizer: Adam\n* Loss: LabelSmoothLoss (smooth=0.3)\n* Scheduler: CosineWarmRestarts\n* Augmentation: Cutmix + Fmix (25% prob each, independently) with Albumentation\n* Epoch: 15\n\n