Wow what a competition! We jumped from our public leaderboard position of 1138th place to the private position of 36th place, giving us a silver medal. Our solution is quite simple, we didn't utilize any techniques to deal with noise. Did I mention our submissions finished just minutes before the deadline?
\nThe components of our silver medal solution:
\nEfficientNet-B7:
\nThis is basically the same as this kernel but with an EfficientNet-B7 model, 20 epochs, and image size of 512x512. 5-fold CV of:
\nThis model was submitted with a 3x TTA of the following augmentations:
RandomResizedCrop, \n Transpose(p=0.5), \n HorizontalFlip(p=0.5), \n VerticalFlip(p=0.5), \n RandomBrightnessContrast(\n brightness_limit=(-0.1,0.1), \n contrast_limit=(-0.1, 0.1), \n p=0.5\n ),\nWe will note that during the private submissions leak (~1.5 months ago), we noticed this model leaderboard submission was 9th place.
\nEfficientNet-B3a:
\nTraining for this model was inspired by the training procedure in this kernel (but without label correction). Basically, the EfficientNet-B3a (a version of EfficientNet-B3 with weights trained by Ross Wightman) with the model body frozen and the fastai model head unfrozen was
\ntrained for 3 epochs. Following this, the whole model was unfrozen and trained for 10 epochs. 512x512 images were used, batch size of 32. The augmentations used were as follows:
Dihedral(p=0.5),\n Rotate(p=0.5, max_deg=45),\n RandomErasing(p=0.5, sl=0.05, sh=0.05, min_aspect=1., max_count=15),\n Brightness(p=0.5, max_lighting=0.3, batch=False),\n Hue(p=0.5, max_hue=0.1, batch=False),\n Saturation(p=0.5, max_lighting=0.1, batch=False),\n RandomResizedCropGPU(size, min_scale=0.4),\n CutMix()\nLabel smoothing loss function with Ranger optimizer was used. The CutMix+label smoothing was quite helpful, and we noticed a significant CV boost because of this. The model with the best validation loss was saved. This resulted in a model with a 5-fold CV of: 0.8914
\nThe model was submitted with 5x TTA with the training augmentations (learn.tta(n=5,beta=0)).
SE-ResNext50:
\nI just changed the model from the previous pipeline to SE-ResNext50. A 5-fold CV of: 0.8938
\nThese three models were ensembled and submitted.
\nPublic LB: 0.898
\nPrivate LB: 0.900
We also had another SE-ResNext50 model with different augmentations, LR schedule, etc. along with stochastic weight averaging during the end of training. This was submitted with the EfficientNet-B3a and the EfficientNet-B7 as an ensemble, and this got Public LB: 0.897, Private LB: 0.901 (potential gold territory), but this was not selected unfortunately. However, we are grateful that we even managed to make a 1135 rank jump!
\nA few of the other things we tried:
\n - Tried Bi-tempered logistic loss, observed no CV improvement
\n - Tried progressive label correction, observed no CV improvement
\n - Tried pseudolabeling on the test set, it was a last minute idea and public LB was 0.139 so likely an error in implementation
Reflections:
\n - This competition was quite interesting, even though the dataset was quite messy and filled with errors
\n - Even with many errors, the model training was robust, leading to decent performance on the CV and private leaderboard
\n - While we all started out quite enthusiastic regarding this competition, other tasks/duties took higher priority for many of us, and only a couple of us dedicated significant time to it. I wonder what the situation would be if all team members could dedicate more time to the competition. Would we reach an even higher place? Or would we overfit to the leaderboard?
Thanks to my great teammates, especially @tanulsingh077 and @abhishek!
", "messages": [ { "id": "1209697", "postDate": "02/19/2021 02:34:04", "content": "Wow what a competition! We jumped from our public leaderboard position of 1138th place to the private position of 36th place, giving us a silver medal. Our solution is quite simple, we didn't utilize any techniques to deal with noise. Did I mention our submissions finished just minutes before the deadline?
\nThe components of our silver medal solution:
\nEfficientNet-B7:
\nThis is basically the same as this kernel but with an EfficientNet-B7 model, 20 epochs, and image size of 512x512. 5-fold CV of:
\nThis model was submitted with a 3x TTA of the following augmentations:
RandomResizedCrop, \n Transpose(p=0.5), \n HorizontalFlip(p=0.5), \n VerticalFlip(p=0.5), \n RandomBrightnessContrast(\n brightness_limit=(-0.1,0.1), \n contrast_limit=(-0.1, 0.1), \n p=0.5\n ),\nWe will note that during the private submissions leak (~1.5 months ago), we noticed this model leaderboard submission was 9th place.
\nEfficientNet-B3a:
\nTraining for this model was inspired by the training procedure in this kernel (but without label correction). Basically, the EfficientNet-B3a (a version of EfficientNet-B3 with weights trained by Ross Wightman) with the model body frozen and the fastai model head unfrozen was
\ntrained for 3 epochs. Following this, the whole model was unfrozen and trained for 10 epochs. 512x512 images were used, batch size of 32. The augmentations used were as follows:
Dihedral(p=0.5),\n Rotate(p=0.5, max_deg=45),\n RandomErasing(p=0.5, sl=0.05, sh=0.05, min_aspect=1., max_count=15),\n Brightness(p=0.5, max_lighting=0.3, batch=False),\n Hue(p=0.5, max_hue=0.1, batch=False),\n Saturation(p=0.5, max_lighting=0.1, batch=False),\n RandomResizedCropGPU(size, min_scale=0.4),\n CutMix()\nLabel smoothing loss function with Ranger optimizer was used. The CutMix+label smoothing was quite helpful, and we noticed a significant CV boost because of this. The model with the best validation loss was saved. This resulted in a model with a 5-fold CV of: 0.8914
\nThe model was submitted with 5x TTA with the training augmentations (learn.tta(n=5,beta=0)).
SE-ResNext50:
\nI just changed the model from the previous pipeline to SE-ResNext50. A 5-fold CV of: 0.8938
\nThese three models were ensembled and submitted.
\nPublic LB: 0.898
\nPrivate LB: 0.900
We also had another SE-ResNext50 model with different augmentations, LR schedule, etc. along with stochastic weight averaging during the end of training. This was submitted with the EfficientNet-B3a and the EfficientNet-B7 as an ensemble, and this got Public LB: 0.897, Private LB: 0.901 (potential gold territory), but this was not selected unfortunately. However, we are grateful that we even managed to make a 1135 rank jump!
\nA few of the other things we tried:
\n - Tried Bi-tempered logistic loss, observed no CV improvement
\n - Tried progressive label correction, observed no CV improvement
\n - Tried pseudolabeling on the test set, it was a last minute idea and public LB was 0.139 so likely an error in implementation
Reflections:
\n - This competition was quite interesting, even though the dataset was quite messy and filled with errors
\n - Even with many errors, the model training was robust, leading to decent performance on the CV and private leaderboard
\n - While we all started out quite enthusiastic regarding this competition, other tasks/duties took higher priority for many of us, and only a couple of us dedicated significant time to it. I wonder what the situation would be if all team members could dedicate more time to the competition. Would we reach an even higher place? Or would we overfit to the leaderboard?
Thanks to my great teammates, especially @tanulsingh077 and @abhishek!
", "rawMarkdown": "Wow what a competition! We jumped from our public leaderboard position of 1138th place to the private position of 36th place, giving us a silver medal. Our solution is quite simple, we didn't utilize any techniques to deal with noise. Did I mention our submissions finished just minutes before the deadline?\n\n# Solution:\n\nThe components of our silver medal solution:\n\n1. EfficientNet-B7\n2. EfficientNet-B3a from timm library\n3. SE-ResNext50\n\n**EfficientNet-B7:**\nThis is basically the same as [this kernel](https://www.kaggle.com/abhishek/tez-faster-and-easier-training-for-leaf-detection) but with an EfficientNet-B7 model, 20 epochs, and image size of 512x512. 5-fold CV of:\nThis model was submitted with a 3x TTA of the following augmentations:\n```\n RandomResizedCrop, \n Transpose(p=0.5), \n HorizontalFlip(p=0.5), \n VerticalFlip(p=0.5), \n RandomBrightnessContrast(\n brightness_limit=(-0.1,0.1), \n contrast_limit=(-0.1, 0.1), \n p=0.5\n ),\n```\nWe will note that during the private submissions leak (~1.5 months ago), we noticed this model leaderboard submission was 9th place. \n\t\n**EfficientNet-B3a:**\n\nTraining for this model was inspired by the training procedure in [this kernel](https://www.kaggle.com/keremt/progressive-label-correction-paper-implementation) (but without label correction). Basically, the EfficientNet-B3a (a version of EfficientNet-B3 with weights trained by Ross Wightman) with the model body frozen and the fastai model head unfrozen was \ntrained for 3 epochs. Following this, the whole model was unfrozen and trained for 10 epochs. 512x512 images were used, batch size of 32. The augmentations used were as follows:\n```\n Dihedral(p=0.5),\n Rotate(p=0.5, max_deg=45),\n RandomErasing(p=0.5, sl=0.05, sh=0.05, min_aspect=1., max_count=15),\n Brightness(p=0.5, max_lighting=0.3, batch=False),\n Hue(p=0.5, max_hue=0.1, batch=False),\n Saturation(p=0.5, max_lighting=0.1, batch=False),\n RandomResizedCropGPU(size, min_scale=0.4),\n CutMix()\n```\nLabel smoothing loss function with Ranger optimizer was used. The CutMix+label smoothing was quite helpful, and we noticed a significant CV boost because of this. The model with the best validation loss was saved. This resulted in a model with a 5-fold CV of: 0.8914\nThe model was submitted with 5x TTA with the training augmentations (`learn.tta(n=5,beta=0)`).\n\n**SE-ResNext50:**\n\nI just changed the model from the previous pipeline to SE-ResNext50. A 5-fold CV of: 0.8938\n\nThese three models were ensembled and submitted. \n\nPublic LB: 0.898\nPrivate LB: 0.900\n\nWe also had another SE-ResNext50 model with different augmentations, LR schedule, etc. along with stochastic weight averaging during the end of training. This was submitted with the EfficientNet-B3a and the EfficientNet-B7 as an ensemble, and this got Public LB: 0.897, Private LB: 0.901 (potential gold territory), but this was not selected unfortunately. However, we are grateful that we even managed to make a 1135 rank jump!\n\n**A few of the other things we tried:**\n\t- Tried Bi-tempered logistic loss, observed no CV improvement\n\t- Tried progressive label correction, observed no CV improvement\n\t- Tried pseudolabeling on the test set, it was a last minute idea and public LB was 0.139 so likely an error in implementation\n\n**Reflections:**\n\t- This competition was quite interesting, even though the dataset was quite messy and filled with errors\n\t- Even with many errors, the model training was robust, leading to decent performance on the CV and private leaderboard\n - While we all started out quite enthusiastic regarding this competition, other tasks/duties took higher priority for many of us, and only a couple of us dedicated significant time to it. I wonder what the situation would be if all team members could dedicate more time to the competition. Would we reach an even higher place? Or would we overfit to the leaderboard? \n\nThanks to my great teammates, especially @tanulsingh077 and @abhishek!", "votes": null }, { "id": "1209733", "postDate": "02/19/2021 02:55:17", "content": "great work!
", "rawMarkdown": "great work!", "votes": null }, { "id": "1209792", "postDate": "02/19/2021 03:36:36", "content": "Thanks for sharing and congrats on 40th place and silver medal.
\ngreat job. :) @tanlikesmath, @tanulsingh077
It was a great competition indeed , I really learned a lot , especially from @piantic , who was kind enough to share everything . I have summarized every experiment of mine except the agus in the kernel here :
\nhttps://www.kaggle.com/tanulsingh077/cassava-general-model-gpu
This is my second competition with @abhishek and both times we have come from the late 1100's position to silver medal zone , but this is special for me as it is my best kaggle finish ever till date .
\nMost of the people in the high public zone were training on 2019+20 data , we tried a different approach we used 2019+20 data for training but we validated only on 20 data and somehow our cv and oof_score both decreased , that to me was the key as then we decided to use only 2020 data as the test data might also contain the noises , hence we didn't try hard to eliminate the noises
\nCall us lucky or unlucky we also had 0.901 in our final ensemble but we didn't select it , but we are exceptionally happy for the turn of events
\nWe were thinking of anything in between 100-200 on private but this is huge , a very big thanks to @ilovescience and @abhishek
\nWish I can team up more with you guys
", "rawMarkdown": "It was a great competition indeed , I really learned a lot , especially from @piantic , who was kind enough to share everything . I have summarized every experiment of mine except the agus in the kernel here :\nhttps://www.kaggle.com/tanulsingh077/cassava-general-model-gpu\n\nThis is my second competition with @abhishek and both times we have come from the late 1100's position to silver medal zone , but this is special for me as it is my best kaggle finish ever till date .\n\nMost of the people in the high public zone were training on 2019+20 data , we tried a different approach we used 2019+20 data for training but we validated only on 20 data and somehow our cv and oof_score both decreased , that to me was the key as then we decided to use only 2020 data as the test data might also contain the noises , hence we didn't try hard to eliminate the noises\n\nCall us lucky or unlucky we also had 0.901 in our final ensemble but we didn't select it , but we are exceptionally happy for the turn of events\n\nWe were thinking of anything in between 100-200 on private but this is huge , a very big thanks to @ilovescience and @abhishek \n\nWish I can team up more with you guys", "votes": null }, { "id": "1209830", "postDate": "02/19/2021 04:04:54", "content": "Thanks bro, I have learned a lot from you
", "rawMarkdown": "Thanks bro, I have learned a lot from you", "votes": null }, { "id": "1210044", "postDate": "02/19/2021 07:08:10", "content": "good stuff ;)
", "rawMarkdown": "good stuff ;)", "votes": null }, { "id": "1210594", "postDate": "02/19/2021 14:42:23", "content": "Congratulations,
\nIts great feeling when u realize model's low Public Leaderboard score is actually getting the most of the problem stated !
Such great work. Congrats on silver medal.
", "rawMarkdown": "Such great work. Congrats on silver medal.", "votes": null } ], "comments": [ { "id": 1209733, "author_name": "wonjunpark", "author_url": "", "post_date": "02/19/2021 02:55:17", "content": "great work!
", "votes": null, "replies": [] }, { "id": 1209792, "author_name": "piantic", "author_url": "", "post_date": "02/19/2021 03:36:36", "content": "Thanks for sharing and congrats on 40th place and silver medal.
\ngreat job. :) @tanlikesmath, @tanulsingh077
Thanks bro, I have learned a lot from you
", "votes": null, "replies": [] } ] }, { "id": 1209828, "author_name": "tanulsingh077", "author_url": "", "post_date": "02/19/2021 04:03:54", "content": "It was a great competition indeed , I really learned a lot , especially from @piantic , who was kind enough to share everything . I have summarized every experiment of mine except the agus in the kernel here :
\nhttps://www.kaggle.com/tanulsingh077/cassava-general-model-gpu
This is my second competition with @abhishek and both times we have come from the late 1100's position to silver medal zone , but this is special for me as it is my best kaggle finish ever till date .
\nMost of the people in the high public zone were training on 2019+20 data , we tried a different approach we used 2019+20 data for training but we validated only on 20 data and somehow our cv and oof_score both decreased , that to me was the key as then we decided to use only 2020 data as the test data might also contain the noises , hence we didn't try hard to eliminate the noises
\nCall us lucky or unlucky we also had 0.901 in our final ensemble but we didn't select it , but we are exceptionally happy for the turn of events
\nWe were thinking of anything in between 100-200 on private but this is huge , a very big thanks to @ilovescience and @abhishek
\nWish I can team up more with you guys
", "votes": null, "replies": [] }, { "id": 1210044, "author_name": "abhishek", "author_url": "", "post_date": "02/19/2021 07:08:10", "content": "good stuff ;)
", "votes": null, "replies": [] }, { "id": 1210594, "author_name": "akhileshdkapse", "author_url": "", "post_date": "02/19/2021 14:42:23", "content": "Congratulations,
\nIts great feeling when u realize model's low Public Leaderboard score is actually getting the most of the problem stated !
Such great work. Congrats on silver medal.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1209697": "Wow what a competition! We jumped from our public leaderboard position of 1138th place to the private position of 36th place, giving us a silver medal. Our solution is quite simple, we didn't utilize any techniques to deal with noise. Did I mention our submissions finished just minutes before the deadline?\n\n# Solution:\n\nThe components of our silver medal solution:\n\n1. EfficientNet-B7\n2. EfficientNet-B3a from timm library\n3. SE-ResNext50\n\n**EfficientNet-B7:**\nThis is basically the same as [this kernel](https://www.kaggle.com/abhishek/tez-faster-and-easier-training-for-leaf-detection) but with an EfficientNet-B7 model, 20 epochs, and image size of 512x512. 5-fold CV of:\nThis model was submitted with a 3x TTA of the following augmentations:\n```\n RandomResizedCrop, \n Transpose(p=0.5), \n HorizontalFlip(p=0.5), \n VerticalFlip(p=0.5), \n RandomBrightnessContrast(\n brightness_limit=(-0.1,0.1), \n contrast_limit=(-0.1, 0.1), \n p=0.5\n ),\n```\nWe will note that during the private submissions leak (~1.5 months ago), we noticed this model leaderboard submission was 9th place. \n\t\n**EfficientNet-B3a:**\n\nTraining for this model was inspired by the training procedure in [this kernel](https://www.kaggle.com/keremt/progressive-label-correction-paper-implementation) (but without label correction). Basically, the EfficientNet-B3a (a version of EfficientNet-B3 with weights trained by Ross Wightman) with the model body frozen and the fastai model head unfrozen was \ntrained for 3 epochs. Following this, the whole model was unfrozen and trained for 10 epochs. 512x512 images were used, batch size of 32. The augmentations used were as follows:\n```\n Dihedral(p=0.5),\n Rotate(p=0.5, max_deg=45),\n RandomErasing(p=0.5, sl=0.05, sh=0.05, min_aspect=1., max_count=15),\n Brightness(p=0.5, max_lighting=0.3, batch=False),\n Hue(p=0.5, max_hue=0.1, batch=False),\n Saturation(p=0.5, max_lighting=0.1, batch=False),\n RandomResizedCropGPU(size, min_scale=0.4),\n CutMix()\n```\nLabel smoothing loss function with Ranger optimizer was used. The CutMix+label smoothing was quite helpful, and we noticed a significant CV boost because of this. The model with the best validation loss was saved. This resulted in a model with a 5-fold CV of: 0.8914\nThe model was submitted with 5x TTA with the training augmentations (`learn.tta(n=5,beta=0)`).\n\n**SE-ResNext50:**\n\nI just changed the model from the previous pipeline to SE-ResNext50. A 5-fold CV of: 0.8938\n\nThese three models were ensembled and submitted. \n\nPublic LB: 0.898\nPrivate LB: 0.900\n\nWe also had another SE-ResNext50 model with different augmentations, LR schedule, etc. along with stochastic weight averaging during the end of training. This was submitted with the EfficientNet-B3a and the EfficientNet-B7 as an ensemble, and this got Public LB: 0.897, Private LB: 0.901 (potential gold territory), but this was not selected unfortunately. However, we are grateful that we even managed to make a 1135 rank jump!\n\n**A few of the other things we tried:**\n\t- Tried Bi-tempered logistic loss, observed no CV improvement\n\t- Tried progressive label correction, observed no CV improvement\n\t- Tried pseudolabeling on the test set, it was a last minute idea and public LB was 0.139 so likely an error in implementation\n\n**Reflections:**\n\t- This competition was quite interesting, even though the dataset was quite messy and filled with errors\n\t- Even with many errors, the model training was robust, leading to decent performance on the CV and private leaderboard\n - While we all started out quite enthusiastic regarding this competition, other tasks/duties took higher priority for many of us, and only a couple of us dedicated significant time to it. I wonder what the situation would be if all team members could dedicate more time to the competition. Would we reach an even higher place? Or would we overfit to the leaderboard? \n\nThanks to my great teammates, especially @tanulsingh077 and @abhishek!", "1209733": "great work!", "1209792": "Thanks for sharing and congrats on 40th place and silver medal.\ngreat job. :) @tanlikesmath, @tanulsingh077", "1209828": "It was a great competition indeed , I really learned a lot , especially from @piantic , who was kind enough to share everything . I have summarized every experiment of mine except the agus in the kernel here :\nhttps://www.kaggle.com/tanulsingh077/cassava-general-model-gpu\n\nThis is my second competition with @abhishek and both times we have come from the late 1100's position to silver medal zone , but this is special for me as it is my best kaggle finish ever till date .\n\nMost of the people in the high public zone were training on 2019+20 data , we tried a different approach we used 2019+20 data for training but we validated only on 20 data and somehow our cv and oof_score both decreased , that to me was the key as then we decided to use only 2020 data as the test data might also contain the noises , hence we didn't try hard to eliminate the noises\n\nCall us lucky or unlucky we also had 0.901 in our final ensemble but we didn't select it , but we are exceptionally happy for the turn of events\n\nWe were thinking of anything in between 100-200 on private but this is huge , a very big thanks to @ilovescience and @abhishek \n\nWish I can team up more with you guys", "1209830": "Thanks bro, I have learned a lot from you", "1210044": "good stuff ;)", "1210594": "Congratulations, \nIts great feeling when u realize model's low Public Leaderboard score is actually getting the most of the problem stated !", "1224695": "Such great work. Congrats on silver medal." }, "source": "meta" }