This competition is indeed coming to an end. With 3 days remaining(as this post was made), lets overview important things that I found quite useful learning wise. This competition is my first competition, and set up a really good impression of kaggle for me. I believe that this competition is a great competition to not only learn, but to test your knowledge and discover new ideas.
\nNotebooks down below.
\nWith over 3.k+ members attending, 4.8k competitors, with over 85.5k+ submissions…
\nCassava 2019 tournament: A classification problem of this tournament back in 2019. Winning solutions for this classification are
\n1st place: https://www.kaggle.com/c/cassava-disease/discussion/94114
\n2nd place: https://www.kaggle.com/c/cassava-disease/discussion/94112
\n3rd place: https://www.kaggle.com/c/cassava-disease/discussion/94102
These notebooks came in really handy, as they provide useful ideas that we competitors can integrate in.
\nSome useful notebooks in this competition that benefitted me and many others.
\n(Note: These are not all of the helpful notebooks that have helped me, but instead the notebooks I believe started me up the fastest.)
Dimitre's cassava leaf disease training
\nHeroseo's various loss functions
\nKun Hao Yeh's inferencing notebook +training
\nY.Nakama's notebooks
My team and I realized that this dataset marginally have a lot of noisy labels, and so in order to address this problem.. We smoothed the labels and later combined knowledge distillation to feed into our model.
\nWe realized that certain augmentations are really useful of course during training stage.
\nWith 8x TTA in inference giving us a
\nPublic LB 0.904 and Private LB 0.900
\nWhile 5x TTA was a tad bit worse.
Center crop + hue saturation effected out PB LB, alongside with 5 folds for efficientnetB4.
\nBecause the training time was 8 hours max, all my teammates trained separate folds for each and later did the same to resnext for inference. You can check this out on the inference and training notebook down below.
Our best fit model with no distillation was:
\n| Model | \nDistillation | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| Efficientnet B4 | \nNo | \n0.899 | \n- | \n
| Efficientnet B4 | \nYes | \n0.902~0.9003 | \n0.9034 | \n
| Efficientnet B4 + Resnext50 | \nYes | \n0.903~0.904 | \n0.90036 | \n
Augs:
\nTTA:
\nLosses:
\nLucky to not have a bad shakedown!
\nWe did try to implement NFnet, however we didn't have time when the weights were released so we bailed on that. Check it out as it is revolutionary!
Again, these are just a list of the things that came off the top of my head! First competition, and I don't know if it is finalized so I won't be uploading our notebook at the moment for solution. However, I will be uploading it tomorrow! Just making sure that the place my amazing team @teodorbogoeski and @varungadre0910 are finalized! Still don't quite understand how kaggle really works but definitely interested in learning more! If this is finalized, we will be placed 75th on the first competition that my team and I have done! Learned lots from kaggle and this competition! Definitely going to do more… Thanks to many wonderful people who shared their ideas and knowledge in a meaningful manner for novices like me.
\n(Edit): Never mind, turns out I will release the notebooks now.
\nI will be planning to upload notebooks for various datasets… Just waiting for this to finalize just to be sure! Thank you kaggle!
\nTraining: Here
\nInference: Here
\nThanks to my amazing teammates who built these models efficiently! Looking forward to next competitions!
\nUseless note: Did pretty bad on my unit final today but that's alright.. Machine Learning Forever!
", "messages": [ { "id": "1209593", "postDate": "02/19/2021 00:48:20", "content": "This competition is indeed coming to an end. With 3 days remaining(as this post was made), lets overview important things that I found quite useful learning wise. This competition is my first competition, and set up a really good impression of kaggle for me. I believe that this competition is a great competition to not only learn, but to test your knowledge and discover new ideas.
\nNotebooks down below.
\nWith over 3.k+ members attending, 4.8k competitors, with over 85.5k+ submissions…
\nCassava 2019 tournament: A classification problem of this tournament back in 2019. Winning solutions for this classification are
\n1st place: https://www.kaggle.com/c/cassava-disease/discussion/94114
\n2nd place: https://www.kaggle.com/c/cassava-disease/discussion/94112
\n3rd place: https://www.kaggle.com/c/cassava-disease/discussion/94102
These notebooks came in really handy, as they provide useful ideas that we competitors can integrate in.
\nSome useful notebooks in this competition that benefitted me and many others.
\n(Note: These are not all of the helpful notebooks that have helped me, but instead the notebooks I believe started me up the fastest.)
Dimitre's cassava leaf disease training
\nHeroseo's various loss functions
\nKun Hao Yeh's inferencing notebook +training
\nY.Nakama's notebooks
My team and I realized that this dataset marginally have a lot of noisy labels, and so in order to address this problem.. We smoothed the labels and later combined knowledge distillation to feed into our model.
\nWe realized that certain augmentations are really useful of course during training stage.
\nWith 8x TTA in inference giving us a
\nPublic LB 0.904 and Private LB 0.900
\nWhile 5x TTA was a tad bit worse.
Center crop + hue saturation effected out PB LB, alongside with 5 folds for efficientnetB4.
\nBecause the training time was 8 hours max, all my teammates trained separate folds for each and later did the same to resnext for inference. You can check this out on the inference and training notebook down below.
Our best fit model with no distillation was:
\n| Model | \nDistillation | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| Efficientnet B4 | \nNo | \n0.899 | \n- | \n
| Efficientnet B4 | \nYes | \n0.902~0.9003 | \n0.9034 | \n
| Efficientnet B4 + Resnext50 | \nYes | \n0.903~0.904 | \n0.90036 | \n
Augs:
\nTTA:
\nLosses:
\nLucky to not have a bad shakedown!
\nWe did try to implement NFnet, however we didn't have time when the weights were released so we bailed on that. Check it out as it is revolutionary!
Again, these are just a list of the things that came off the top of my head! First competition, and I don't know if it is finalized so I won't be uploading our notebook at the moment for solution. However, I will be uploading it tomorrow! Just making sure that the place my amazing team @teodorbogoeski and @varungadre0910 are finalized! Still don't quite understand how kaggle really works but definitely interested in learning more! If this is finalized, we will be placed 75th on the first competition that my team and I have done! Learned lots from kaggle and this competition! Definitely going to do more… Thanks to many wonderful people who shared their ideas and knowledge in a meaningful manner for novices like me.
\n(Edit): Never mind, turns out I will release the notebooks now.
\nI will be planning to upload notebooks for various datasets… Just waiting for this to finalize just to be sure! Thank you kaggle!
\nTraining: Here
\nInference: Here
\nThanks to my amazing teammates who built these models efficiently! Looking forward to next competitions!
\nUseless note: Did pretty bad on my unit final today but that's alright.. Machine Learning Forever!
", "rawMarkdown": "This competition is indeed coming to an end. With 3 days remaining(as this post was made), lets overview important things that I found quite useful learning wise. This competition is my first competition, and set up a really good impression of kaggle for me. I believe that this competition is a great competition to not only learn, but to test your knowledge and discover new ideas.\n\nNotebooks down below.\n\nWith over 3.k+ members attending, 4.8k competitors, with over 85.5k+ submissions…\n\nCassava 2019 tournament: A classification problem of this tournament back in 2019. Winning solutions for this classification are\n1st place: https://www.kaggle.com/c/cassava-disease/discussion/94114\n2nd place: https://www.kaggle.com/c/cassava-disease/discussion/94112\n3rd place: https://www.kaggle.com/c/cassava-disease/discussion/94102\n\nThese notebooks came in really handy, as they provide useful ideas that we competitors can integrate in.\n\nSome useful notebooks in this competition that benefitted me and many others.\n(Note: These are not all of the helpful notebooks that have helped me, but instead the notebooks I believe started me up the fastest.)\n\n[Dimitre's cassava leaf disease training](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tpu-tensorflow-training)\n[Heroseo's various loss functions](https://www.kaggle.com/piantic/train-cassava-starter-using-various-loss-funcs)\n[Kun Hao Yeh's inferencing notebook +training](https://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-inference-tta)\n[Y.Nakama's notebooks](https://www.kaggle.com/yasufuminakama/cassava-resnext50-32x4d-starter-training)\n\n## Step 1: Knowledge distillation\nMy team and I realized that this dataset marginally have a lot of noisy labels, and so in order to address this problem.. We smoothed the labels and later combined knowledge distillation to feed into our model.\n\n## Step 2:\nWe realized that certain augmentations are really useful of course during training stage.\nWith 8x TTA in inference giving us a \n**Public LB 0.904** and **Private LB 0.900**\nWhile 5x TTA was a tad bit worse.\n\n## Step 3:\nCenter crop + hue saturation effected out PB LB, alongside with 5 folds for efficientnetB4.\nBecause the training time was 8 hours max, all my teammates trained separate folds for each and later did the same to resnext for inference. You can check this out on the inference and training notebook down below.\n\n## Efficientnet B4 + Resnext:\n\nOur best fit model with no distillation was:\n\n| Model|Distillation| Public LB|Private LB|\n| --- | --- | --- | --- |\n| Efficientnet B4| No |0.899 |-|\n| Efficientnet B4 | Yes|0.902~0.9003 | 0.9034 |\n|Efficientnet B4 + Resnext50|Yes|0.903~0.904| 0.90036|\n\n## Experimentations\n**Augs**:\n- Fmix\n- SnapMix\n- Training: Hue Saturation + Coarse dropout + Random Brightness\n- Inference: Standard TTA augs\n\n\n**TTA:**\n- Light augs(only flips): 4~8x TTA \n- Standard augs: 5~8x TTA\n\n\n**Losses**:\n- Taylor crossentropy + smoothening(best)\n- Bi-tempered loss\n- Categorical crossentropy\n\n## Conclusion\nLucky to not have a bad shakedown!\nWe did try to implement NFnet, however we didn't have time when the weights were released so we bailed on that. Check it out as it is revolutionary!\n\nAgain, these are just a list of the things that came off the top of my head! First competition, and I don't know if it is finalized so I won't be uploading our notebook at the moment for solution. However, I will be uploading it tomorrow! Just making sure that the place my amazing team @teodorbogoeski and @varungadre0910 are finalized! Still don't quite understand how kaggle really works but definitely interested in learning more! If this is finalized, we will be placed 75th on the first competition that my team and I have done! Learned lots from kaggle and this competition! Definitely going to do more… Thanks to many wonderful people who shared their ideas and knowledge in a meaningful manner for novices like me. \n\n(Edit): Never mind, turns out I will release the notebooks now.\n\nI will be planning to upload notebooks for various datasets… Just waiting for this to finalize just to be sure! Thank you kaggle!\n\nTraining: [Here](https://www.kaggle.com/andyjianzhou/cassava-training-fastai-efficnetnet-distill)\n\nInference: [Here](https://www.kaggle.com/andyjianzhou/cassava-inferencing-efficientnetb4-resnext50?scriptVersionId=54404877)\n\nThanks to my amazing teammates who built these models efficiently! Looking forward to next competitions! \n\nUseless note: Did pretty bad on my unit final today but that's alright.. Machine Learning Forever!", "votes": null }, { "id": "1209647", "postDate": "02/19/2021 01:50:49", "content": "Welcome. Great job earning silver in your first comp!
", "rawMarkdown": "Welcome. Great job earning silver in your first comp!", "votes": null }, { "id": "1209773", "postDate": "02/19/2021 03:24:03", "content": "Congrats on you and your team to get silver medal!
\n@andyjianzhou
Haha thank you! Couldn't do it without your amazing notebooks.
", "rawMarkdown": "Haha thank you! Couldn't do it without your amazing notebooks.", "votes": null }, { "id": "1209804", "postDate": "02/19/2021 03:44:37", "content": "Thank you Chris! All of your notebooks are really useful. Looked at possibly all your notebooks to learn. Thanks a lot for improving the kaggle community along with many others.
", "rawMarkdown": "Thank you Chris! All of your notebooks are really useful. Looked at possibly all your notebooks to learn. Thanks a lot for improving the kaggle community along with many others.", "votes": null }, { "id": "1209874", "postDate": "02/19/2021 04:48:15", "content": "@piantic Learnt a lot from your notebooks and discussions.
", "rawMarkdown": "piantic Learnt a lot from your notebooks and discussions.", "votes": null } ], "comments": [ { "id": 1209647, "author_name": "cdeotte", "author_url": "", "post_date": "02/19/2021 01:50:49", "content": "Welcome. Great job earning silver in your first comp!
", "votes": null, "replies": [ { "id": 1209804, "author_name": "andyjianzhou", "author_url": "", "post_date": "02/19/2021 03:44:37", "content": "Thank you Chris! All of your notebooks are really useful. Looked at possibly all your notebooks to learn. Thanks a lot for improving the kaggle community along with many others.
", "votes": null, "replies": [] } ] }, { "id": 1209773, "author_name": "piantic", "author_url": "", "post_date": "02/19/2021 03:24:03", "content": "Congrats on you and your team to get silver medal!
\n@andyjianzhou
Haha thank you! Couldn't do it without your amazing notebooks.
", "votes": null, "replies": [] }, { "id": 1209874, "author_name": "varungadre0910", "author_url": "", "post_date": "02/19/2021 04:48:15", "content": "@piantic Learnt a lot from your notebooks and discussions.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1209593": "This competition is indeed coming to an end. With 3 days remaining(as this post was made), lets overview important things that I found quite useful learning wise. This competition is my first competition, and set up a really good impression of kaggle for me. I believe that this competition is a great competition to not only learn, but to test your knowledge and discover new ideas.\n\nNotebooks down below.\n\nWith over 3.k+ members attending, 4.8k competitors, with over 85.5k+ submissions…\n\nCassava 2019 tournament: A classification problem of this tournament back in 2019. Winning solutions for this classification are\n1st place: https://www.kaggle.com/c/cassava-disease/discussion/94114\n2nd place: https://www.kaggle.com/c/cassava-disease/discussion/94112\n3rd place: https://www.kaggle.com/c/cassava-disease/discussion/94102\n\nThese notebooks came in really handy, as they provide useful ideas that we competitors can integrate in.\n\nSome useful notebooks in this competition that benefitted me and many others.\n(Note: These are not all of the helpful notebooks that have helped me, but instead the notebooks I believe started me up the fastest.)\n\n[Dimitre's cassava leaf disease training](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tpu-tensorflow-training)\n[Heroseo's various loss functions](https://www.kaggle.com/piantic/train-cassava-starter-using-various-loss-funcs)\n[Kun Hao Yeh's inferencing notebook +training](https://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-inference-tta)\n[Y.Nakama's notebooks](https://www.kaggle.com/yasufuminakama/cassava-resnext50-32x4d-starter-training)\n\n## Step 1: Knowledge distillation\nMy team and I realized that this dataset marginally have a lot of noisy labels, and so in order to address this problem.. We smoothed the labels and later combined knowledge distillation to feed into our model.\n\n## Step 2:\nWe realized that certain augmentations are really useful of course during training stage.\nWith 8x TTA in inference giving us a \n**Public LB 0.904** and **Private LB 0.900**\nWhile 5x TTA was a tad bit worse.\n\n## Step 3:\nCenter crop + hue saturation effected out PB LB, alongside with 5 folds for efficientnetB4.\nBecause the training time was 8 hours max, all my teammates trained separate folds for each and later did the same to resnext for inference. You can check this out on the inference and training notebook down below.\n\n## Efficientnet B4 + Resnext:\n\nOur best fit model with no distillation was:\n\n| Model|Distillation| Public LB|Private LB|\n| --- | --- | --- | --- |\n| Efficientnet B4| No |0.899 |-|\n| Efficientnet B4 | Yes|0.902~0.9003 | 0.9034 |\n|Efficientnet B4 + Resnext50|Yes|0.903~0.904| 0.90036|\n\n## Experimentations\n**Augs**:\n- Fmix\n- SnapMix\n- Training: Hue Saturation + Coarse dropout + Random Brightness\n- Inference: Standard TTA augs\n\n\n**TTA:**\n- Light augs(only flips): 4~8x TTA \n- Standard augs: 5~8x TTA\n\n\n**Losses**:\n- Taylor crossentropy + smoothening(best)\n- Bi-tempered loss\n- Categorical crossentropy\n\n## Conclusion\nLucky to not have a bad shakedown!\nWe did try to implement NFnet, however we didn't have time when the weights were released so we bailed on that. Check it out as it is revolutionary!\n\nAgain, these are just a list of the things that came off the top of my head! First competition, and I don't know if it is finalized so I won't be uploading our notebook at the moment for solution. However, I will be uploading it tomorrow! Just making sure that the place my amazing team @teodorbogoeski and @varungadre0910 are finalized! Still don't quite understand how kaggle really works but definitely interested in learning more! If this is finalized, we will be placed 75th on the first competition that my team and I have done! Learned lots from kaggle and this competition! Definitely going to do more… Thanks to many wonderful people who shared their ideas and knowledge in a meaningful manner for novices like me. \n\n(Edit): Never mind, turns out I will release the notebooks now.\n\nI will be planning to upload notebooks for various datasets… Just waiting for this to finalize just to be sure! Thank you kaggle!\n\nTraining: [Here](https://www.kaggle.com/andyjianzhou/cassava-training-fastai-efficnetnet-distill)\n\nInference: [Here](https://www.kaggle.com/andyjianzhou/cassava-inferencing-efficientnetb4-resnext50?scriptVersionId=54404877)\n\nThanks to my amazing teammates who built these models efficiently! Looking forward to next competitions! \n\nUseless note: Did pretty bad on my unit final today but that's alright.. Machine Learning Forever!", "1209647": "Welcome. Great job earning silver in your first comp!", "1209773": "Congrats on you and your team to get silver medal!\n@andyjianzhou", "1209801": "Haha thank you! Couldn't do it without your amazing notebooks.", "1209804": "Thank you Chris! All of your notebooks are really useful. Looked at possibly all your notebooks to learn. Thanks a lot for improving the kaggle community along with many others.", "1209874": "piantic Learnt a lot from your notebooks and discussions." }, "source": "meta" }