\n
![\"Final](\"https://i.ibb.co/fCdNjTY/1stplace.png\")
Our overall strategy was to test as many models as possible and spend less time on fine-tuning. The goal was to have many diverse models for ensembling rather than some highly tuned ones.
\nIn the end, we had tried a variety of different architectures (e.g., all EfficientNet architectures, Resnet, ResNext, Xception, ViT, DeiT, Inception and MobileNet) while working with different pre-trained weights (trained e.g. on Imagenet, NoisyStudent, Plantvillage, iNaturalist…) some of which were available on Tensorflow Hub.
\nOur winning submission was an ensemble of four different models.
\n
The final score on the public leaderboard was 91.36% and 91.32% on the private leaderboard. We opted to turn in this combination as it achieved a higher CV score than other combinations (which sometimes scored slightly better on the public leaderboard). We tested some of the models separately on the leaderboard (public/private): B4: 89.4%/89.5% , MobileNet: 89.5%/89.4%, ViT:~89.0%/88.8%
\nThe others were only evaluated using cross-validation.
\nOverall, we can conclude that the key to victory was the use of CropNet from Tensorflow Hub, as it brought a lot of diversity to our ensemble. Although it did not perform better on the leaderboard as a standalone model than the other models, the ensembles that used this model brought a significant boost on the leaderboard.
\nAt this point I would like to thank hengck23, whose discussion post brought this model to our attention (as well as other pre-trained models for image classification available on Tensorflow Hub).
\nFurthermore, we would like to thank all the participants who have helped us learn a lot in this contest through their contributions here in the board and their published notebooks. Finally, we would also like to thank the Competition Host, who made this exciting competition possible by releasing the data.
\nYou can find our inference code in this notebook: Inference Notebook
\nAnd you can find the according training code in these notebooks:
\n\nDetailed information about the configuration & fine-tuning of our used models:
\nWe used a ResNeXt model of the structure “resnext50_32x4d” with the following configurations:
\nWe used the Vision Transformer Architecture with ImageNet weights (ViT-B/16)
\nWe tried different EfficientNet architectures but finally only used a B4 with NoisyStudent weights:
\nFinally, our ensemble included a pretrained CropNet (MobileNetv3) Model from Tensorflow Hub:
\nFor the ensembling, we experimented with different methods and found that in our case a stacked-mean approach worked best. For this purpose, the class probabilities returned by the models were averaged on several levels and finally the class with the highest probability was returned.
\nOur final submission first averaged the probabilities of the predicted classes of ViT and ResNext. This averaged probability vector was then merged with the predicted probabilities of EfficientnetB4 and CropNet in the second stage. For this purpose, the values were simply summed up.\n
\nAnother solution which also generated good results on the leaderboard was finding weights before calculating the mean of models using an optimization. You can find the code for that in our published notebooks. Generally, we were surprised how stable the solutions with optimized weights were. It turned out that they only had small differences (often +/-0.1%) between our CV scores and the leaderboard score.
\nOne thing which didn’t work out in our use case was an ensemble approach with an additional classifier (Gradient Boosted Trees) stack on top of our models. We did several experiments using also additional features, like e.g., the entropy of the model’s prediction, however we were not able to build a solution which generalized good enough.
\nThanks for reading.
", "messages": [ { "id": "1216990", "postDate": "02/24/2021 17:23:08", "content": "Our overall strategy was to test as many models as possible and spend less time on fine-tuning. The goal was to have many diverse models for ensembling rather than some highly tuned ones.
\nIn the end, we had tried a variety of different architectures (e.g., all EfficientNet architectures, Resnet, ResNext, Xception, ViT, DeiT, Inception and MobileNet) while working with different pre-trained weights (trained e.g. on Imagenet, NoisyStudent, Plantvillage, iNaturalist…) some of which were available on Tensorflow Hub.
\nOur winning submission was an ensemble of four different models.
\n
The final score on the public leaderboard was 91.36% and 91.32% on the private leaderboard. We opted to turn in this combination as it achieved a higher CV score than other combinations (which sometimes scored slightly better on the public leaderboard). We tested some of the models separately on the leaderboard (public/private): B4: 89.4%/89.5% , MobileNet: 89.5%/89.4%, ViT:~89.0%/88.8%
\nThe others were only evaluated using cross-validation.
\nOverall, we can conclude that the key to victory was the use of CropNet from Tensorflow Hub, as it brought a lot of diversity to our ensemble. Although it did not perform better on the leaderboard as a standalone model than the other models, the ensembles that used this model brought a significant boost on the leaderboard.
\nAt this point I would like to thank hengck23, whose discussion post brought this model to our attention (as well as other pre-trained models for image classification available on Tensorflow Hub).
\nFurthermore, we would like to thank all the participants who have helped us learn a lot in this contest through their contributions here in the board and their published notebooks. Finally, we would also like to thank the Competition Host, who made this exciting competition possible by releasing the data.
\nYou can find our inference code in this notebook: Inference Notebook
\nAnd you can find the according training code in these notebooks:
\n\nDetailed information about the configuration & fine-tuning of our used models:
\nWe used a ResNeXt model of the structure “resnext50_32x4d” with the following configurations:
\nWe used the Vision Transformer Architecture with ImageNet weights (ViT-B/16)
\nWe tried different EfficientNet architectures but finally only used a B4 with NoisyStudent weights:
\nFinally, our ensemble included a pretrained CropNet (MobileNetv3) Model from Tensorflow Hub:
\nFor the ensembling, we experimented with different methods and found that in our case a stacked-mean approach worked best. For this purpose, the class probabilities returned by the models were averaged on several levels and finally the class with the highest probability was returned.
\nOur final submission first averaged the probabilities of the predicted classes of ViT and ResNext. This averaged probability vector was then merged with the predicted probabilities of EfficientnetB4 and CropNet in the second stage. For this purpose, the values were simply summed up.\n
\nAnother solution which also generated good results on the leaderboard was finding weights before calculating the mean of models using an optimization. You can find the code for that in our published notebooks. Generally, we were surprised how stable the solutions with optimized weights were. It turned out that they only had small differences (often +/-0.1%) between our CV scores and the leaderboard score.
\nOne thing which didn’t work out in our use case was an ensemble approach with an additional classifier (Gradient Boosted Trees) stack on top of our models. We did several experiments using also additional features, like e.g., the entropy of the model’s prediction, however we were not able to build a solution which generalized good enough.
\nThanks for reading.
", "rawMarkdown": "Our overall strategy was to test as many models as possible and spend less time on fine-tuning. The goal was to have many diverse models for ensembling rather than some highly tuned ones.\n\nIn the end, we had tried a variety of different architectures (e.g., all EfficientNet architectures, Resnet, ResNext, Xception, ViT, DeiT, Inception and MobileNet) while working with different pre-trained weights (trained e.g. on Imagenet, NoisyStudent, Plantvillage, iNaturalist...) some of which were available on Tensorflow Hub. \n\nOur winning submission was an ensemble of four different models.\n\n\nThe final score on the public leaderboard was 91.36% and 91.32% on the private leaderboard. We opted to turn in this combination as it achieved a higher CV score than other combinations (which sometimes scored slightly better on the public leaderboard). We tested some of the models separately on the leaderboard (public/private): B4: 89.4%/89.5% , MobileNet: 89.5%/89.4%, ViT:~89.0%/88.8% \n\nThe others were only evaluated using cross-validation.\n\nOverall, we can conclude that the key to victory was the use of CropNet from Tensorflow Hub, as it brought a lot of diversity to our ensemble. Although it did not perform better on the leaderboard as a standalone model than the other models, the ensembles that used this model brought a significant boost on the leaderboard.\n\nAt this point I would like to thank hengck23, whose [discussion post](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/199276) brought this model to our attention (as well as other pre-trained models for image classification available on Tensorflow Hub).\n\nFurthermore, we would like to thank all the participants who have helped us learn a lot in this contest through their contributions here in the board and their published notebooks. Finally, we would also like to thank the Competition Host, who made this exciting competition possible by releasing the data.\n\nYou can find our inference code in this notebook: Inference Notebook\n\nAnd you can find the according training code in these notebooks:\n\n\nDetailed information about the configuration & fine-tuning of our used models:\n\nWe used a ResNeXt model of the structure “resnext50_32x4d” with the following configurations:\n\nCongratulations, and your team's hard work achieved well earned 1st place.
", "rawMarkdown": "Congratulations, and your team's hard work achieved well earned 1st place.", "votes": null }, { "id": "1217188", "postDate": "02/24/2021 20:25:52", "content": "Congrats on your win! Very strong performance until the very end. If you have, what was your score with only CropNet model?
", "rawMarkdown": "Congrats on your win! Very strong performance until the very end. If you have, what was your score with only CropNet model?", "votes": null }, { "id": "1217336", "postDate": "02/25/2021 00:48:52", "content": "Congratulations on the win! Your solution stood apart from the very beginning and survived the LB shakeup. A couple of questions:
\n1) I dont see any mention of TTA - Did you use any?
\n2) Interesting to combine the pre-trained Cropnet model for ensemble even after it did poorly on the LB - What was the score of just this model on public LB?
\n3) What were the scores of your individual models?
Holy smokes! Instantly pressed on this discussion post! Congrats on your first place win! However, how did you guys test each model scenario? That would take really long wouldn't it? Did you test each model one by one? Or did you do something else that I do not know of? How did you test each model's scenario that fast? Using two stages is really smart as well alongside with the usage of both GPU and TPU. Congrats!
", "rawMarkdown": "Holy smokes! Instantly pressed on this discussion post! Congrats on your first place win! However, how did you guys test each model scenario? That would take really long wouldn't it? Did you test each model one by one? Or did you do something else that I do not know of? How did you test each model's scenario that fast? Using two stages is really smart as well alongside with the usage of both GPU and TPU. Congrats!", "votes": null }, { "id": "1217402", "postDate": "02/25/2021 02:52:42", "content": "Very cool !
", "rawMarkdown": "Very cool !", "votes": null }, { "id": "1217466", "postDate": "02/25/2021 05:22:33", "content": "Congrats on 1st place! @jannish @sebastiangnther @hiarsl
\nI am also really surprised that CV and LB are stable.
I have some questions.
\n1) If the above inference code is the final submission, can we assume that other models have not applied TTA except for Keras inference?
\nViT uses ceptercrop for inference but it is always applied so it seems close to No TTA.
2) Have you ever tried the distillation method? Was the model helpful when performing ensemble optimization?
\nthanks for sharing the code for the training pipelines.
\ncongratulations again!
Thanks :) We uploaded the CropNet once with Centercrop 0.9 and resizing alone and it came up with a score of 89.42 on the private leaderboard at the end
", "rawMarkdown": "Thanks :) We uploaded the CropNet once with Centercrop 0.9 and resizing alone and it came up with a score of 89.42 on the private leaderboard at the end", "votes": null }, { "id": "1217587", "postDate": "02/25/2021 07:12:22", "content": "Thanks for your comment. We used (random) TTA only for the EfficientNetB4, but only light ones like Flipping and Transpose. The score of CropNet on private leaderboard was 89.42 using centercrop and resizing, almost all our single models scored in that range (i.e., B4 ~89.5 and ViT ~88.8) As far as i remember we didn' upload this specific ResNext as single model due to time
", "rawMarkdown": "Thanks for your comment. We used (random) TTA only for the EfficientNetB4, but only light ones like Flipping and Transpose. The score of CropNet on private leaderboard was 89.42 using centercrop and resizing, almost all our single models scored in that range (i.e., B4 ~89.5 and ViT ~88.8) As far as i remember we didn' upload this specific ResNext as single model due to time", "votes": null }, { "id": "1217593", "postDate": "02/25/2021 07:18:01", "content": "Thank you. We have calculated the CV scores for many combinations using the out-of-fold predictions of the different models. We then uploaded the most promising ones. You can find some infos on that in the ensembling notebook
", "rawMarkdown": "Thank you. We have calculated the CV scores for many combinations using the out-of-fold predictions of the different models. We then uploaded the most promising ones. You can find some infos on that in the ensembling [notebook](https://www.kaggle.com/jannish/cassava-leaf-disease-finding-final-ensembles)", "votes": null }, { "id": "1217604", "postDate": "02/25/2021 07:24:14", "content": "Thank you Heroseo. We are glad you found the code helpful! Yes that is the cleaned-up inference code and except for B4 we didn't use random augmentations. Maybe that also made the score more stable on the private leaderboard. To be honest, I never heard of \"distillation method\" before. I'll have to read up on that
", "rawMarkdown": "Thank you Heroseo. We are glad you found the code helpful! Yes that is the cleaned-up inference code and except for B4 we didn't use random augmentations. Maybe that also made the score more stable on the private leaderboard. To be honest, I never heard of \"distillation method\" before. I'll have to read up on that", "votes": null }, { "id": "1217662", "postDate": "02/25/2021 08:30:30", "content": "Congratulations on 1st place, I am happy to see a \"simple\" solution was by far the best for this competition!
\nI underestimated a lot the power of ensembling in this case: it seems like the variety of models you used really killed it.
\nThank you very much for the detailed report and code!
Looks like the secret was to use pretrained CropNet. First and second positions are both used it. A little dissapointed as I was expecting some fancy data scince tricks.
\nBut my congratulations as well guys, good job any way.
", "rawMarkdown": "Looks like the secret was to use pretrained CropNet. First and second positions are both used it. A little dissapointed as I was expecting some fancy data scince tricks.\n\nBut my congratulations as well guys, good job any way.", "votes": null }, { "id": "1217830", "postDate": "02/25/2021 10:54:09", "content": "very good writeup and congrats for getting first!
", "rawMarkdown": "very good writeup and congrats for getting first!", "votes": null }, { "id": "1217832", "postDate": "02/25/2021 10:57:38", "content": "about CropNet.
\ni was wondering if this model is trained on noisy images or clean images.
\nif it was trained on clean images, then it means that the solution to due with noisy train/test is still to get clean train images.
\nit is only mentioned on tf hub \"The training dataset is curated by the Mak-AI team at the Makerere University.\"
\nbut we can test it. we can test cropNet on some images and then compare with manual inspection.
i have a feeling that cropNet is trained on clean images (from my test results and reading the paper)
", "rawMarkdown": "about CropNet.\n\ni was wondering if this model is trained on noisy images or clean images.\n\nif it was trained on clean images, then it means that the solution to due with noisy train/test is still to get clean train images.\n\nit is only mentioned on tf hub \"The training dataset is curated by the Mak-AI team at the Makerere University.\"\nbut we can test it. we can test cropNet on some images and then compare with manual inspection.\n\ni have a feeling that cropNet is trained on clean images (from my test results and reading the paper)", "votes": null }, { "id": "1217848", "postDate": "02/25/2021 11:12:12", "content": "first and third solutions are using transformer ViT. Can we conclude that the transformer did indeed work better than conv net?
", "rawMarkdown": "first and third solutions are using transformer ViT. Can we conclude that the transformer did indeed work better than conv net?", "votes": null }, { "id": "1217928", "postDate": "02/25/2021 12:06:08", "content": "Thanks again for sharing your knowledge here in the discussion board!
", "rawMarkdown": "Thanks again for sharing your knowledge here in the discussion board!", "votes": null }, { "id": "1217931", "postDate": "02/25/2021 12:07:35", "content": "Thanks. Yes, using CropNet in the ensemble helped us to boost our final score, even when considered individually it was not stronger than the other models.
", "rawMarkdown": "Thanks. Yes, using CropNet in the ensemble helped us to boost our final score, even when considered individually it was not stronger than the other models.", "votes": null }, { "id": "1218148", "postDate": "02/25/2021 15:40:12", "content": "Congratulation for getting first! Thank you for sharing with us your solution :)
", "rawMarkdown": "Congratulation for getting first! Thank you for sharing with us your solution :)", "votes": null }, { "id": "1218482", "postDate": "02/25/2021 23:26:42", "content": "interesting. one way to prove this is to use a similar approach for ensemble and use a model trained on clean data (I know people tried to deonoise data) to see if there is a significant boost on LB. My \"clean\" models did pretty badly as a standalone model on the LB but I didnt try an ensemble.
", "rawMarkdown": "interesting. one way to prove this is to use a similar approach for ensemble and use a model trained on clean data (I know people tried to deonoise data) to see if there is a significant boost on LB. My \"clean\" models did pretty badly as a standalone model on the LB but I didnt try an ensemble.", "votes": null }, { "id": "1218483", "postDate": "02/25/2021 23:27:21", "content": "Thanks for the details!
", "rawMarkdown": "Thanks for the details!", "votes": null }, { "id": "1218514", "postDate": "02/26/2021 00:30:13", "content": "Ah alright, thank you for the speedy reply! Thanks for sharing once again!
", "rawMarkdown": "Ah alright, thank you for the speedy reply! Thanks for sharing once again!", "votes": null }, { "id": "1218529", "postDate": "02/26/2021 00:54:50", "content": "Congratulation on 1st place!!
\nI have some questions.
\n1) Why did you use ResNext50 and VIT in Stage 1 model? Is it just ImageNet Pretrained? Or is it because PB or LB performance is high?
\n2) I really enjoyed using differential_evolution to find the optimization weight. But I'm curious why you used 4fold instead of 5fold. And why used mobilenet, b4, vit, resnext instead of mobileenet, b4, vit_resnext. And there's a way of bayesian optimization, is there a reason you used differential_evolution?
\nthanks for sharing the code !!
\nCongratulations!!
Thanks for your congrats. We saw that ResNext and ViT together gave quite good results and we had no improvement in our CV scores by mixing in another model (e.g. B3/B5).
\nI can't remember exactly why we only chose 4 folds for the optimisation, but I think we were just impatient and didn't always want to wait for 5 folds. Besides, we didn't have the impression that the weights were less stable. A look at a bayesian approach would be really interesting: can you recommend a tutorial or notebook for this?
Can you link the paper about CropNet please? I could not find it.
", "rawMarkdown": "Can you link the paper about CropNet please? I could not find it.", "votes": null }, { "id": "1219326", "postDate": "02/26/2021 17:31:35", "content": "Congratulation to your team! Thanks for sharing your solution!
", "rawMarkdown": "Congratulation to your team! Thanks for sharing your solution!", "votes": null }, { "id": "1219913", "postDate": "02/27/2021 11:00:38", "content": "Thanks for Replying!! Bayesian Optimization is optimization strategy when tuning hyperparmeter. When I saw your writing, I thought I could use this to optimize weight.
\nWow, 2nd place team finished the competition with only CropNet. I was expecting it to be higher. Anyway, congrats again!
", "rawMarkdown": "Wow, 2nd place team finished the competition with only CropNet. I was expecting it to be higher. Anyway, congrats again!", "votes": null }, { "id": "1220471", "postDate": "02/28/2021 03:18:44", "content": "Thank you so much!
", "rawMarkdown": "Thank you so much!", "votes": null }, { "id": "1221309", "postDate": "02/28/2021 21:38:53", "content": "I was waiting for this one, congratz and thanks for sharing !
", "rawMarkdown": "I was waiting for this one, congratz and thanks for sharing !", "votes": null }, { "id": "1221875", "postDate": "03/01/2021 11:13:41", "content": "Congrats! Any particular reason you used ReduceLROnPlateau for ResNeXt, cosine annealing for ViT and cosine decay for EfficientNet? Was it just repeated testing with different LR schedulers?
", "rawMarkdown": "Congrats! Any particular reason you used ReduceLROnPlateau for ResNeXt, cosine annealing for ViT and cosine decay for EfficientNet? Was it just repeated testing with different LR schedulers?", "votes": null }, { "id": "1222299", "postDate": "03/01/2021 17:30:15", "content": "Congratulations! Brilliant approach! I like the diversity you introduced into the ensemble.
", "rawMarkdown": "Congratulations! Brilliant approach! I like the diversity you introduced into the ensemble.", "votes": null }, { "id": "1222427", "postDate": "03/01/2021 19:29:52", "content": "@jannish Congratulation to your team!
\nLet me please ask you several questions that seems to be not covered in this thread (sorry if I missed something):
\nThanks.
", "rawMarkdown": "jannish Congratulation to your team!\n\nLet me please ask you several questions that seems to be not covered in this thread (sorry if I missed something):\n1. Did you do any class balancing (via loss sample weights or oversampling)?\n2. Am I correct that you used only center crop instead of random crop for you inference? (except the scheme EfficientNet + B4 with NoisyStudent weights where you used six patches with two versions of central crop)\n3. Did you try hue augmentation?\n4. Did you fix \"submission CSV Not Found error\" using working directory \"/kaggle/working/\"? (your post in this thread: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/206273)\nBecause I have met the same issue (https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/220861)\n5. How did you measure diversity of models in your ensamble (std/entropy of prediction probabilities?)\n\nThanks.", "votes": null }, { "id": "1224384", "postDate": "03/02/2021 17:10:25", "content": "Great idea, Thank you !
", "rawMarkdown": "Great idea, Thank you !", "votes": null }, { "id": "1224589", "postDate": "03/02/2021 21:57:42", "content": "Great write-up! Congratulations on coming 1st!
", "rawMarkdown": "Great write-up! Congratulations on coming 1st!", "votes": null }, { "id": "1226637", "postDate": "03/04/2021 17:51:40", "content": "It must have been taking very long for making predictions using ensemble ?
", "rawMarkdown": "It must have been taking very long for making predictions using ensemble ?", "votes": null }, { "id": "1234512", "postDate": "03/11/2021 10:14:35", "content": "Congratulations on 1st place !
", "rawMarkdown": "Congratulations on 1st place !", "votes": null }, { "id": "1236677", "postDate": "03/13/2021 11:07:28", "content": "Thanks for your relpy and sorry for the late answer:
\nI tried class weights with the standard Keras method but that didn't work well. I don't know exactly why that didn't bring any improvement.
Yes thats correct. We used this patch-approach only for the efficientnet
Yes, we tried it but only in combination with Saturation and Contrast. This did not bring any improvement, which is why we tried to use as few augmentations as possible in the end.
The problem at the beginning was that I wanted to copy the images to another folder via shell command, this threw some kind of error in the submission.
Entropy
Congratulations on your win and thanks for sharing your detailed effort
", "rawMarkdown": "Congratulations on your win and thanks for sharing your detailed effort", "votes": null }, { "id": "1246855", "postDate": "03/21/2021 07:21:55", "content": "@jannish , Great solution! Thanks for sharing. I added it to my collection in \"Data Science with DL & NLP: Advanced Techniques\", section \"Prize Competition Winners: notebooks (kernels) and posts with Magic\".
", "rawMarkdown": "jannish , Great solution! Thanks for sharing. I added it to my collection in [\"Data Science with DL & NLP: Advanced Techniques\"](https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques), section \"Prize Competition Winners: notebooks (kernels) and posts with Magic\".", "votes": null }, { "id": "1364372", "postDate": "06/24/2021 22:29:54", "content": "Thank you so much
", "rawMarkdown": "Thank you so much", "votes": null }, { "id": "1404238", "postDate": "07/29/2021 17:08:00", "content": "Congratulations on the win!
\nthanks for sharing the code for the training pipelines.
Congratulations! Great post.
", "rawMarkdown": "Congratulations! Great post.", "votes": null }, { "id": "1954104", "postDate": "09/25/2022 03:08:43", "content": "Thanks a lot for sharing!
", "rawMarkdown": "Thanks a lot for sharing!", "votes": null }, { "id": "2656981", "postDate": "02/18/2024 07:24:25", "content": "thanks you very much
", "rawMarkdown": "thanks you very much", "votes": null }, { "id": "2969897", "postDate": "08/25/2024 14:18:51", "content": "congrats man!
", "rawMarkdown": "congrats man!", "votes": null } ], "comments": [ { "id": 1217081, "author_name": "returnofsputnik", "author_url": "", "post_date": "02/24/2021 18:27:47", "content": "Congratulations, and your team's hard work achieved well earned 1st place.
", "votes": null, "replies": [] }, { "id": 1217188, "author_name": "snnclsr", "author_url": "", "post_date": "02/24/2021 20:25:52", "content": "Congrats on your win! Very strong performance until the very end. If you have, what was your score with only CropNet model?
", "votes": null, "replies": [ { "id": 1217580, "author_name": "jannish", "author_url": "", "post_date": "02/25/2021 07:06:14", "content": "Thanks :) We uploaded the CropNet once with Centercrop 0.9 and resizing alone and it came up with a score of 89.42 on the private leaderboard at the end
", "votes": null, "replies": [] }, { "id": 1220244, "author_name": "snnclsr", "author_url": "", "post_date": "02/27/2021 18:59:58", "content": "Wow, 2nd place team finished the competition with only CropNet. I was expecting it to be higher. Anyway, congrats again!
", "votes": null, "replies": [] } ] }, { "id": 1217336, "author_name": "trushk", "author_url": "", "post_date": "02/25/2021 00:48:52", "content": "Congratulations on the win! Your solution stood apart from the very beginning and survived the LB shakeup. A couple of questions:
\n1) I dont see any mention of TTA - Did you use any?
\n2) Interesting to combine the pre-trained Cropnet model for ensemble even after it did poorly on the LB - What was the score of just this model on public LB?
\n3) What were the scores of your individual models?
Thanks for your comment. We used (random) TTA only for the EfficientNetB4, but only light ones like Flipping and Transpose. The score of CropNet on private leaderboard was 89.42 using centercrop and resizing, almost all our single models scored in that range (i.e., B4 ~89.5 and ViT ~88.8) As far as i remember we didn' upload this specific ResNext as single model due to time
", "votes": null, "replies": [] }, { "id": 1218483, "author_name": "trushk", "author_url": "", "post_date": "02/25/2021 23:27:21", "content": "Thanks for the details!
", "votes": null, "replies": [] } ] }, { "id": 1217348, "author_name": "andyjianzhou", "author_url": "", "post_date": "02/25/2021 01:16:50", "content": "Holy smokes! Instantly pressed on this discussion post! Congrats on your first place win! However, how did you guys test each model scenario? That would take really long wouldn't it? Did you test each model one by one? Or did you do something else that I do not know of? How did you test each model's scenario that fast? Using two stages is really smart as well alongside with the usage of both GPU and TPU. Congrats!
", "votes": null, "replies": [ { "id": 1217593, "author_name": "jannish", "author_url": "", "post_date": "02/25/2021 07:18:01", "content": "Thank you. We have calculated the CV scores for many combinations using the out-of-fold predictions of the different models. We then uploaded the most promising ones. You can find some infos on that in the ensembling notebook
", "votes": null, "replies": [] }, { "id": 1218514, "author_name": "andyjianzhou", "author_url": "", "post_date": "02/26/2021 00:30:13", "content": "Ah alright, thank you for the speedy reply! Thanks for sharing once again!
", "votes": null, "replies": [] } ] }, { "id": 1217402, "author_name": "majunfu", "author_url": "", "post_date": "02/25/2021 02:52:42", "content": "Very cool !
", "votes": null, "replies": [] }, { "id": 1217466, "author_name": "piantic", "author_url": "", "post_date": "02/25/2021 05:22:33", "content": "Congrats on 1st place! @jannish @sebastiangnther @hiarsl
\nI am also really surprised that CV and LB are stable.
I have some questions.
\n1) If the above inference code is the final submission, can we assume that other models have not applied TTA except for Keras inference?
\nViT uses ceptercrop for inference but it is always applied so it seems close to No TTA.
2) Have you ever tried the distillation method? Was the model helpful when performing ensemble optimization?
\nthanks for sharing the code for the training pipelines.
\ncongratulations again!
Thank you Heroseo. We are glad you found the code helpful! Yes that is the cleaned-up inference code and except for B4 we didn't use random augmentations. Maybe that also made the score more stable on the private leaderboard. To be honest, I never heard of \"distillation method\" before. I'll have to read up on that
", "votes": null, "replies": [] } ] }, { "id": 1217662, "author_name": "mviola", "author_url": "", "post_date": "02/25/2021 08:30:30", "content": "Congratulations on 1st place, I am happy to see a \"simple\" solution was by far the best for this competition!
\nI underestimated a lot the power of ensembling in this case: it seems like the variety of models you used really killed it.
\nThank you very much for the detailed report and code!
Looks like the secret was to use pretrained CropNet. First and second positions are both used it. A little dissapointed as I was expecting some fancy data scince tricks.
\nBut my congratulations as well guys, good job any way.
", "votes": null, "replies": [ { "id": 1217931, "author_name": "jannish", "author_url": "", "post_date": "02/25/2021 12:07:35", "content": "Thanks. Yes, using CropNet in the ensemble helped us to boost our final score, even when considered individually it was not stronger than the other models.
", "votes": null, "replies": [] } ] }, { "id": 1217830, "author_name": "hengck23", "author_url": "", "post_date": "02/25/2021 10:54:09", "content": "very good writeup and congrats for getting first!
", "votes": null, "replies": [ { "id": 1217928, "author_name": "jannish", "author_url": "", "post_date": "02/25/2021 12:06:08", "content": "Thanks again for sharing your knowledge here in the discussion board!
", "votes": null, "replies": [] } ] }, { "id": 1217832, "author_name": "hengck23", "author_url": "", "post_date": "02/25/2021 10:57:38", "content": "about CropNet.
\ni was wondering if this model is trained on noisy images or clean images.
\nif it was trained on clean images, then it means that the solution to due with noisy train/test is still to get clean train images.
\nit is only mentioned on tf hub \"The training dataset is curated by the Mak-AI team at the Makerere University.\"
\nbut we can test it. we can test cropNet on some images and then compare with manual inspection.
i have a feeling that cropNet is trained on clean images (from my test results and reading the paper)
", "votes": null, "replies": [ { "id": 1218482, "author_name": "trushk", "author_url": "", "post_date": "02/25/2021 23:26:42", "content": "interesting. one way to prove this is to use a similar approach for ensemble and use a model trained on clean data (I know people tried to deonoise data) to see if there is a significant boost on LB. My \"clean\" models did pretty badly as a standalone model on the LB but I didnt try an ensemble.
", "votes": null, "replies": [] }, { "id": 1219096, "author_name": "aliabdin1", "author_url": "", "post_date": "02/26/2021 12:52:11", "content": "Can you link the paper about CropNet please? I could not find it.
", "votes": null, "replies": [] } ] }, { "id": 1217848, "author_name": "hengck23", "author_url": "", "post_date": "02/25/2021 11:12:12", "content": "first and third solutions are using transformer ViT. Can we conclude that the transformer did indeed work better than conv net?
", "votes": null, "replies": [] }, { "id": 1218148, "author_name": "angqx95", "author_url": "", "post_date": "02/25/2021 15:40:12", "content": "Congratulation for getting first! Thank you for sharing with us your solution :)
", "votes": null, "replies": [] }, { "id": 1218529, "author_name": "chocozzz", "author_url": "", "post_date": "02/26/2021 00:54:50", "content": "Congratulation on 1st place!!
\nI have some questions.
\n1) Why did you use ResNext50 and VIT in Stage 1 model? Is it just ImageNet Pretrained? Or is it because PB or LB performance is high?
\n2) I really enjoyed using differential_evolution to find the optimization weight. But I'm curious why you used 4fold instead of 5fold. And why used mobilenet, b4, vit, resnext instead of mobileenet, b4, vit_resnext. And there's a way of bayesian optimization, is there a reason you used differential_evolution?
\nthanks for sharing the code !!
\nCongratulations!!
Thanks for your congrats. We saw that ResNext and ViT together gave quite good results and we had no improvement in our CV scores by mixing in another model (e.g. B3/B5).
\nI can't remember exactly why we only chose 4 folds for the optimisation, but I think we were just impatient and didn't always want to wait for 5 folds. Besides, we didn't have the impression that the weights were less stable. A look at a bayesian approach would be really interesting: can you recommend a tutorial or notebook for this?
Thanks for Replying!! Bayesian Optimization is optimization strategy when tuning hyperparmeter. When I saw your writing, I thought I could use this to optimize weight.
\nCongratulation to your team! Thanks for sharing your solution!
", "votes": null, "replies": [] }, { "id": 1220471, "author_name": "startjapan", "author_url": "", "post_date": "02/28/2021 03:18:44", "content": "Thank you so much!
", "votes": null, "replies": [] }, { "id": 1221309, "author_name": "theoviel", "author_url": "", "post_date": "02/28/2021 21:38:53", "content": "I was waiting for this one, congratz and thanks for sharing !
", "votes": null, "replies": [] }, { "id": 1221875, "author_name": "rushatrai", "author_url": "", "post_date": "03/01/2021 11:13:41", "content": "Congrats! Any particular reason you used ReduceLROnPlateau for ResNeXt, cosine annealing for ViT and cosine decay for EfficientNet? Was it just repeated testing with different LR schedulers?
", "votes": null, "replies": [] }, { "id": 1222299, "author_name": "drcodikpollonny", "author_url": "", "post_date": "03/01/2021 17:30:15", "content": "Congratulations! Brilliant approach! I like the diversity you introduced into the ensemble.
", "votes": null, "replies": [] }, { "id": 1222427, "author_name": "dmitrynovikov", "author_url": "", "post_date": "03/01/2021 19:29:52", "content": "@jannish Congratulation to your team!
\nLet me please ask you several questions that seems to be not covered in this thread (sorry if I missed something):
\nThanks.
", "votes": null, "replies": [ { "id": 1236677, "author_name": "jannish", "author_url": "", "post_date": "03/13/2021 11:07:28", "content": "Thanks for your relpy and sorry for the late answer:
\nI tried class weights with the standard Keras method but that didn't work well. I don't know exactly why that didn't bring any improvement.
Yes thats correct. We used this patch-approach only for the efficientnet
Yes, we tried it but only in combination with Saturation and Contrast. This did not bring any improvement, which is why we tried to use as few augmentations as possible in the end.
The problem at the beginning was that I wanted to copy the images to another folder via shell command, this threw some kind of error in the submission.
Entropy
Great idea, Thank you !
", "votes": null, "replies": [] }, { "id": 1224589, "author_name": "anthonydwan", "author_url": "", "post_date": "03/02/2021 21:57:42", "content": "Great write-up! Congratulations on coming 1st!
", "votes": null, "replies": [] }, { "id": 1226637, "author_name": "abhiagwl", "author_url": "", "post_date": "03/04/2021 17:51:40", "content": "It must have been taking very long for making predictions using ensemble ?
", "votes": null, "replies": [] }, { "id": 1234512, "author_name": "hujihong", "author_url": "", "post_date": "03/11/2021 10:14:35", "content": "Congratulations on 1st place !
", "votes": null, "replies": [] }, { "id": 1239808, "author_name": "viswanathravindran", "author_url": "", "post_date": "03/16/2021 03:19:14", "content": "Congratulations on your win and thanks for sharing your detailed effort
", "votes": null, "replies": [] }, { "id": 1246855, "author_name": "vbmokin", "author_url": "", "post_date": "03/21/2021 07:21:55", "content": "@jannish , Great solution! Thanks for sharing. I added it to my collection in \"Data Science with DL & NLP: Advanced Techniques\", section \"Prize Competition Winners: notebooks (kernels) and posts with Magic\".
", "votes": null, "replies": [] }, { "id": 1364372, "author_name": "jaechanlee", "author_url": "", "post_date": "06/24/2021 22:29:54", "content": "Thank you so much
", "votes": null, "replies": [] }, { "id": 1404238, "author_name": "vipin20", "author_url": "", "post_date": "07/29/2021 17:08:00", "content": "Congratulations on the win!
\nthanks for sharing the code for the training pipelines.
Congratulations! Great post.
", "votes": null, "replies": [] }, { "id": 1954104, "author_name": "nghihuynh", "author_url": "", "post_date": "09/25/2022 03:08:43", "content": "Thanks a lot for sharing!
", "votes": null, "replies": [] }, { "id": 2656981, "author_name": "ashirovadil", "author_url": "", "post_date": "02/18/2024 07:24:25", "content": "thanks you very much
", "votes": null, "replies": [] }, { "id": 2969897, "author_name": "vamsianem", "author_url": "", "post_date": "08/25/2024 14:18:51", "content": "congrats man!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1216990": "Our overall strategy was to test as many models as possible and spend less time on fine-tuning. The goal was to have many diverse models for ensembling rather than some highly tuned ones.\n\nIn the end, we had tried a variety of different architectures (e.g., all EfficientNet architectures, Resnet, ResNext, Xception, ViT, DeiT, Inception and MobileNet) while working with different pre-trained weights (trained e.g. on Imagenet, NoisyStudent, Plantvillage, iNaturalist...) some of which were available on Tensorflow Hub. \n\nOur winning submission was an ensemble of four different models.\n\n\nThe final score on the public leaderboard was 91.36% and 91.32% on the private leaderboard. We opted to turn in this combination as it achieved a higher CV score than other combinations (which sometimes scored slightly better on the public leaderboard). We tested some of the models separately on the leaderboard (public/private): B4: 89.4%/89.5% , MobileNet: 89.5%/89.4%, ViT:~89.0%/88.8% \n\nThe others were only evaluated using cross-validation.\n\nOverall, we can conclude that the key to victory was the use of CropNet from Tensorflow Hub, as it brought a lot of diversity to our ensemble. Although it did not perform better on the leaderboard as a standalone model than the other models, the ensembles that used this model brought a significant boost on the leaderboard.\n\nAt this point I would like to thank hengck23, whose [discussion post](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/199276) brought this model to our attention (as well as other pre-trained models for image classification available on Tensorflow Hub).\n\nFurthermore, we would like to thank all the participants who have helped us learn a lot in this contest through their contributions here in the board and their published notebooks. Finally, we would also like to thank the Competition Host, who made this exciting competition possible by releasing the data.\n\nYou can find our inference code in this notebook: Inference Notebook\n\nAnd you can find the according training code in these notebooks:\n\n\nDetailed information about the configuration & fine-tuning of our used models:\n\nWe used a ResNeXt model of the structure “resnext50_32x4d” with the following configurations:\n\n