![\"\"](\"https://i.ibb.co/rbBCL4c/Capture.png\")
Please consider the following links: writeup, code. Below I provide just a short description.
\nOur team would like to thank organizers and kaggle for making this competition possible. And a really big thanks to my amazing teammates @theoviel and @optimo for their incredible efforts in working on the competition and putting everything into writeup for the judge prize. Also @optimo received his first gold medal in this competition, big congratulation to him! Finally, congratulations to all participants of this challenge.
\nIt was quite a long journey for some of us, since the competition started back in 2020. There were a number of challenges we needed to overcome, and there were many learning opportunities. For me, as well as for some participants, it was the opportunity to learn labeling medical data. It may sound funny, but doing this job I realized the amount of work organizers put into this challenge.
\nIn core of our approach is proper treatment of healthy and unhealthy glomeruli.
\nIn short, we quickly realized that d488c759a with FC annotated is destabilizing LB and creating a gap between CV and LB. Use of external data with this annotation improved LB but did not affect or even degraded CV. The explanation is simple: it contained annotation of both healthy and unhealthy glomeruli (at least large portion of them), while the train set contained only annotation of healthy glomeruli. So how can we make a model to predict FC: train it on images with FC… But it was not a right thing: organizers were looking for healthy glomeruli only, and all images except d488c759a did not include FC into annotation. So, use the external data with the shared hand labeling was dangerous, and some people, I may expect, got affected by shake up at the end of the competition because of this.
Our strategy was proper hand annotation of this external dataset with 2 classes: healthy and unhealthy glomeruli. There are many kinds of unhealthy glomeruli, but we decided to keep the things simple, combining them into a single class. We also hand annotated unhealthy glomeruli class in test images, which combined with PL for healthy glomeruli, were used in our training pipeline. We also used zondo dataset, which naturally includes two classes. However, there was quite a number of missed glomeruli, especially at the image edges, and we hand annotated them. Finally, we used 2 images from HuBMAP portal, not included into train/test datasets with PL (both healthy and unhealthy) generated by our model. Using of multiple data sources helped us to ensure that the model is able to generalize well on new cases. We also used a combined human+AI guided method to add missing annotation to the train data (there were just a few instances tough).
\nSecret sauce: ignore d488c759a from LB evaluation. LB with fixed predictions on this image correlated quite well with CV
\nFrom the model perspective of view, there is nothing really special, EfficientNet, Unet… Though, we tried more advanced things, like BoT middle layer, Swin transformer backbone, FPN skip connections, but it was not giving any noticeable benefit. This competition really was not about it. The thing that helped was using 2 classes during training. The level of injury of glomeruli is on a continuous scale. So instead of confusing the model by providing nearly identical glomeruli included/excluded from masks, we made the model to learn that unhealthy glomeruli are also glomeruli, and at the end model decides whether a particular instance is healthy enough. To be honest, this competition was not really about segmentation but rather about classification. The uncertainty of including a particular glomerulus in the mask predicted by the model is the major factor degrading the model performance.
\nOur best submission (ensemble of two models) got 0.9439 CV, 0.941 LB (with including hand annotation of unhealthy glomeruli for d488c759a image), and 0.950 private LB. There were some technical difficulties with making submissions on the last day, so we could not make a submission with all our final models(
\nPlease consider the following links: writeup, code. Below I provide just a short description.
\nOur team would like to thank organizers and kaggle for making this competition possible. And a really big thanks to my amazing teammates @theoviel and @optimo for their incredible efforts in working on the competition and putting everything into writeup for the judge prize. Also @optimo received his first gold medal in this competition, big congratulation to him! Finally, congratulations to all participants of this challenge.
\nIt was quite a long journey for some of us, since the competition started back in 2020. There were a number of challenges we needed to overcome, and there were many learning opportunities. For me, as well as for some participants, it was the opportunity to learn labeling medical data. It may sound funny, but doing this job I realized the amount of work organizers put into this challenge.
\nIn core of our approach is proper treatment of healthy and unhealthy glomeruli.
\nIn short, we quickly realized that d488c759a with FC annotated is destabilizing LB and creating a gap between CV and LB. Use of external data with this annotation improved LB but did not affect or even degraded CV. The explanation is simple: it contained annotation of both healthy and unhealthy glomeruli (at least large portion of them), while the train set contained only annotation of healthy glomeruli. So how can we make a model to predict FC: train it on images with FC… But it was not a right thing: organizers were looking for healthy glomeruli only, and all images except d488c759a did not include FC into annotation. So, use the external data with the shared hand labeling was dangerous, and some people, I may expect, got affected by shake up at the end of the competition because of this.
Our strategy was proper hand annotation of this external dataset with 2 classes: healthy and unhealthy glomeruli. There are many kinds of unhealthy glomeruli, but we decided to keep the things simple, combining them into a single class. We also hand annotated unhealthy glomeruli class in test images, which combined with PL for healthy glomeruli, were used in our training pipeline. We also used zondo dataset, which naturally includes two classes. However, there was quite a number of missed glomeruli, especially at the image edges, and we hand annotated them. Finally, we used 2 images from HuBMAP portal, not included into train/test datasets with PL (both healthy and unhealthy) generated by our model. Using of multiple data sources helped us to ensure that the model is able to generalize well on new cases. We also used a combined human+AI guided method to add missing annotation to the train data (there were just a few instances tough).
\nSecret sauce: ignore d488c759a from LB evaluation. LB with fixed predictions on this image correlated quite well with CV
\nFrom the model perspective of view, there is nothing really special, EfficientNet, Unet… Though, we tried more advanced things, like BoT middle layer, Swin transformer backbone, FPN skip connections, but it was not giving any noticeable benefit. This competition really was not about it. The thing that helped was using 2 classes during training. The level of injury of glomeruli is on a continuous scale. So instead of confusing the model by providing nearly identical glomeruli included/excluded from masks, we made the model to learn that unhealthy glomeruli are also glomeruli, and at the end model decides whether a particular instance is healthy enough. To be honest, this competition was not really about segmentation but rather about classification. The uncertainty of including a particular glomerulus in the mask predicted by the model is the major factor degrading the model performance.
\nOur best submission (ensemble of two models) got 0.9439 CV, 0.941 LB (with including hand annotation of unhealthy glomeruli for d488c759a image), and 0.950 private LB. There were some technical difficulties with making submissions on the last day, so we could not make a submission with all our final models(
\n@iafoss Congratulations on getting 4th place. This is the 3rd competition I have participated on Kaggle and in each of them, your code/discussions have really been of help to me.
", "rawMarkdown": "iafoss Congratulations on getting 4th place. This is the 3rd competition I have participated on Kaggle and in each of them, your code/discussions have really been of help to me.", "votes": null }, { "id": "1301255", "postDate": "05/11/2021 02:02:20", "content": "@iafoss Congratulations on 4 th Place and thanks for sharing the solution
", "rawMarkdown": "iafoss Congratulations on 4 th Place and thanks for sharing the solution", "votes": null }, { "id": "1301274", "postDate": "05/11/2021 02:14:53", "content": "@nauyan thank you, I'm glad that you found my kernels helpful
", "rawMarkdown": "nauyan thank you, I'm glad that you found my kernels helpful", "votes": null }, { "id": "1301275", "postDate": "05/11/2021 02:15:39", "content": "@usharengaraju Thanks so much, and you are very welcome.
", "rawMarkdown": "usharengaraju Thanks so much, and you are very welcome.", "votes": null }, { "id": "1301328", "postDate": "05/11/2021 03:04:46", "content": "Congratulations a lot ! your code was truly helped. and I learned a lot from on it.
\nAnd thanks for sharing the solution :)
You are very welcome
", "rawMarkdown": "You are very welcome", "votes": null }, { "id": "1301367", "postDate": "05/11/2021 03:43:33", "content": "Congrats! I've learned a lot from your code both in the early and final step of this competition!
", "rawMarkdown": "Congrats! I've learned a lot from your code both in the early and final step of this competition!", "votes": null }, { "id": "1301374", "postDate": "05/11/2021 03:48:32", "content": "Thanks, you are very welcome
", "rawMarkdown": "Thanks, you are very welcome", "votes": null }, { "id": "1301402", "postDate": "05/11/2021 04:09:42", "content": "Very great solution indeed !
", "rawMarkdown": "Very great solution indeed !", "votes": null }, { "id": "1301408", "postDate": "05/11/2021 04:15:35", "content": "@jaideepvalani Thank you so much
", "rawMarkdown": "jaideepvalani Thank you so much", "votes": null }, { "id": "1301418", "postDate": "05/11/2021 04:25:09", "content": "Congrats @iafoss @theoviel and @optimo. Nice job and thanks for the writeup.
", "rawMarkdown": "Congrats @iafoss @theoviel and @optimo. Nice job and thanks for the writeup.", "votes": null }, { "id": "1301440", "postDate": "05/11/2021 04:45:09", "content": "Thank you so much!
", "rawMarkdown": "Thank you so much!", "votes": null }, { "id": "1301461", "postDate": "05/11/2021 05:07:19", "content": "It was my first time competing and I thought I knew what was important. I thought I knew what was important, but I didn't realize how important preprocessing was. I tried to cross-validate (eefficientnet) and your wonderful stuff, but I couldn't get it done in time. I realized it was 4 hours before the end of the day, so I was done. But I didn't realize that if I wanted to go higher, I should spend more time on the pre-processing at the beginning. I'll keep that in mind next time. I'd like to thank you for all the help you've given me, from the code to the content. I have been referring to you for everything, from the code to the content. You are the only one who writes so clearly and concisely. I've only been doing this for about 7 days, but I've packed 3 years worth of experience. Let's meet at the leaderboard next time!
\nSorry for the long post.
\nThank you very much!!!!!!!!!!!!!!!!!!!!
Translated with www.DeepL.com/Translator (free version)
", "rawMarkdown": "It was my first time competing and I thought I knew what was important. I thought I knew what was important, but I didn't realize how important preprocessing was. I tried to cross-validate (eefficientnet) and your wonderful stuff, but I couldn't get it done in time. I realized it was 4 hours before the end of the day, so I was done. But I didn't realize that if I wanted to go higher, I should spend more time on the pre-processing at the beginning. I'll keep that in mind next time. I'd like to thank you for all the help you've given me, from the code to the content. I have been referring to you for everything, from the code to the content. You are the only one who writes so clearly and concisely. I've only been doing this for about 7 days, but I've packed 3 years worth of experience. Let's meet at the leaderboard next time!\nSorry for the long post.\nThank you very much!!!!!!!!!!!!!!!!!!!!\n\n\n\nTranslated with www.DeepL.com/Translator (free version)", "votes": null }, { "id": "1301462", "postDate": "05/11/2021 05:07:39", "content": "The secret sauce is really on point. I also realized that but 3 days before the competition ended so there was no time to experiment based on that.
\nAggressive augmentations are extremely important as you mentioned in the write-up. I actually had a vague idea about it during the PANDA conference, where my models did not generalize well to external data. I think that was because my PANDA models were trained on simple augmentations such as rotation and cutout, so now I think augmentations probably should be as aggressive as possible without hurting cv/lb
", "rawMarkdown": "The secret sauce is really on point. I also realized that but 3 days before the competition ended so there was no time to experiment based on that. \n\nAggressive augmentations are extremely important as you mentioned in the write-up. I actually had a vague idea about it during the PANDA conference, where my models did not generalize well to external data. I think that was because my PANDA models were trained on simple augmentations such as rotation and cutout, so now I think augmentations probably should be as aggressive as possible without hurting cv/lb", "votes": null }, { "id": "1301464", "postDate": "05/11/2021 05:08:47", "content": "@iafoss wcm .learning is always mine.
\n Did u you use post processing also . In our case we came up with two
\n a) Area based -Used Pneumothorax competition .It gave slight improvement.
\n b) To be tested :Filtering Extra borderers based on HSV. This was some thing that i learnt off lately by that time it was late to use and test it but looked promising. I will see if it works in late submissions
Thank you so much, and congratulations with completing your first competition.
", "rawMarkdown": "Thank you so much, and congratulations with completing your first competition.", "votes": null }, { "id": "1301483", "postDate": "05/11/2021 05:30:04", "content": "Yes, the augmentation may be sometimes quite tricky, and there could be a trade of between CV and the ability of the model to work on a completely new data. Doing some analysis based on activation and/or attention maps is often quite helpful to get an idea if model really good, or just pretending to be good by capturing some irrelevant things of the particular set… And based on that one can adjust the augmentation.
", "rawMarkdown": "Yes, the augmentation may be sometimes quite tricky, and there could be a trade of between CV and the ability of the model to work on a completely new data. Doing some analysis based on activation and/or attention maps is often quite helpful to get an idea if model really good, or just pretending to be good by capturing some irrelevant things of the particular set... And based on that one can adjust the augmentation.", "votes": null }, { "id": "1301502", "postDate": "05/11/2021 05:40:55", "content": "@theoviel super congratulations for you :) you deserve a gold !
", "rawMarkdown": "theoviel super congratulations for you :) you deserve a gold !", "votes": null }, { "id": "1301538", "postDate": "05/11/2021 06:11:14", "content": "Thanks @jinssaa, it wouldn't have been possible without my teammates though :)
", "rawMarkdown": "Thanks @jinssaa, it wouldn't have been possible without my teammates though :)", "votes": null }, { "id": "1301542", "postDate": "05/11/2021 06:13:50", "content": "@jaideepvalani I tried a lot of post-processing but couldn't find a reliable strategy.
\nCRFs, remove small components, remove components with a small max proba, use the convex hull of components… Nothing really worked.
removing small components seems worked for me. since my model is overfit and aggresive to predict the target.
", "rawMarkdown": "removing small components seems worked for me. since my model is overfit and aggresive to predict the target.", "votes": null }, { "id": "1301618", "postDate": "05/11/2021 07:01:06", "content": "Congratulations! Without your code I would never get a silver. Everytime I had problems in my competition, I would always go throught your code to get some ideas.
", "rawMarkdown": "Congratulations! Without your code I would never get a silver. Everytime I had problems in my competition, I would always go throught your code to get some ideas.", "votes": null }, { "id": "1301651", "postDate": "05/11/2021 07:17:54", "content": "Yes its difficult to set any generalized threshold to select or unselect contour . I checked some of my predictions , found that some masks always had an extra border or better to say dilated predictions i suppose there may at least 1 to 2 in every 100 prediction that one could find such adding to FPS.
", "rawMarkdown": "Yes its difficult to set any generalized threshold to select or unselect contour . I checked some of my predictions , found that some masks always had an extra border or better to say dilated predictions i suppose there may at least 1 to 2 in every 100 prediction that one could find such adding to FPS.", "votes": null }, { "id": "1301864", "postDate": "05/11/2021 09:26:01", "content": "Well done!
", "rawMarkdown": "Well done!", "votes": null }, { "id": "1301955", "postDate": "05/11/2021 10:30:57", "content": "Congrats, Elegant solution
", "rawMarkdown": "Congrats, Elegant solution", "votes": null }, { "id": "1302137", "postDate": "05/11/2021 12:02:49", "content": "great work @iafoss
\nI also experimented with the zenodo dataset (provided no boost for me),the mendeley dataset (used pseudo labels and used those 227x227 glomes as mixup; no boost for me) and pseudo labels on the two tiffs from the hubmap portal (again no boost). I didn't try to hand label these datasets tho.
\nOn the other hand, wrt to ignoring d488, I'm struggling to understand why one model I trained with the d488 hand labels score the same (0.947) as 4 other models with very similar config but no d488 hand labels.
\n
Thanks
", "rawMarkdown": "Thanks", "votes": null }, { "id": "1303211", "postDate": "05/12/2021 00:55:41", "content": "Thanks so much
", "rawMarkdown": "Thanks so much", "votes": null }, { "id": "1303231", "postDate": "05/12/2021 01:18:54", "content": "Thanks @rosuluc, quite interesting observation. My best explanation could be that tissue is extremally different from once patient to another (I spent really a lot of time annotating and looking into the images). It may happen that for a specific subset of images the impact of adding FC to d488c759a is not significant, like, for example, there may be no images with FC in the private test set (adding FC to d488c759a does not generate FP at private set). It is difficult to say without looking into images and actual predictions. Meanwhile, it could be another private subset of images where the effect will be game changing.
\nIn our work we tried to minimize the risks and create a model that generalizes well on diverse images. It is obvious that there may be some randomness in the LB, but we tried to follow a methodology driven approach rather than at the end, if get lucky enough, write about a success story.
", "rawMarkdown": "Thanks @rosuluc, quite interesting observation. My best explanation could be that tissue is extremally different from once patient to another (I spent really a lot of time annotating and looking into the images). It may happen that for a specific subset of images the impact of adding FC to d488c759a is not significant, like, for example, there may be no images with FC in the private test set (adding FC to d488c759a does not generate FP at private set). It is difficult to say without looking into images and actual predictions. Meanwhile, it could be another private subset of images where the effect will be game changing.\n\nIn our work we tried to minimize the risks and create a model that generalizes well on diverse images. It is obvious that there may be some randomness in the LB, but we tried to follow a methodology driven approach rather than at the end, if get lucky enough, write about a success story.", "votes": null }, { "id": "1303533", "postDate": "05/12/2021 05:49:38", "content": "Thanks @duykhanh99 !
", "rawMarkdown": "Thanks @duykhanh99 !", "votes": null }, { "id": "1304225", "postDate": "05/12/2021 13:44:12", "content": "Thank you so much, I'm very happy.
", "rawMarkdown": "Thank you so much, I'm very happy.", "votes": null }, { "id": "1304636", "postDate": "05/12/2021 18:46:22", "content": "\n\nOn the other hand, wrt to ignoring d488, I'm struggling to understand why one model I trained with the d488 hand labels score the same (0.947) as 4 other models with very similar config but no d488 hand labels.
\n
The same thing happened to me described here. If i added my very accurate public test d488 labels to model training, it did not change my CV score nor did it change my private LB score. (It did increase my public LB score and it would have increased private LB, if private LB contained the same outliers).
", "rawMarkdown": "> On the other hand, wrt to ignoring d488, I'm struggling to understand why one model I trained with the d488 hand labels score the same (0.947) as 4 other models with very similar config but no d488 hand labels.\n\nThe same thing happened to me described [here][1]. If i added my very accurate public test d488 labels to model training, it did not change my CV score nor did it change my private LB score. (It did increase my public LB score and it would have increased private LB, if private LB contained the same outliers).\n\n[1]: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/237999", "votes": null }, { "id": "1304642", "postDate": "05/12/2021 18:49:57", "content": "Congratulations @iafoss @theoviel @optimo This is a wonderful solution, i like your idea of 2 class segmentation and beneficial use of external data.
\nYour secret sauce is very smart. The presence of d488c759a made public LB score meaningless, but your trick allowed you to collect meaningful feedback from public LB score.
\n\n", "rawMarkdown": "Congratulations @iafoss @theoviel @optimo This is a wonderful solution, i like your idea of 2 class segmentation and beneficial use of external data. \n\nYour secret sauce is very smart. The presence of d488c759a made public LB score meaningless, but your trick allowed you to collect meaningful feedback from public LB score.\n\n> Secret sauce: ignore d488c759a from LB evaluation. LB with fixed predictions on this image correlated quite well with CV", "votes": null }, { "id": "1304687", "postDate": "05/12/2021 19:18:05", "content": "Secret sauce: ignore d488c759a from LB evaluation. LB with fixed predictions on this image correlated quite well with CV
\n
Thank you @cdeotte , this competition was quite confusing because of incorrect annotation of 'd488c759a', and host never clarified it. There is a number of other images where FC instances are not annotated, so we assumed that 'd488c759a' is just and exception annotated by some external person, who didn't know what to include. We also had a plan B, though, where we used PL and hand annotation of FC for 'd488c759a' (but not as elaborated as yours), in case if the private test set had this kind of mistakes as train.
", "rawMarkdown": "Thank you @cdeotte , this competition was quite confusing because of incorrect annotation of 'd488c759a', and host never clarified it. There is a number of other images where FC instances are not annotated, so we assumed that 'd488c759a' is just and exception annotated by some external person, who didn't know what to include. We also had a plan B, though, where we used PL and hand annotation of FC for 'd488c759a' (but not as elaborated as yours), in case if the private test set had this kind of mistakes as train.", "votes": null }, { "id": "1304719", "postDate": "05/12/2021 19:50:55", "content": "Thanks Chris !
\nCongratz to you as well, grabbing a solo gold and a solo silver the same day is far from easy :)
@iafoss Congratulations on getting 4th place. This is the 3rd competition I have participated on Kaggle and in each of them, your code/discussions have really been of help to me.
", "votes": null, "replies": [ { "id": 1301274, "author_name": "iafoss", "author_url": "", "post_date": "05/11/2021 02:14:53", "content": "@nauyan thank you, I'm glad that you found my kernels helpful
", "votes": null, "replies": [] } ] }, { "id": 1301255, "author_name": "usharengaraju", "author_url": "", "post_date": "05/11/2021 02:02:20", "content": "@iafoss Congratulations on 4 th Place and thanks for sharing the solution
", "votes": null, "replies": [ { "id": 1301275, "author_name": "iafoss", "author_url": "", "post_date": "05/11/2021 02:15:39", "content": "@usharengaraju Thanks so much, and you are very welcome.
", "votes": null, "replies": [] } ] }, { "id": 1301328, "author_name": "jinssaa", "author_url": "", "post_date": "05/11/2021 03:04:46", "content": "Congratulations a lot ! your code was truly helped. and I learned a lot from on it.
\nAnd thanks for sharing the solution :)
You are very welcome
", "votes": null, "replies": [] } ] }, { "id": 1301367, "author_name": "southsakura", "author_url": "", "post_date": "05/11/2021 03:43:33", "content": "Congrats! I've learned a lot from your code both in the early and final step of this competition!
", "votes": null, "replies": [ { "id": 1301374, "author_name": "iafoss", "author_url": "", "post_date": "05/11/2021 03:48:32", "content": "Thanks, you are very welcome
", "votes": null, "replies": [] }, { "id": 1301464, "author_name": "jaideepvalani", "author_url": "", "post_date": "05/11/2021 05:08:47", "content": "@iafoss wcm .learning is always mine.
\n Did u you use post processing also . In our case we came up with two
\n a) Area based -Used Pneumothorax competition .It gave slight improvement.
\n b) To be tested :Filtering Extra borderers based on HSV. This was some thing that i learnt off lately by that time it was late to use and test it but looked promising. I will see if it works in late submissions
@jaideepvalani I tried a lot of post-processing but couldn't find a reliable strategy.
\nCRFs, remove small components, remove components with a small max proba, use the convex hull of components… Nothing really worked.
removing small components seems worked for me. since my model is overfit and aggresive to predict the target.
", "votes": null, "replies": [] }, { "id": 1301651, "author_name": "jaideepvalani", "author_url": "", "post_date": "05/11/2021 07:17:54", "content": "Yes its difficult to set any generalized threshold to select or unselect contour . I checked some of my predictions , found that some masks always had an extra border or better to say dilated predictions i suppose there may at least 1 to 2 in every 100 prediction that one could find such adding to FPS.
", "votes": null, "replies": [] } ] }, { "id": 1301402, "author_name": "jaideepvalani", "author_url": "", "post_date": "05/11/2021 04:09:42", "content": "Very great solution indeed !
", "votes": null, "replies": [ { "id": 1301408, "author_name": "iafoss", "author_url": "", "post_date": "05/11/2021 04:15:35", "content": "@jaideepvalani Thank you so much
", "votes": null, "replies": [] } ] }, { "id": 1301418, "author_name": "duykhanh99", "author_url": "", "post_date": "05/11/2021 04:25:09", "content": "Congrats @iafoss @theoviel and @optimo. Nice job and thanks for the writeup.
", "votes": null, "replies": [ { "id": 1301440, "author_name": "iafoss", "author_url": "", "post_date": "05/11/2021 04:45:09", "content": "Thank you so much!
", "votes": null, "replies": [] }, { "id": 1303533, "author_name": "theoviel", "author_url": "", "post_date": "05/12/2021 05:49:38", "content": "Thanks @duykhanh99 !
", "votes": null, "replies": [] } ] }, { "id": 1301461, "author_name": "taiyofuckyou", "author_url": "", "post_date": "05/11/2021 05:07:19", "content": "It was my first time competing and I thought I knew what was important. I thought I knew what was important, but I didn't realize how important preprocessing was. I tried to cross-validate (eefficientnet) and your wonderful stuff, but I couldn't get it done in time. I realized it was 4 hours before the end of the day, so I was done. But I didn't realize that if I wanted to go higher, I should spend more time on the pre-processing at the beginning. I'll keep that in mind next time. I'd like to thank you for all the help you've given me, from the code to the content. I have been referring to you for everything, from the code to the content. You are the only one who writes so clearly and concisely. I've only been doing this for about 7 days, but I've packed 3 years worth of experience. Let's meet at the leaderboard next time!
\nSorry for the long post.
\nThank you very much!!!!!!!!!!!!!!!!!!!!
Translated with www.DeepL.com/Translator (free version)
", "votes": null, "replies": [ { "id": 1301476, "author_name": "iafoss", "author_url": "", "post_date": "05/11/2021 05:21:19", "content": "Thank you so much, and congratulations with completing your first competition.
", "votes": null, "replies": [] } ] }, { "id": 1301462, "author_name": "shujun717", "author_url": "", "post_date": "05/11/2021 05:07:39", "content": "The secret sauce is really on point. I also realized that but 3 days before the competition ended so there was no time to experiment based on that.
\nAggressive augmentations are extremely important as you mentioned in the write-up. I actually had a vague idea about it during the PANDA conference, where my models did not generalize well to external data. I think that was because my PANDA models were trained on simple augmentations such as rotation and cutout, so now I think augmentations probably should be as aggressive as possible without hurting cv/lb
", "votes": null, "replies": [ { "id": 1301483, "author_name": "iafoss", "author_url": "", "post_date": "05/11/2021 05:30:04", "content": "Yes, the augmentation may be sometimes quite tricky, and there could be a trade of between CV and the ability of the model to work on a completely new data. Doing some analysis based on activation and/or attention maps is often quite helpful to get an idea if model really good, or just pretending to be good by capturing some irrelevant things of the particular set… And based on that one can adjust the augmentation.
", "votes": null, "replies": [] } ] }, { "id": 1301502, "author_name": "jinssaa", "author_url": "", "post_date": "05/11/2021 05:40:55", "content": "@theoviel super congratulations for you :) you deserve a gold !
", "votes": null, "replies": [ { "id": 1301538, "author_name": "theoviel", "author_url": "", "post_date": "05/11/2021 06:11:14", "content": "Thanks @jinssaa, it wouldn't have been possible without my teammates though :)
", "votes": null, "replies": [] } ] }, { "id": 1301618, "author_name": "luciusk", "author_url": "", "post_date": "05/11/2021 07:01:06", "content": "Congratulations! Without your code I would never get a silver. Everytime I had problems in my competition, I would always go throught your code to get some ideas.
", "votes": null, "replies": [ { "id": 1304225, "author_name": "iafoss", "author_url": "", "post_date": "05/12/2021 13:44:12", "content": "Thank you so much, I'm very happy.
", "votes": null, "replies": [] } ] }, { "id": 1301864, "author_name": "yassinealouini", "author_url": "", "post_date": "05/11/2021 09:26:01", "content": "Well done!
", "votes": null, "replies": [ { "id": 1303207, "author_name": "iafoss", "author_url": "", "post_date": "05/12/2021 00:51:24", "content": "Thanks
", "votes": null, "replies": [] } ] }, { "id": 1301955, "author_name": "salimkhazem", "author_url": "", "post_date": "05/11/2021 10:30:57", "content": "Congrats, Elegant solution
", "votes": null, "replies": [ { "id": 1303211, "author_name": "iafoss", "author_url": "", "post_date": "05/12/2021 00:55:41", "content": "Thanks so much
", "votes": null, "replies": [] } ] }, { "id": 1302137, "author_name": "rosuluc", "author_url": "", "post_date": "05/11/2021 12:02:49", "content": "great work @iafoss
\nI also experimented with the zenodo dataset (provided no boost for me),the mendeley dataset (used pseudo labels and used those 227x227 glomes as mixup; no boost for me) and pseudo labels on the two tiffs from the hubmap portal (again no boost). I didn't try to hand label these datasets tho.
\nOn the other hand, wrt to ignoring d488, I'm struggling to understand why one model I trained with the d488 hand labels score the same (0.947) as 4 other models with very similar config but no d488 hand labels.
\n
Thanks @rosuluc, quite interesting observation. My best explanation could be that tissue is extremally different from once patient to another (I spent really a lot of time annotating and looking into the images). It may happen that for a specific subset of images the impact of adding FC to d488c759a is not significant, like, for example, there may be no images with FC in the private test set (adding FC to d488c759a does not generate FP at private set). It is difficult to say without looking into images and actual predictions. Meanwhile, it could be another private subset of images where the effect will be game changing.
\nIn our work we tried to minimize the risks and create a model that generalizes well on diverse images. It is obvious that there may be some randomness in the LB, but we tried to follow a methodology driven approach rather than at the end, if get lucky enough, write about a success story.
", "votes": null, "replies": [] }, { "id": 1304636, "author_name": "cdeotte", "author_url": "", "post_date": "05/12/2021 18:46:22", "content": "\n\nOn the other hand, wrt to ignoring d488, I'm struggling to understand why one model I trained with the d488 hand labels score the same (0.947) as 4 other models with very similar config but no d488 hand labels.
\n
The same thing happened to me described here. If i added my very accurate public test d488 labels to model training, it did not change my CV score nor did it change my private LB score. (It did increase my public LB score and it would have increased private LB, if private LB contained the same outliers).
", "votes": null, "replies": [] } ] }, { "id": 1304642, "author_name": "cdeotte", "author_url": "", "post_date": "05/12/2021 18:49:57", "content": "Congratulations @iafoss @theoviel @optimo This is a wonderful solution, i like your idea of 2 class segmentation and beneficial use of external data.
\nYour secret sauce is very smart. The presence of d488c759a made public LB score meaningless, but your trick allowed you to collect meaningful feedback from public LB score.
\n\n", "votes": null, "replies": [ { "id": 1304687, "author_name": "iafoss", "author_url": "", "post_date": "05/12/2021 19:18:05", "content": "Secret sauce: ignore d488c759a from LB evaluation. LB with fixed predictions on this image correlated quite well with CV
\n
Thank you @cdeotte , this competition was quite confusing because of incorrect annotation of 'd488c759a', and host never clarified it. There is a number of other images where FC instances are not annotated, so we assumed that 'd488c759a' is just and exception annotated by some external person, who didn't know what to include. We also had a plan B, though, where we used PL and hand annotation of FC for 'd488c759a' (but not as elaborated as yours), in case if the private test set had this kind of mistakes as train.
", "votes": null, "replies": [] }, { "id": 1304719, "author_name": "theoviel", "author_url": "", "post_date": "05/12/2021 19:50:55", "content": "Thanks Chris !
\nCongratz to you as well, grabbing a solo gold and a solo silver the same day is far from easy :)