{ "id": 176382, "title": "Report of 408 submissions!", "url": "/competitions/siim-isic-melanoma-classification/discussion/176382", "author_name": "Amin", "post_date": "2020-08-21T14:26:18.549000", "votes": 23, "comment_count": 21, "views": 0, "content": "

First of all, I would like to thank the organizers, Kaggle and my teammates @underwearfitting, @hawkey and @nicohrubec for this amazing experience.

\n

I am happy that I picked SIIM-Melanoma to be my first competition because it had everything: A great leader @cdeotte, active teammates, unstable public LB, a terrible shake-up and finally, the drama related to the participation of the brilliant students of the medal farming academy.

\n

We won a bronze medal thanks to the clean up done by kaggle and the investigation started by @group16. Big thanks!

\n

As most of the participants, we didn’t select our best submission that would have landed us a silver medal. Actually 60 submissions scored higher than our best selected sub. This is part of the game, taking good decisions is an essential skill in ML/DL. Here is a first glimpse of our submissions.
\n\"\"

\n

First time crossing the medal zone:

\n

The first time I crossed the medal zone was the 15th of July with private LB= 0.9375 (Slightly higher than our best selected submission).
\n\"\"
\nWhy did this sub score higher than our selected submission? Because it had an ingredient that was missing in the final sub: Higher image_size= 768. I was experimenting with higher image sizes but keeping the CNN's input_size smaller for faster training, it's a clear information loss/training time trade-off.

\n
Image size= 512. Input size= 226\n\nImage size= 768. Input size= 384\n\nImage size= 1024 Input size= 512\n
\n

It scored higher than the models with image_size 512 or 384, which is good news, but I decided not to go further with this experiment because it was risky since parts of the images are lost with the center crops.

\n

I was happy with this decision until I saw @cdeotte's solution where he used smaller input_size with random crops instead of center crops at each epoch to make sure the model loops over all the pixels instead of center cropping the same pixels over the epochs. Another trick learned to be used in future competitions ;)

\n

Teaming-up:

\n

The day we teamed-up has seen a boost in our private LB score. It was just a rank ensemble of our best submissions. This shows the clear benefit of teaming up late because you will definitely add diversity to the final solution.

\n

The team era:

\n

\"\"
\nYou can see in the plot that the best era was after teaming up because we experimented many ideas together that helped us reach the silver zone at our peak. I am not gonna lie, at that point I thought we were overfitting to public LB because most of the silver zone submissions are nested blends of our +0.95 submissions. However, the correlation of our public LB scores in the range (0.955, 0.9642) with private LB was 0.89! The question here is: When did we start overfitting?
\n\"\"
\nOverfitting (the decay) starts after including the last public notebook in our blend that was terribly overfitting (Public= 0.96, Private= 0.91). The blends scoring +0.9643 included overfitted public notebooks

\n

It is worth mentioning that the highest public/private LB correlation we had was with subs in the medal zone, so our subs definitely had something special to score high private scores without any CV strategy. What ideas or ingredients did the silver submissions have? Simple answer: Post-processing.

\n

Post-processing:

\n

We had 2 post-processing techniques, they both boost the public and private LB score.
\nThe first was a finding by @underwearfitting where he set all the palms/soles and oral/genital predictions to 0, code below:

\n
ps= test.loc[test['anatom_site_general_challenge']=='palms/soles'].image_name\nog= test.loc[test['anatom_site_general_challenge']=='oral/genital'].image_name\nsubmission= submission.set_index('image_name')\nsubmission.loc[ps]=0\nsubmission.loc[og]=0\n
\n

This pp gives a +0.001 boost in private LB (Try it at home! +0.001 boost guaranteed 😉).

\n

The second was image embeddings extraction using RAPIDS cuML TSNE shared by @cdeotte, thanks Chris for everything you shared in this competition.
\n@hawkey tried to detect the malignant images in the test set by extracting the patients around the red area (malignants) as they are more susceptible. This pp was unstable but led to some high private LB scores.
\n(If you have any questions about this method, please tag @hawkey in the comment)
\n\"\":

\n

The final solution :

\n

\"\"
\nThe blue line marks the switch from trusting(climbing) LB to trusting CV.

\n

The final step in this competition was to prepare a solid solution. Even though we had some high scoring subs, interesting ideas (training with high resolution images and smaller CNN input size + post-processing), we decided to do it the safest way and not use any of them and stack models trained with different architecture (B3-B7 + seresnext50) and different image sizes (384, 512).
\nWith stacking, we managed to reduce the gap between CV/LB to 0 in our best selected submission, where CV=LB=0.9483!

\n

Conclusion:

\n

We ended up with a very correct solution that is missing the special ingredient to make it a top solution. However, I am happy to see that the ideas we have been testing actually work, even though not using them in our final solution penalized our private LB score. Those are techniques learned to be used in future competitions.

\n

Best private and public subs:

\n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
SubmissionCVPublic LBPrivate LB
Best public N/A0.96570.9374
Best privateN/A0.96130.9410
\n

3 Selected submissions

\n\n \n \n \n \n \n\n \n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n
SubmissionCVPublic LBPrivate LB
Most stable0.94830.94830.9374
Best CV0.95410.95100.9359
2 Best CV0.95350.95260.9337
", "messages": [ { "id": 980349, "postDate": "2020-08-21T14:26:18.550Z", "content": "

First of all, I would like to thank the organizers, Kaggle and my teammates @underwearfitting, @hawkey and @nicohrubec for this amazing experience.

\n

I am happy that I picked SIIM-Melanoma to be my first competition because it had everything: A great leader @cdeotte, active teammates, unstable public LB, a terrible shake-up and finally, the drama related to the participation of the brilliant students of the medal farming academy.

\n

We won a bronze medal thanks to the clean up done by kaggle and the investigation started by @group16. Big thanks!

\n

As most of the participants, we didn’t select our best submission that would have landed us a silver medal. Actually 60 submissions scored higher than our best selected sub. This is part of the game, taking good decisions is an essential skill in ML/DL. Here is a first glimpse of our submissions.
\n\"\"

\n

First time crossing the medal zone:

\n

The first time I crossed the medal zone was the 15th of July with private LB= 0.9375 (Slightly higher than our best selected submission).
\n\"\"
\nWhy did this sub score higher than our selected submission? Because it had an ingredient that was missing in the final sub: Higher image_size= 768. I was experimenting with higher image sizes but keeping the CNN's input_size smaller for faster training, it's a clear information loss/training time trade-off.

\n
Image size= 512. Input size= 226\n\nImage size= 768. Input size= 384\n\nImage size= 1024 Input size= 512\n
\n

It scored higher than the models with image_size 512 or 384, which is good news, but I decided not to go further with this experiment because it was risky since parts of the images are lost with the center crops.

\n

I was happy with this decision until I saw @cdeotte's solution where he used smaller input_size with random crops instead of center crops at each epoch to make sure the model loops over all the pixels instead of center cropping the same pixels over the epochs. Another trick learned to be used in future competitions ;)

\n

Teaming-up:

\n

The day we teamed-up has seen a boost in our private LB score. It was just a rank ensemble of our best submissions. This shows the clear benefit of teaming up late because you will definitely add diversity to the final solution.

\n

The team era:

\n

\"\"
\nYou can see in the plot that the best era was after teaming up because we experimented many ideas together that helped us reach the silver zone at our peak. I am not gonna lie, at that point I thought we were overfitting to public LB because most of the silver zone submissions are nested blends of our +0.95 submissions. However, the correlation of our public LB scores in the range (0.955, 0.9642) with private LB was 0.89! The question here is: When did we start overfitting?
\n\"\"
\nOverfitting (the decay) starts after including the last public notebook in our blend that was terribly overfitting (Public= 0.96, Private= 0.91). The blends scoring +0.9643 included overfitted public notebooks

\n

It is worth mentioning that the highest public/private LB correlation we had was with subs in the medal zone, so our subs definitely had something special to score high private scores without any CV strategy. What ideas or ingredients did the silver submissions have? Simple answer: Post-processing.

\n

Post-processing:

\n

We had 2 post-processing techniques, they both boost the public and private LB score.
\nThe first was a finding by @underwearfitting where he set all the palms/soles and oral/genital predictions to 0, code below:

\n
ps= test.loc[test['anatom_site_general_challenge']=='palms/soles'].image_name\nog= test.loc[test['anatom_site_general_challenge']=='oral/genital'].image_name\nsubmission= submission.set_index('image_name')\nsubmission.loc[ps]=0\nsubmission.loc[og]=0\n
\n

This pp gives a +0.001 boost in private LB (Try it at home! +0.001 boost guaranteed 😉).

\n

The second was image embeddings extraction using RAPIDS cuML TSNE shared by @cdeotte, thanks Chris for everything you shared in this competition.
\n@hawkey tried to detect the malignant images in the test set by extracting the patients around the red area (malignants) as they are more susceptible. This pp was unstable but led to some high private LB scores.
\n(If you have any questions about this method, please tag @hawkey in the comment)
\n\"\":

\n

The final solution :

\n

\"\"
\nThe blue line marks the switch from trusting(climbing) LB to trusting CV.

\n

The final step in this competition was to prepare a solid solution. Even though we had some high scoring subs, interesting ideas (training with high resolution images and smaller CNN input size + post-processing), we decided to do it the safest way and not use any of them and stack models trained with different architecture (B3-B7 + seresnext50) and different image sizes (384, 512).
\nWith stacking, we managed to reduce the gap between CV/LB to 0 in our best selected submission, where CV=LB=0.9483!

\n

Conclusion:

\n

We ended up with a very correct solution that is missing the special ingredient to make it a top solution. However, I am happy to see that the ideas we have been testing actually work, even though not using them in our final solution penalized our private LB score. Those are techniques learned to be used in future competitions.

\n

Best private and public subs:

\n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
SubmissionCVPublic LBPrivate LB
Best public N/A0.96570.9374
Best privateN/A0.96130.9410
\n

3 Selected submissions

\n\n \n \n \n \n \n\n \n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n
SubmissionCVPublic LBPrivate LB
Most stable0.94830.94830.9374
Best CV0.95410.95100.9359
2 Best CV0.95350.95260.9337
", "rawMarkdown": "First of all, I would like to thank the organizers, Kaggle and my teammates @underwearfitting, @hawkey and @nicohrubec for this amazing experience.\n\nI am happy that I picked SIIM-Melanoma to be my first competition because it had everything: A great leader @cdeotte, active teammates, unstable public LB, a terrible shake-up and finally, the drama related to [the participation of the brilliant students of the medal farming academy](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/174635).\n\nWe won a bronze medal thanks to the clean up done by kaggle and the investigation started by @group16. Big thanks!\n\nAs most of the participants, we didn’t select our best submission that would have landed us a silver medal. Actually 60 submissions scored higher than our best selected sub. This is part of the game, taking good decisions is an essential skill in ML/DL. Here is a first glimpse of our submissions.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3451735%2F45cdd5353e253758de95d199217756ce%2FScreen%20Shot%202020-08-21%20at%2017.03.07.png?generation=1598015028261645&alt=media)\n### First time crossing the medal zone:\nThe first time I crossed the medal zone was the 15th of July with private LB= 0.9375 (Slightly higher than our best selected submission).\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3451735%2F7ccedce9337d62222933c5ae7acf95ea%2FScreen%20Shot%202020-08-21%20at%2017.00.32.png?generation=1598014866149997&alt=media)\nWhy did this sub score higher than our selected submission? Because it had an ingredient that was missing in the final sub: Higher `image_size= 768`. I was experimenting with higher image sizes but keeping the CNN's `input_size` smaller for faster training, it's a clear information loss/training time trade-off. \n```\nImage size= 512. Input size= 226\n\nImage size= 768. Input size= 384\n\nImage size= 1024 Input size= 512\n```\nIt scored higher than the models with `image_size` 512 or 384, which is good news, but I decided not to go further with this experiment because it was risky since parts of the images are lost with the center crops.\n\nI was happy with this decision until I saw @cdeotte's [solution](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175344) where he used smaller `input_size` with random crops instead of center crops at each epoch to make sure the model loops over all the pixels instead of center cropping the same pixels over the epochs. Another trick learned to be used in future competitions ;)\n\n### Teaming-up:\nThe day we teamed-up has seen a boost in our private LB score. It was just a rank ensemble of our best submissions. This shows the clear benefit of teaming up late because you will definitely add diversity to the final solution.\n\n### The team era:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3451735%2F0ff03771186de3fafc3b9eed04860d8c%2FScreen%20Shot%202020-08-21%20at%2017.12.11.png?generation=1598015689624500&alt=media)\nYou can see in the plot that the best era was after teaming up because we experimented many ideas together that helped us reach the silver zone at our peak. I am not gonna lie, at that point I thought we were overfitting to public LB because most of the silver zone submissions are nested blends of our +0.95 submissions. However, the correlation of our public LB scores in the range (0.955, 0.9642) with private LB was 0.89! The question here is: When did we start overfitting?\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3451735%2Fa319913972edb44e93fc1c68c56dd4c2%2FScreen%20Shot%202020-08-21%20at%2017.24.13.png?generation=1598016297260469&alt=media)\nOverfitting (the decay) starts after including the [last public notebook in our blend that was terribly overfitting (Public= 0.96, Private= 0.91)](https://www.kaggle.com/paklau9/minmax-highest-public-lb-9619). The blends scoring +0.9643 included overfitted public notebooks\n \nIt is worth mentioning that the highest public/private LB correlation we had was with subs in the medal zone, so our subs definitely had something special to score high private scores without any CV strategy. What ideas or ingredients did the silver submissions have? Simple answer: Post-processing.\n\n\n### Post-processing:\nWe had 2 post-processing techniques, they both boost the public and private LB score.\nThe first was a finding by @underwearfitting where he set all the palms/soles and oral/genital predictions to 0, code below:\n```\nps= test.loc[test['anatom_site_general_challenge']=='palms/soles'].image_name\nog= test.loc[test['anatom_site_general_challenge']=='oral/genital'].image_name\nsubmission= submission.set_index('image_name')\nsubmission.loc[ps]=0\nsubmission.loc[og]=0\n```\n\n\nThis pp gives a +0.001 boost in private LB (Try it at home! +0.001 boost guaranteed 😉).\n\nThe second was image embeddings extraction using RAPIDS cuML TSNE shared by @cdeotte, thanks Chris for everything you shared in this competition.\n@hawkey tried to detect the malignant images in the test set by extracting the patients around the red area (malignants) as they are more susceptible. This pp was unstable but led to some high private LB scores.\n(If you have any questions about this method, please tag @hawkey in the comment)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3451735%2F55bebda96a72c7bca0fc39dfa2bae19b%2FScreen%20Shot%202020-08-21%20at%2016.36.14.png?generation=1598013477503902&alt=media):\n\n### The final solution :\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3451735%2F6312bdd9b6c769f81efb3dccdeefaccd%2FScreen%20Shot%202020-08-23%20at%2010.36.49.png?generation=1598164656057343&alt=media)\n**The blue line marks the switch from trusting(climbing) LB to trusting CV.**\n\nThe final step in this competition was to prepare a solid solution. Even though we had some high scoring subs, interesting ideas (training with high resolution images and smaller CNN input size + post-processing), we decided to do it the safest way and not use any of them and stack models trained with different architecture (B3-B7 + seresnext50) and different image sizes (384, 512).\nWith stacking, we managed to reduce the gap between CV/LB to 0 in our best selected submission, where CV=LB=0.9483!\n\n\n### Conclusion:\nWe ended up with a very correct solution that is missing the special ingredient to make it a top solution. However, I am happy to see that the ideas we have been testing actually work, even though not using them in our final solution penalized our private LB score. Those are techniques learned to be used in future competitions.\n\n**Best private and public subs:**\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n
SubmissionCVPublic LBPrivate LB
Best public N/A0.96570.9374
Best privateN/A0.96130.9410
\n**3 Selected submissions** \n\n\n \n \n \n \n \n \n \n\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
SubmissionCVPublic LBPrivate LB
Most stable0.94830.94830.9374
Best CV0.95410.95100.9359
2 Best CV0.95350.95260.9337
\n", "votes": 21 }, { "id": 981964, "postDate": "2020-08-22T22:19:29.583Z", "content": "

It was a great pleasure teaming up with you Amin. I definitely learned a lot from you and I really enjoyed our teamwork. We relentlessly trained so many models here and there, expecting to ensemble to get good results. However, this time we lost to high quality models, in other words, we couldn't build models that score around 0.94 CV due to our lack of experience. We learned a lot in this competition from @cdeotte about the usage of TPU and TSNE projections, now the winning solutions from all the winners; I can't say I am not satisfied😄. It was great to have diverse teammates like @hawkey @amiiiney @nicohrubec, coming from different background and cultures. Hoping to connect with you all soon.😬

", "rawMarkdown": "It was a great pleasure teaming up with you Amin. I definitely learned a lot from you and I really enjoyed our teamwork. We relentlessly trained so many models here and there, expecting to ensemble to get good results. However, this time we lost to high quality models, in other words, we couldn't build models that score around 0.94 CV due to our lack of experience. We learned a lot in this competition from @cdeotte about the usage of TPU and TSNE projections, now the winning solutions from all the winners; I can't say I am not satisfied😄. It was great to have diverse teammates like @hawkey @amiiiney @nicohrubec, coming from different background and cultures. Hoping to connect with you all soon.😬", "votes": 3, "replies": [ { "id": 981969, "postDate": "2020-08-22T22:31:23.040Z", "content": "

@underwearfitting Thanks Eric for your kind words! Teaming up was the best thing I did in this competition, I definitely learned a lot from you, you would make a great professor after your PhD 😊
\nI also wouldn't say I am not satisfied with the results because it was a solid solution that was missing some ingredients that we could find and learn from the top solutions, now our baseline/pipeline is richer and ready for new challenges :)

", "rawMarkdown": "@underwearfitting Thanks Eric for your kind words! Teaming up was the best thing I did in this competition, I definitely learned a lot from you, you would make a great professor after your PhD 😊\nI also wouldn't say I am not satisfied with the results because it was a solid solution that was missing some ingredients that we could find and learn from the top solutions, now our baseline/pipeline is richer and ready for new challenges :)", "votes": 1 } ] }, { "id": 980877, "postDate": "2020-08-22T00:30:11.910Z", "content": "

Wow, that's magic. My best (unselected submission) was Private LB 0.9450. When I use your trick it increases to Private LB 0.9459 Gold Medal !!

\n

\"\"

\n

I did some quick probing where I set all targets to zero and then all of a specific body part to 1. From the submissions we can see that it is rare for there to be positive cases on soles, palms, oral, and genital. Great detective work!

\n

\"\"

", "rawMarkdown": "Wow, that's magic. My best (unselected submission) was Private LB 0.9450. When I use your trick it increases to Private LB 0.9459 Gold Medal !!\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2Fe6cbf0c0f839fe713317ddbf17a1d161%2Fbest.png?generation=1598055407574078&alt=media)\n\nI did some quick probing where I set all targets to zero and then all of a specific body part to 1. From the submissions we can see that it is rare for there to be positive cases on soles, palms, oral, and genital. Great detective work!\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2Fdaea882ba0b991086abdcc7142decef2%2Fprobe.png?generation=1598056151310064&alt=media)", "votes": 4, "replies": [ { "id": 980878, "postDate": "2020-08-22T00:38:14.563Z", "content": "

Actually the above probes prove there are no malignant palm, sole, oral, nor genital in test.

\n

There are approximately 115 images out of 10,000 test images with palm/sole (based on fact that train has 1.1% palm/sole). And there are approximately 38 test images with oral/genital. So these probe submissions, imply there are no malignant palm, sole, oral, nor genital in the test images. Since a benign will lower LB score by approx 0.00015 and a malignant would increase 0.006

", "rawMarkdown": "Actually the above probes prove there are no malignant palm, sole, oral, nor genital in test.\n\nThere are approximately 115 images out of 10,000 test images with palm/sole (based on fact that train has 1.1% palm/sole). And there are approximately 38 test images with oral/genital. So these probe submissions, imply there are no malignant palm, sole, oral, nor genital in the test images. Since a benign will lower LB score by approx 0.00015 and a malignant would increase 0.006", "votes": 2 }, { "id": 981913, "postDate": "2020-08-22T20:46:33.790Z", "content": "

Thanks @cdeotte for trying our method (by detective @underwearfitting :) )! We haven't used it either in our final solution, unfortunately, it turned out that there was really no palm, sole, oral nor genital in test, but in case there were, as you said, this could lower significantly our private score, maybe we should have risked with this pp in one of the 3 subs.

", "rawMarkdown": "Thanks @cdeotte for trying our method (by detective @underwearfitting :) )! We haven't used it either in our final solution, unfortunately, it turned out that there was really no palm, sole, oral nor genital in test, but in case there were, as you said, this could lower significantly our private score, maybe we should have risked with this pp in one of the 3 subs.", "votes": 1 }, { "id": 981955, "postDate": "2020-08-22T22:08:46.170Z", "content": "

@cdeotte thanks for trying this. We also learned a lesson about trying out something risky!

", "rawMarkdown": "@cdeotte thanks for trying this. We also learned a lesson about trying out something risky!", "votes": 3 } ] }, { "id": 980868, "postDate": "2020-08-22T00:09:31.703Z", "content": "

Great analysis. I love all the plots. It tells a great story. Ensembling and teaming help increase score. What's the blue vertical line in the bottom plot? Your private LB scores were not after the blue line.

\n

When you say Image size= 512. Input size= 226, how did you reduce 512 to 226? Did you resize or crop? was it random or centered?

", "rawMarkdown": "Great analysis. I love all the plots. It tells a great story. Ensembling and teaming help increase score. What's the blue vertical line in the bottom plot? Your private LB scores were not after the blue line.\n\nWhen you say `Image size= 512. Input size= 226`, how did you reduce 512 to 226? Did you resize or crop? was it random or centered?", "votes": 1, "replies": [ { "id": 981908, "postDate": "2020-08-22T20:30:57.803Z", "content": "

Thank you @cdeotte for your compliment, it means a lot coming from you! And thank you again for all the knowledge you shared in this competition.

\n\n
  read_size         = 256, \n  crop_size         = 250, \n  net_size          = 224,\n
\n

This is the set of parameters he used. I tweaked the crop_size (in his prepare_imagefunction he used random_crop and centered_crop) but what made my training faster was not tweaking the crop_size but the net_size, I set the CNN net_size =224 with read_size=512 and I got better results than image_size=256 with net_size=224, probably I was doing something wrong but it had a good private score.

\n

I would like to learn more about your approach because this is a technique I would like to use in future competitions, how did you make you model random_crop different crops at each epoch?
\nIs this augmentation line of code enough to make the training go correctly?
\nimg = tf.image.random_crop(img, crop_size, crop_size, 3])

", "rawMarkdown": "Thank you @cdeotte for your compliment, it means a lot coming from you! And thank you again for all the knowledge you shared in this competition.\n\n* Regarding the blue vertical line: Before the blue line our strategy was training models and adding them to the blend, this way we were climbing LB. After the blue line we stopped blending, thus stopped climbing LB and started saving the oofs for stacking for a final stable solution. **The blue line marks the switch from trusting LB to trusting CV.**\n\n* I reduced 512 images to 226 but setting the CNN input_size to 226. I used [AgentAuers's notebook](https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once), he used 3 parameters,\n``` \n read_size = 256, \n crop_size = 250, \n net_size = 224,\n```\nThis is the set of parameters he used. I tweaked the crop_size (in his `prepare_image`function he used random_crop and centered_crop) but what made my training faster was not tweaking the crop_size but the net_size, I set the CNN `net_size =224` with `read_size=512` and I got better results than `image_size=256` with `net_size=224`, probably I was doing something wrong but it had a good private score.\n\nI would like to learn more about your approach because this is a technique I would like to use in future competitions, how did you make you model random_crop different crops at each epoch? \nIs this augmentation line of code enough to make the training go correctly?\n`img = tf.image.random_crop(img, crop_size, crop_size, 3])`\n" }, { "id": 981941, "postDate": "2020-08-22T21:41:59.443Z", "content": "

Using AgentAuers's terminology, your read size was 512 and your net size was 224. What what your crop size?

\n

The single line img = tf.image.random_crop(img, crop_size, crop_size, 3]) does random cropping each epoch if you put it inside the block if augment: inside prepare_image() function.

\n

Looking closely at AgentAuers's notebook now, I see he uses random cropping. His \"read_size\" are the TFRecords that are used. Then his \"crop_size\" is a random crop done each epoch. Lastly, his \"net_size\" is where he resizes (scales) the random crops. This serves as a blurring effect. The high resolution is reduced to lower resolution.

\n

Afterward, your network is effectively using X = read_size * net_size / crop_size as it's \"image input size\". So once you set those 3 variables, you can say that your network is learning from images sized X. And to make a diverse set of models you want to build multiple models using X = 256, 384, 512, 768, 1024.

", "rawMarkdown": "Using AgentAuers's terminology, your read size was 512 and your net size was 224. What what your crop size?\n\nThe single line `img = tf.image.random_crop(img, crop_size, crop_size, 3])` does random cropping each epoch if you put it inside the block `if augment:` inside ` prepare_image()` function.\n\nLooking closely at AgentAuers's notebook now, I see he uses random cropping. His \"read_size\" are the TFRecords that are used. Then his \"crop_size\" is a random crop done each epoch. Lastly, his \"net_size\" is where he resizes (scales) the random crops. This serves as a blurring effect. The high resolution is reduced to lower resolution.\n\nAfterward, your network is effectively using `X = read_size * net_size / crop_size` as it's \"image input size\". So once you set those 3 variables, you can say that your network is learning from images sized `X`. And to make a diverse set of models you want to build multiple models using `X = 256, 384, 512, 768, 1024`.", "votes": 1 }, { "id": 981948, "postDate": "2020-08-22T21:57:18.953Z", "content": "

Thanks again @cdeotte. My crop_size was 256 when image_size=512.
\nLets consider just the read_size and crop-size, I understand that if you set the crop size smaller than the read_size and inside the block if augment you use random_crop then ALL the images are random cropped differently at every epoch?

\n

In other words, was your approach similar to AgentAuer's notebook if you set read_size=512 and crop_size=256, with random_crop inside the if augment block?

", "rawMarkdown": "Thanks again @cdeotte. My crop_size was 256 when image_size=512.\nLets consider just the read_size and crop-size, I understand that if you set the crop size smaller than the read_size and inside the block `if augment` you use random_crop then **ALL** the images are random cropped differently at every epoch?\n\nIn other words, was your approach similar to AgentAuer's notebook if you set read_size=512 and crop_size=256, with random_crop inside the `if augment` block?\n\n", "votes": 1 }, { "id": 981952, "postDate": "2020-08-22T22:06:00.017Z", "content": "

Yes, my approach was exactly AgentAuer's notebook. (However i didn't realize that AgentAuer's was doing this when i added random crop to my private notebook forked from my public notebook).

\n

In my final writeup here, when i say read_size = 512 and crop_size = 256, then that is AgentAuer's read_size = 512, crop_size = 256, net_size = crop_size.

\n

Each epoch, our models will see different crops from the train images. (And after many epochs, it will have learned about the entire images). It looks like you were using random cropping too. And then when you used net_size = 224 after crop_size = 256, then you were slightly blurring the images and effectively reducing the image size from 512 to 448.

\n

(So when you used 512, 256, 224, then you were effectively reading 448 and cropping 224).

", "rawMarkdown": "Yes, my approach was exactly AgentAuer's notebook. (However i didn't realize that AgentAuer's was doing this when i added random crop to my private notebook forked from my public notebook).\n\nIn my final writeup [here][1], when i say `read_size = 512` and `crop_size = 256`, then that is AgentAuer's `read_size = 512`, `crop_size = 256`, `net_size = crop_size`.\n\nEach epoch, our models will see **different crops** from the train images. (And after many epochs, it will have learned about the entire images). It looks like you were using random cropping too. And then when you used `net_size = 224` after `crop_size = 256`, then you were slightly blurring the images and effectively reducing the image size from 512 to 448.\n\n(So when you used 512, 256, 224, then you were effectively reading 448 and cropping 224).\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175344", "votes": 1 }, { "id": 981966, "postDate": "2020-08-22T22:20:15.633Z", "content": "

Thanks a lot @cdeotte for your explanation. That's exactly what I needed to understand, I actually gave up on this method because I didn't fully understand it, now that I understand, training stronger models will be more affordable in the future.

\n

One last question: in this competition, the target was small (the Mole), so random cropping mostly will crop and include the whole Mole. Do you think this method can be effective in competitions where the target covers a larger area in the image?

", "rawMarkdown": "Thanks a lot @cdeotte for your explanation. That's exactly what I needed to understand, I actually gave up on this method because I didn't fully understand it, now that I understand, training stronger models will be more affordable in the future.\n\nOne last question: in this competition, the target was small (the Mole), so random cropping **mostly** will crop and include the whole Mole. Do you think this method can be effective in competitions where the target covers a larger area in the image? ", "votes": 1 }, { "id": 981992, "postDate": "2020-08-22T23:39:44.700Z", "content": "

Random cropping is one type of augmentation. In every image comp, you need to discover what are the best types of augmentation. The goal of augmentation is to make more training data. When experimenting with augmentation, just look at the result and ask \"are these new images helpful if i were training a human?\"

\n

Random cropping is nearly always effective because you can apply it gently. For example, when you have 512x512, you can start by using 480x480 crops. (Note that this is similar to shift by 32 pixels augmentation). There's no harm there. Using 256x256 crops within 512 is aggressive and it worked in this comp because there was so much unnecessary blank space in each image and the mole was centered (so every random crop captured a piece of mole). Such an aggressive crop doesn't always work.

", "rawMarkdown": "Random cropping is one type of augmentation. In every image comp, you need to discover what are the best types of augmentation. The goal of augmentation is to make more training data. When experimenting with augmentation, just look at the result and ask \"are these new images helpful if i were training a human?\"\n\nRandom cropping is nearly always effective because you can apply it gently. For example, when you have 512x512, you can start by using 480x480 crops. (Note that this is similar to shift by 32 pixels augmentation). There's no harm there. Using 256x256 crops within 512 is aggressive and it worked in this comp because there was so much unnecessary blank space in each image and the mole was centered (so every random crop captured a piece of mole). Such an aggressive crop doesn't always work.", "votes": 1 } ] }, { "id": 980545, "postDate": "2020-08-21T17:11:54.897Z", "content": "

Very nice meta analysis, thanks for sharing. Interesting to see the result of stacking.

", "rawMarkdown": "Very nice meta analysis, thanks for sharing. Interesting to see the result of stacking.", "votes": 1, "replies": [ { "id": 980575, "postDate": "2020-08-21T17:39:11.600Z", "content": "

Thank you @fchmiel! I think stacking would have scored higher with deeper architectures and higher image resolutions.
\nAnother interesting thing is that in our case, we could have trust public LB because it had high correlation with private LB (if we don't consider the subs including overfitted public notebooks).

", "rawMarkdown": "Thank you @fchmiel! I think stacking would have scored higher with deeper architectures and higher image resolutions.\nAnother interesting thing is that in our case, we could have trust public LB because it had high correlation with private LB (if we don't consider the subs including overfitted public notebooks)." }, { "id": 981052, "postDate": "2020-08-22T06:15:24.270Z", "content": "

Have did you do the cross-validation for stacking?

\n

I thought about trying some stacking, but settled for a blend, because I couldn't properly validate it without using a hold-out test set and I didn't want to waste the data. I guess one could use leaderboard feeback as your hold out test set but this prooved risky in this competition!

", "rawMarkdown": "Have did you do the cross-validation for stacking? \n\nI thought about trying some stacking, but settled for a blend, because I couldn't properly validate it without using a hold-out test set and I didn't want to waste the data. I guess one could use leaderboard feeback as your hold out test set but this prooved risky in this competition!" }, { "id": 981910, "postDate": "2020-08-22T20:42:03.463Z", "content": "

Yes @fchmiel We used 5 fold strategy and did rank ensemble the 5 folds to get the final CV score, which was very stable compared to LB. Stacking can give you a stable CV that you can trust. In blending you just trust the LB feedback, which in this worked out well for you, congrats for the medal :) ( blending worked well for us as well, but we didn't trust LB).

", "rawMarkdown": "Yes @fchmiel We used 5 fold strategy and did rank ensemble the 5 folds to get the final CV score, which was very stable compared to LB. Stacking can give you a stable CV that you can trust. In blending you just trust the LB feedback, which in this worked out well for you, congrats for the medal :) ( blending worked well for us as well, but we didn't trust LB)." } ] }, { "id": 980879, "postDate": "2020-08-22T00:44:49.343Z", "content": "
\n

@hawkey tried to detect the malignant images in the test set by extracting the patients around the red area (malignants) as they are more susceptible. This pp was unstable but led to some high private LB scores.

\n
\n

Great use of t-SNE. I really like this idea.

", "rawMarkdown": "> @hawkey tried to detect the malignant images in the test set by extracting the patients around the red area (malignants) as they are more susceptible. This pp was unstable but led to some high private LB scores.\n\nGreat use of t-SNE. I really like this idea.", "votes": 2, "replies": [ { "id": 981953, "postDate": "2020-08-22T22:06:07.407Z", "content": "

@cdeotte Chris, we abandoned this because we extracted embedding from different architectures and they seem to produce different embeddings, therefore unstable.

", "rawMarkdown": "@cdeotte Chris, we abandoned this because we extracted embedding from different architectures and they seem to produce different embeddings, therefore unstable.", "votes": 3 } ] }, { "id": 981306, "postDate": "2020-08-22T11:05:16.697Z", "rawMarkdown": "", "votes": 2, "isDeleted": true, "replies": [ { "id": 981917, "postDate": "2020-08-22T20:50:58.207Z", "content": "

Thanks @synked for your feedback! Regarding stacking. We used:

\n", "rawMarkdown": "Thanks @synked for your feedback! Regarding stacking. We used:\n* 5 folds CV strategy.\n* Optuna for hyperparameters tuning\n* Metamodels: LGB, XGB and catboost to stack our best oofs.", "votes": 1 } ] } ], "comments": [ { "id": 981964, "author_name": "sin", "author_url": "", "post_date": "2020-08-22T22:19:29.583000", "content": "

It was a great pleasure teaming up with you Amin. I definitely learned a lot from you and I really enjoyed our teamwork. We relentlessly trained so many models here and there, expecting to ensemble to get good results. However, this time we lost to high quality models, in other words, we couldn't build models that score around 0.94 CV due to our lack of experience. We learned a lot in this competition from @cdeotte about the usage of TPU and TSNE projections, now the winning solutions from all the winners; I can't say I am not satisfied😄. It was great to have diverse teammates like @hawkey @amiiiney @nicohrubec, coming from different background and cultures. Hoping to connect with you all soon.😬

", "votes": 3, "replies": [ { "id": 981969, "author_name": "Amin", "author_url": "", "post_date": "2020-08-22T22:31:23.040000", "content": "

@underwearfitting Thanks Eric for your kind words! Teaming up was the best thing I did in this competition, I definitely learned a lot from you, you would make a great professor after your PhD 😊
\nI also wouldn't say I am not satisfied with the results because it was a solid solution that was missing some ingredients that we could find and learn from the top solutions, now our baseline/pipeline is richer and ready for new challenges :)

", "votes": 1, "replies": [] } ] }, { "id": 980877, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-22T00:30:11.910000", "content": "

Wow, that's magic. My best (unselected submission) was Private LB 0.9450. When I use your trick it increases to Private LB 0.9459 Gold Medal !!

\n

\"\"

\n

I did some quick probing where I set all targets to zero and then all of a specific body part to 1. From the submissions we can see that it is rare for there to be positive cases on soles, palms, oral, and genital. Great detective work!

\n

\"\"

", "votes": 4, "replies": [ { "id": 980878, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-22T00:38:14.563000", "content": "

Actually the above probes prove there are no malignant palm, sole, oral, nor genital in test.

\n

There are approximately 115 images out of 10,000 test images with palm/sole (based on fact that train has 1.1% palm/sole). And there are approximately 38 test images with oral/genital. So these probe submissions, imply there are no malignant palm, sole, oral, nor genital in the test images. Since a benign will lower LB score by approx 0.00015 and a malignant would increase 0.006

", "votes": 2, "replies": [] }, { "id": 981913, "author_name": "Amin", "author_url": "", "post_date": "2020-08-22T20:46:33.790000", "content": "

Thanks @cdeotte for trying our method (by detective @underwearfitting :) )! We haven't used it either in our final solution, unfortunately, it turned out that there was really no palm, sole, oral nor genital in test, but in case there were, as you said, this could lower significantly our private score, maybe we should have risked with this pp in one of the 3 subs.

", "votes": 1, "replies": [] }, { "id": 981955, "author_name": "sin", "author_url": "", "post_date": "2020-08-22T22:08:46.170000", "content": "

@cdeotte thanks for trying this. We also learned a lesson about trying out something risky!

", "votes": 3, "replies": [] } ] }, { "id": 980868, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-22T00:09:31.703000", "content": "

Great analysis. I love all the plots. It tells a great story. Ensembling and teaming help increase score. What's the blue vertical line in the bottom plot? Your private LB scores were not after the blue line.

\n

When you say Image size= 512. Input size= 226, how did you reduce 512 to 226? Did you resize or crop? was it random or centered?

", "votes": 1, "replies": [ { "id": 981908, "author_name": "Amin", "author_url": "", "post_date": "2020-08-22T20:30:57.803000", "content": "

Thank you @cdeotte for your compliment, it means a lot coming from you! And thank you again for all the knowledge you shared in this competition.

\n\n
  read_size         = 256, \n  crop_size         = 250, \n  net_size          = 224,\n
\n

This is the set of parameters he used. I tweaked the crop_size (in his prepare_imagefunction he used random_crop and centered_crop) but what made my training faster was not tweaking the crop_size but the net_size, I set the CNN net_size =224 with read_size=512 and I got better results than image_size=256 with net_size=224, probably I was doing something wrong but it had a good private score.

\n

I would like to learn more about your approach because this is a technique I would like to use in future competitions, how did you make you model random_crop different crops at each epoch?
\nIs this augmentation line of code enough to make the training go correctly?
\nimg = tf.image.random_crop(img, crop_size, crop_size, 3])

", "votes": 0, "replies": [] }, { "id": 981941, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-22T21:41:59.443000", "content": "

Using AgentAuers's terminology, your read size was 512 and your net size was 224. What what your crop size?

\n

The single line img = tf.image.random_crop(img, crop_size, crop_size, 3]) does random cropping each epoch if you put it inside the block if augment: inside prepare_image() function.

\n

Looking closely at AgentAuers's notebook now, I see he uses random cropping. His \"read_size\" are the TFRecords that are used. Then his \"crop_size\" is a random crop done each epoch. Lastly, his \"net_size\" is where he resizes (scales) the random crops. This serves as a blurring effect. The high resolution is reduced to lower resolution.

\n

Afterward, your network is effectively using X = read_size * net_size / crop_size as it's \"image input size\". So once you set those 3 variables, you can say that your network is learning from images sized X. And to make a diverse set of models you want to build multiple models using X = 256, 384, 512, 768, 1024.

", "votes": 1, "replies": [] }, { "id": 981948, "author_name": "Amin", "author_url": "", "post_date": "2020-08-22T21:57:18.953000", "content": "

Thanks again @cdeotte. My crop_size was 256 when image_size=512.
\nLets consider just the read_size and crop-size, I understand that if you set the crop size smaller than the read_size and inside the block if augment you use random_crop then ALL the images are random cropped differently at every epoch?

\n

In other words, was your approach similar to AgentAuer's notebook if you set read_size=512 and crop_size=256, with random_crop inside the if augment block?

", "votes": 1, "replies": [] }, { "id": 981952, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-22T22:06:00.017000", "content": "

Yes, my approach was exactly AgentAuer's notebook. (However i didn't realize that AgentAuer's was doing this when i added random crop to my private notebook forked from my public notebook).

\n

In my final writeup here, when i say read_size = 512 and crop_size = 256, then that is AgentAuer's read_size = 512, crop_size = 256, net_size = crop_size.

\n

Each epoch, our models will see different crops from the train images. (And after many epochs, it will have learned about the entire images). It looks like you were using random cropping too. And then when you used net_size = 224 after crop_size = 256, then you were slightly blurring the images and effectively reducing the image size from 512 to 448.

\n

(So when you used 512, 256, 224, then you were effectively reading 448 and cropping 224).

", "votes": 1, "replies": [] }, { "id": 981966, "author_name": "Amin", "author_url": "", "post_date": "2020-08-22T22:20:15.633000", "content": "

Thanks a lot @cdeotte for your explanation. That's exactly what I needed to understand, I actually gave up on this method because I didn't fully understand it, now that I understand, training stronger models will be more affordable in the future.

\n

One last question: in this competition, the target was small (the Mole), so random cropping mostly will crop and include the whole Mole. Do you think this method can be effective in competitions where the target covers a larger area in the image?

", "votes": 1, "replies": [] }, { "id": 981992, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-22T23:39:44.700000", "content": "

Random cropping is one type of augmentation. In every image comp, you need to discover what are the best types of augmentation. The goal of augmentation is to make more training data. When experimenting with augmentation, just look at the result and ask \"are these new images helpful if i were training a human?\"

\n

Random cropping is nearly always effective because you can apply it gently. For example, when you have 512x512, you can start by using 480x480 crops. (Note that this is similar to shift by 32 pixels augmentation). There's no harm there. Using 256x256 crops within 512 is aggressive and it worked in this comp because there was so much unnecessary blank space in each image and the mole was centered (so every random crop captured a piece of mole). Such an aggressive crop doesn't always work.

", "votes": 1, "replies": [] } ] }, { "id": 980545, "author_name": "FChmiel", "author_url": "", "post_date": "2020-08-21T17:11:54.897000", "content": "

Very nice meta analysis, thanks for sharing. Interesting to see the result of stacking.

", "votes": 1, "replies": [ { "id": 980575, "author_name": "Amin", "author_url": "", "post_date": "2020-08-21T17:39:11.600000", "content": "

Thank you @fchmiel! I think stacking would have scored higher with deeper architectures and higher image resolutions.
\nAnother interesting thing is that in our case, we could have trust public LB because it had high correlation with private LB (if we don't consider the subs including overfitted public notebooks).

", "votes": 0, "replies": [] }, { "id": 981052, "author_name": "FChmiel", "author_url": "", "post_date": "2020-08-22T06:15:24.270000", "content": "

Have did you do the cross-validation for stacking?

\n

I thought about trying some stacking, but settled for a blend, because I couldn't properly validate it without using a hold-out test set and I didn't want to waste the data. I guess one could use leaderboard feeback as your hold out test set but this prooved risky in this competition!

", "votes": 0, "replies": [] }, { "id": 981910, "author_name": "Amin", "author_url": "", "post_date": "2020-08-22T20:42:03.463000", "content": "

Yes @fchmiel We used 5 fold strategy and did rank ensemble the 5 folds to get the final CV score, which was very stable compared to LB. Stacking can give you a stable CV that you can trust. In blending you just trust the LB feedback, which in this worked out well for you, congrats for the medal :) ( blending worked well for us as well, but we didn't trust LB).

", "votes": 0, "replies": [] } ] }, { "id": 980879, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-22T00:44:49.343000", "content": "
\n

@hawkey tried to detect the malignant images in the test set by extracting the patients around the red area (malignants) as they are more susceptible. This pp was unstable but led to some high private LB scores.

\n
\n

Great use of t-SNE. I really like this idea.

", "votes": 2, "replies": [ { "id": 981953, "author_name": "sin", "author_url": "", "post_date": "2020-08-22T22:06:07.407000", "content": "

@cdeotte Chris, we abandoned this because we extracted embedding from different architectures and they seem to produce different embeddings, therefore unstable.

", "votes": 3, "replies": [] } ] }, { "id": 981306, "author_name": "", "author_url": "", "post_date": "2020-08-22T11:05:16.697000", "content": "", "votes": 2, "replies": [ { "id": 981917, "author_name": "Amin", "author_url": "", "post_date": "2020-08-22T20:50:58.207000", "content": "

Thanks @synked for your feedback! Regarding stacking. We used:

\n", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "980349": "First of all, I would like to thank the organizers, Kaggle and my teammates @underwearfitting, @hawkey and @nicohrubec for this amazing experience.\n\nI am happy that I picked SIIM-Melanoma to be my first competition because it had everything: A great leader @cdeotte, active teammates, unstable public LB, a terrible shake-up and finally, the drama related to [the participation of the brilliant students of the medal farming academy](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/174635).\n\nWe won a bronze medal thanks to the clean up done by kaggle and the investigation started by @group16. Big thanks!\n\nAs most of the participants, we didn’t select our best submission that would have landed us a silver medal. Actually 60 submissions scored higher than our best selected sub. This is part of the game, taking good decisions is an essential skill in ML/DL. Here is a first glimpse of our submissions.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3451735%2F45cdd5353e253758de95d199217756ce%2FScreen%20Shot%202020-08-21%20at%2017.03.07.png?generation=1598015028261645&alt=media)\n### First time crossing the medal zone:\nThe first time I crossed the medal zone was the 15th of July with private LB= 0.9375 (Slightly higher than our best selected submission).\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3451735%2F7ccedce9337d62222933c5ae7acf95ea%2FScreen%20Shot%202020-08-21%20at%2017.00.32.png?generation=1598014866149997&alt=media)\nWhy did this sub score higher than our selected submission? Because it had an ingredient that was missing in the final sub: Higher `image_size= 768`. I was experimenting with higher image sizes but keeping the CNN's `input_size` smaller for faster training, it's a clear information loss/training time trade-off. \n```\nImage size= 512. Input size= 226\n\nImage size= 768. Input size= 384\n\nImage size= 1024 Input size= 512\n```\nIt scored higher than the models with `image_size` 512 or 384, which is good news, but I decided not to go further with this experiment because it was risky since parts of the images are lost with the center crops.\n\nI was happy with this decision until I saw @cdeotte's [solution](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175344) where he used smaller `input_size` with random crops instead of center crops at each epoch to make sure the model loops over all the pixels instead of center cropping the same pixels over the epochs. Another trick learned to be used in future competitions ;)\n\n### Teaming-up:\nThe day we teamed-up has seen a boost in our private LB score. It was just a rank ensemble of our best submissions. This shows the clear benefit of teaming up late because you will definitely add diversity to the final solution.\n\n### The team era:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3451735%2F0ff03771186de3fafc3b9eed04860d8c%2FScreen%20Shot%202020-08-21%20at%2017.12.11.png?generation=1598015689624500&alt=media)\nYou can see in the plot that the best era was after teaming up because we experimented many ideas together that helped us reach the silver zone at our peak. I am not gonna lie, at that point I thought we were overfitting to public LB because most of the silver zone submissions are nested blends of our +0.95 submissions. However, the correlation of our public LB scores in the range (0.955, 0.9642) with private LB was 0.89! The question here is: When did we start overfitting?\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3451735%2Fa319913972edb44e93fc1c68c56dd4c2%2FScreen%20Shot%202020-08-21%20at%2017.24.13.png?generation=1598016297260469&alt=media)\nOverfitting (the decay) starts after including the [last public notebook in our blend that was terribly overfitting (Public= 0.96, Private= 0.91)](https://www.kaggle.com/paklau9/minmax-highest-public-lb-9619). The blends scoring +0.9643 included overfitted public notebooks\n \nIt is worth mentioning that the highest public/private LB correlation we had was with subs in the medal zone, so our subs definitely had something special to score high private scores without any CV strategy. What ideas or ingredients did the silver submissions have? Simple answer: Post-processing.\n\n\n### Post-processing:\nWe had 2 post-processing techniques, they both boost the public and private LB score.\nThe first was a finding by @underwearfitting where he set all the palms/soles and oral/genital predictions to 0, code below:\n```\nps= test.loc[test['anatom_site_general_challenge']=='palms/soles'].image_name\nog= test.loc[test['anatom_site_general_challenge']=='oral/genital'].image_name\nsubmission= submission.set_index('image_name')\nsubmission.loc[ps]=0\nsubmission.loc[og]=0\n```\n\n\nThis pp gives a +0.001 boost in private LB (Try it at home! +0.001 boost guaranteed 😉).\n\nThe second was image embeddings extraction using RAPIDS cuML TSNE shared by @cdeotte, thanks Chris for everything you shared in this competition.\n@hawkey tried to detect the malignant images in the test set by extracting the patients around the red area (malignants) as they are more susceptible. This pp was unstable but led to some high private LB scores.\n(If you have any questions about this method, please tag @hawkey in the comment)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3451735%2F55bebda96a72c7bca0fc39dfa2bae19b%2FScreen%20Shot%202020-08-21%20at%2016.36.14.png?generation=1598013477503902&alt=media):\n\n### The final solution :\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3451735%2F6312bdd9b6c769f81efb3dccdeefaccd%2FScreen%20Shot%202020-08-23%20at%2010.36.49.png?generation=1598164656057343&alt=media)\n**The blue line marks the switch from trusting(climbing) LB to trusting CV.**\n\nThe final step in this competition was to prepare a solid solution. Even though we had some high scoring subs, interesting ideas (training with high resolution images and smaller CNN input size + post-processing), we decided to do it the safest way and not use any of them and stack models trained with different architecture (B3-B7 + seresnext50) and different image sizes (384, 512).\nWith stacking, we managed to reduce the gap between CV/LB to 0 in our best selected submission, where CV=LB=0.9483!\n\n\n### Conclusion:\nWe ended up with a very correct solution that is missing the special ingredient to make it a top solution. However, I am happy to see that the ideas we have been testing actually work, even though not using them in our final solution penalized our private LB score. Those are techniques learned to be used in future competitions.\n\n**Best private and public subs:**\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n
SubmissionCVPublic LBPrivate LB
Best public N/A0.96570.9374
Best privateN/A0.96130.9410
\n**3 Selected submissions** \n\n\n \n \n \n \n \n \n \n\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
SubmissionCVPublic LBPrivate LB
Most stable0.94830.94830.9374
Best CV0.95410.95100.9359
2 Best CV0.95350.95260.9337
\n", "981964": "It was a great pleasure teaming up with you Amin. I definitely learned a lot from you and I really enjoyed our teamwork. We relentlessly trained so many models here and there, expecting to ensemble to get good results. However, this time we lost to high quality models, in other words, we couldn't build models that score around 0.94 CV due to our lack of experience. We learned a lot in this competition from @cdeotte about the usage of TPU and TSNE projections, now the winning solutions from all the winners; I can't say I am not satisfied😄. It was great to have diverse teammates like @hawkey @amiiiney @nicohrubec, coming from different background and cultures. Hoping to connect with you all soon.😬", "980877": "Wow, that's magic. My best (unselected submission) was Private LB 0.9450. When I use your trick it increases to Private LB 0.9459 Gold Medal !!\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2Fe6cbf0c0f839fe713317ddbf17a1d161%2Fbest.png?generation=1598055407574078&alt=media)\n\nI did some quick probing where I set all targets to zero and then all of a specific body part to 1. From the submissions we can see that it is rare for there to be positive cases on soles, palms, oral, and genital. Great detective work!\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2Fdaea882ba0b991086abdcc7142decef2%2Fprobe.png?generation=1598056151310064&alt=media)", "980868": "Great analysis. I love all the plots. It tells a great story. Ensembling and teaming help increase score. What's the blue vertical line in the bottom plot? Your private LB scores were not after the blue line.\n\nWhen you say `Image size= 512. Input size= 226`, how did you reduce 512 to 226? Did you resize or crop? was it random or centered?", "980545": "Very nice meta analysis, thanks for sharing. Interesting to see the result of stacking.", "980879": "> @hawkey tried to detect the malignant images in the test set by extracting the patients around the red area (malignants) as they are more susceptible. This pp was unstable but led to some high private LB scores.\n\nGreat use of t-SNE. I really like this idea.", "981306": "" } }