{
  "id": 176338,
  "title": "24th place solution and my way to get it ",
  "url": "/competitions/siim-isic-melanoma-classification/writeups/ivan-razumov-24th-place-solution-and-my-way-to-get",
  "author_name": "",
  "post_date": "2020-09-27T15:49:06.197Z",
  "votes": 18,
  "comment_count": 11,
  "views": 0,
  "content": "<p>Hello everyone, congrats to all winners and thanks Kaggle and SIIM for this competition </p>\n<p>This is my second medal awarded competiton (finally I'm an expert, yeah!!!), and my second medical image analisys competiton. My first one was  <a href=\"https://www.kaggle.com/c/aptos2019-blindness-detection\" target=\"_blank\">APTOS Blindness Detection</a> last year and these are some similar problems I faced here:</p>\n<p><strong>Imbalanced classes</strong></p>\n<p>A common problem for many medical image analysis is imbalanced data distribution (few images of positive cases). There were 5 classes (or 5 degrees of disease, because the problem could be also interpreted as regression) and  2 classes strongly prevailed. In this competition it was binary classification case and the first one was much larger. The obvious way to fix classes balance was to use external data, and here is the second challenge:</p>\n<p><strong>Use of external data</strong> </p>\n<p>In both competitins we had to deal with quite a large amount of external data, you can use it to balance classes or just use all of this data. In APTOS many people pretrained their models on old external images and finetune in actual data (and so did I). Inept use of external data could decrease your score, old (external) and actual datasets were quite different. In SIIM old and actual data was much less different, so we were more free to use it.<br>\nI used 2018+2019 external data for all training pipeline, validating on 2020 data only. </p>\n<h2>My approcah</h2>\n<p>At the beginning I tried various light models (like EffNet b0-b3, ResNet18 with image size 256x256) using PyTorch and various augmentation techniques both for metadata and for images, experementing with focal loss and label smoothing. But maximum I achived was 0.933 on public LB. But in discussions participants  told about experiments with sizes like  512x512 and quite large models like EffNetB6, these kind of experiments was hard for me because I used PyTorch and GPUs only</p>\n<p>All my way in deep learning I had used PyTorch, but topics related to tensorflow and TPU appeared in discussion and public notebooks more and more, I could not ignore it anymore. Finally I found that using TPU I can train heavier models on larger images! </p>\n<p><strong>Making ensemble</strong></p>\n<p><a href=\"https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154683\" target=\"_blank\">2019 1st place solution</a> is blending of various versions of EfficientNet and various augmentation techniques and image sizes, so I decided to implement something like that</p>\n<p>I used pipeline like <a href=\"https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\" target=\"_blank\">Tripple Stratified KFold</a>, using 2018+2019+2020 data for training and only 2020 data for validation </p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>Image Size</th>\n<th>Initial Weights</th>\n<th>Public Score</th>\n<th>Private Score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>EfficientNetB5</td>\n<td>512</td>\n<td>ImageNet</td>\n<td>0.9524</td>\n<td>0.9200</td>\n</tr>\n<tr>\n<td>EfficientNetB5</td>\n<td>768</td>\n<td>ImageNet</td>\n<td>0.9469</td>\n<td>0.9185</td>\n</tr>\n<tr>\n<td>EfficientNetB5</td>\n<td>1024</td>\n<td>ImageNet</td>\n<td>0.9479</td>\n<td>0.9285</td>\n</tr>\n<tr>\n<td>EfficientNetB6</td>\n<td>384</td>\n<td>ImageNet</td>\n<td>0.9497</td>\n<td>0.9277</td>\n</tr>\n<tr>\n<td>EfficientNetB6</td>\n<td>512</td>\n<td>ImageNet</td>\n<td>0.9488</td>\n<td>0.9258</td>\n</tr>\n<tr>\n<td>EfficientNetB6</td>\n<td>768</td>\n<td>ImageNet</td>\n<td>0.9502</td>\n<td>0.9277</td>\n</tr>\n<tr>\n<td>EfficientNetB6</td>\n<td>1024</td>\n<td>ImageNet</td>\n<td>0.9478</td>\n<td>0.9287</td>\n</tr>\n<tr>\n<td>EfficientNetB7</td>\n<td>512</td>\n<td>ImageNet</td>\n<td>0.9480</td>\n<td>0.9239</td>\n</tr>\n<tr>\n<td>EfficientNetB7</td>\n<td>768</td>\n<td>ImageNet</td>\n<td>0.9379</td>\n<td>0.9083</td>\n</tr>\n<tr>\n<td>EfficientNetB5</td>\n<td>512</td>\n<td>NoisyStudent</td>\n<td>0.9480</td>\n<td>0.9194</td>\n</tr>\n<tr>\n<td>EfficientNetB6</td>\n<td>512</td>\n<td>NoisyStudent</td>\n<td>0.9435</td>\n<td>0.9242</td>\n</tr>\n<tr>\n<td>EfficientNetB7</td>\n<td>512</td>\n<td>NoisyStudent</td>\n<td>0.9510</td>\n<td>0.9218</td>\n</tr>\n<tr>\n<td>EfficientNetB5</td>\n<td>768</td>\n<td>NoisyStudent</td>\n<td>0.9543</td>\n<td>0.9295</td>\n</tr>\n<tr>\n<td>EfficientNetB6</td>\n<td>768</td>\n<td>NoisyStudent</td>\n<td>0.9424</td>\n<td>0.9218</td>\n</tr>\n<tr>\n<td>ResNet152</td>\n<td>512</td>\n<td>ImageNet</td>\n<td>0.9184</td>\n<td>0.8901</td>\n</tr>\n<tr>\n<td>InceptionResNetV2</td>\n<td>768</td>\n<td>ImageNet</td>\n<td>0.9338</td>\n<td>0.9161</td>\n</tr>\n<tr>\n<td>Blending</td>\n<td>---</td>\n<td>---</td>\n<td>0.9602</td>\n<td>0.9406</td>\n</tr>\n</tbody>\n</table>\n<p>Being based on  <a href=\"https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147\" target=\"_blank\">this idea</a>,  I mixed various models with various training image sizes. </p>\n<p><strong>Public submissions</strong></p>\n<p>We can't ignore it since have a public submission in a silver medal zone!<br>\nI also made two submmissions with mixing this ensemble and subs from some public kernels, like these ones <br>\n<a href=\"https://www.kaggle.com/truonghoang/stacking-ensemble-on-my-submissions\" target=\"_blank\">This one</a>, because it consists of  models like VGG and SeResNext, I think they are quite uncorrelated with my effnets</p>\n<p><a href=\"https://www.kaggle.com/datafan07/analysis-of-melanoma-metadata-and-effnet-ensemble\" target=\"_blank\">This one</a>, because it has good work on metadata analysis</p>\n<p>I did not use kernels kind of \"public blending\" or \"public minimax ensemble\" which were in fact combinations of others, trying to choose from public submissions these ones which correlate as little as possible with my ensemble and with each other. </p>\n<p>At the end days of competition I fell from ~80 position to ~240 and it was nervous, but after th shakeup I, like many others, was thrown quite high (about 200 positions)! <br>\nAnd I really not sure about reasons for it! On the one hand blending of all my models without any public submissions was about on 800 + position on Public LB but something like 140-150 on Private LB, on the other simple blending of public subs gained a silver zone.. I think this story teaches us not to trust Public LB and choose the most stable models for final sub. </p>\n<p>I think it was quite a lottery, many conducted successful experiments, but chose wrong final submissions. What about me - I think I was saved by quite an ensemble and careful use of public submissions</p>\n<p>Special thanks to <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> for his great <a href=\"https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\" target=\"_blank\">kernel</a> and TFRecords datasets covering almost all possible approaches. They helped me alot throughout the competition!</p>",
  "messages": [
    {
      "id": "980117",
      "postDate": "08/21/2020 10:39:01",
      "content": "<p>Hello everyone, congrats to all winners and thanks Kaggle and SIIM for this competition </p>\n<p>This is my second medal awarded competiton (finally I'm an expert, yeah!!!), and my second medical image analisys competiton. My first one was  <a href=\"https://www.kaggle.com/c/aptos2019-blindness-detection\" target=\"_blank\">APTOS Blindness Detection</a> last year and these are some similar problems I faced here:</p>\n<p><strong>Imbalanced classes</strong></p>\n<p>A common problem for many medical image analysis is imbalanced data distribution (few images of positive cases). There were 5 classes (or 5 degrees of disease, because the problem could be also interpreted as regression) and  2 classes strongly prevailed. In this competition it was binary classification case and the first one was much larger. The obvious way to fix classes balance was to use external data, and here is the second challenge:</p>\n<p><strong>Use of external data</strong> </p>\n<p>In both competitins we had to deal with quite a large amount of external data, you can use it to balance classes or just use all of this data. In APTOS many people pretrained their models on old external images and finetune in actual data (and so did I). Inept use of external data could decrease your score, old (external) and actual datasets were quite different. In SIIM old and actual data was much less different, so we were more free to use it.<br>\nI used 2018+2019 external data for all training pipeline, validating on 2020 data only. </p>\n<h2>My approcah</h2>\n<p>At the beginning I tried various light models (like EffNet b0-b3, ResNet18 with image size 256x256) using PyTorch and various augmentation techniques both for metadata and for images, experementing with focal loss and label smoothing. But maximum I achived was 0.933 on public LB. But in discussions participants  told about experiments with sizes like  512x512 and quite large models like EffNetB6, these kind of experiments was hard for me because I used PyTorch and GPUs only</p>\n<p>All my way in deep learning I had used PyTorch, but topics related to tensorflow and TPU appeared in discussion and public notebooks more and more, I could not ignore it anymore. Finally I found that using TPU I can train heavier models on larger images! </p>\n<p><strong>Making ensemble</strong></p>\n<p><a href=\"https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154683\" target=\"_blank\">2019 1st place solution</a> is blending of various versions of EfficientNet and various augmentation techniques and image sizes, so I decided to implement something like that</p>\n<p>I used pipeline like <a href=\"https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\" target=\"_blank\">Tripple Stratified KFold</a>, using 2018+2019+2020 data for training and only 2020 data for validation </p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>Image Size</th>\n<th>Initial Weights</th>\n<th>Public Score</th>\n<th>Private Score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>EfficientNetB5</td>\n<td>512</td>\n<td>ImageNet</td>\n<td>0.9524</td>\n<td>0.9200</td>\n</tr>\n<tr>\n<td>EfficientNetB5</td>\n<td>768</td>\n<td>ImageNet</td>\n<td>0.9469</td>\n<td>0.9185</td>\n</tr>\n<tr>\n<td>EfficientNetB5</td>\n<td>1024</td>\n<td>ImageNet</td>\n<td>0.9479</td>\n<td>0.9285</td>\n</tr>\n<tr>\n<td>EfficientNetB6</td>\n<td>384</td>\n<td>ImageNet</td>\n<td>0.9497</td>\n<td>0.9277</td>\n</tr>\n<tr>\n<td>EfficientNetB6</td>\n<td>512</td>\n<td>ImageNet</td>\n<td>0.9488</td>\n<td>0.9258</td>\n</tr>\n<tr>\n<td>EfficientNetB6</td>\n<td>768</td>\n<td>ImageNet</td>\n<td>0.9502</td>\n<td>0.9277</td>\n</tr>\n<tr>\n<td>EfficientNetB6</td>\n<td>1024</td>\n<td>ImageNet</td>\n<td>0.9478</td>\n<td>0.9287</td>\n</tr>\n<tr>\n<td>EfficientNetB7</td>\n<td>512</td>\n<td>ImageNet</td>\n<td>0.9480</td>\n<td>0.9239</td>\n</tr>\n<tr>\n<td>EfficientNetB7</td>\n<td>768</td>\n<td>ImageNet</td>\n<td>0.9379</td>\n<td>0.9083</td>\n</tr>\n<tr>\n<td>EfficientNetB5</td>\n<td>512</td>\n<td>NoisyStudent</td>\n<td>0.9480</td>\n<td>0.9194</td>\n</tr>\n<tr>\n<td>EfficientNetB6</td>\n<td>512</td>\n<td>NoisyStudent</td>\n<td>0.9435</td>\n<td>0.9242</td>\n</tr>\n<tr>\n<td>EfficientNetB7</td>\n<td>512</td>\n<td>NoisyStudent</td>\n<td>0.9510</td>\n<td>0.9218</td>\n</tr>\n<tr>\n<td>EfficientNetB5</td>\n<td>768</td>\n<td>NoisyStudent</td>\n<td>0.9543</td>\n<td>0.9295</td>\n</tr>\n<tr>\n<td>EfficientNetB6</td>\n<td>768</td>\n<td>NoisyStudent</td>\n<td>0.9424</td>\n<td>0.9218</td>\n</tr>\n<tr>\n<td>ResNet152</td>\n<td>512</td>\n<td>ImageNet</td>\n<td>0.9184</td>\n<td>0.8901</td>\n</tr>\n<tr>\n<td>InceptionResNetV2</td>\n<td>768</td>\n<td>ImageNet</td>\n<td>0.9338</td>\n<td>0.9161</td>\n</tr>\n<tr>\n<td>Blending</td>\n<td>---</td>\n<td>---</td>\n<td>0.9602</td>\n<td>0.9406</td>\n</tr>\n</tbody>\n</table>\n<p>Being based on  <a href=\"https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147\" target=\"_blank\">this idea</a>,  I mixed various models with various training image sizes. </p>\n<p><strong>Public submissions</strong></p>\n<p>We can't ignore it since have a public submission in a silver medal zone!<br>\nI also made two submmissions with mixing this ensemble and subs from some public kernels, like these ones <br>\n<a href=\"https://www.kaggle.com/truonghoang/stacking-ensemble-on-my-submissions\" target=\"_blank\">This one</a>, because it consists of  models like VGG and SeResNext, I think they are quite uncorrelated with my effnets</p>\n<p><a href=\"https://www.kaggle.com/datafan07/analysis-of-melanoma-metadata-and-effnet-ensemble\" target=\"_blank\">This one</a>, because it has good work on metadata analysis</p>\n<p>I did not use kernels kind of \"public blending\" or \"public minimax ensemble\" which were in fact combinations of others, trying to choose from public submissions these ones which correlate as little as possible with my ensemble and with each other. </p>\n<p>At the end days of competition I fell from ~80 position to ~240 and it was nervous, but after th shakeup I, like many others, was thrown quite high (about 200 positions)! <br>\nAnd I really not sure about reasons for it! On the one hand blending of all my models without any public submissions was about on 800 + position on Public LB but something like 140-150 on Private LB, on the other simple blending of public subs gained a silver zone.. I think this story teaches us not to trust Public LB and choose the most stable models for final sub. </p>\n<p>I think it was quite a lottery, many conducted successful experiments, but chose wrong final submissions. What about me - I think I was saved by quite an ensemble and careful use of public submissions</p>\n<p>Special thanks to <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> for his great <a href=\"https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\" target=\"_blank\">kernel</a> and TFRecords datasets covering almost all possible approaches. They helped me alot throughout the competition!</p>",
      "rawMarkdown": "Hello everyone, congrats to all winners and thanks Kaggle and SIIM for this competition \n\n\nThis is my second medal awarded competiton (finally I'm an expert, yeah!!!), and my second medical image analisys competiton. My first one was  [APTOS Blindness Detection](https://www.kaggle.com/c/aptos2019-blindness-detection) last year and these are some similar problems I faced here:\n\n**Imbalanced classes**\n\nA common problem for many medical image analysis is imbalanced data distribution (few images of positive cases). There were 5 classes (or 5 degrees of disease, because the problem could be also interpreted as regression) and  2 classes strongly prevailed. In this competition it was binary classification case and the first one was much larger. The obvious way to fix classes balance was to use external data, and here is the second challenge:\n\n**Use of external data** \n\nIn both competitins we had to deal with quite a large amount of external data, you can use it to balance classes or just use all of this data. In APTOS many people pretrained their models on old external images and finetune in actual data (and so did I). Inept use of external data could decrease your score, old (external) and actual datasets were quite different. In SIIM old and actual data was much less different, so we were more free to use it.\nI used 2018+2019 external data for all training pipeline, validating on 2020 data only. \n\n\n## My approcah \n\nAt the beginning I tried various light models (like EffNet b0-b3, ResNet18 with image size 256x256) using PyTorch and various augmentation techniques both for metadata and for images, experementing with focal loss and label smoothing. But maximum I achived was 0.933 on public LB. But in discussions participants  told about experiments with sizes like  512x512 and quite large models like EffNetB6, these kind of experiments was hard for me because I used PyTorch and GPUs only\n\nAll my way in deep learning I had used PyTorch, but topics related to tensorflow and TPU appeared in discussion and public notebooks more and more, I could not ignore it anymore. Finally I found that using TPU I can train heavier models on larger images! \n\n**Making ensemble**\n\n[2019 1st place solution](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154683) is blending of various versions of EfficientNet and various augmentation techniques and image sizes, so I decided to implement something like that\n\nI used pipeline like [Tripple Stratified KFold](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords), using 2018+2019+2020 data for training and only 2020 data for validation \n\n| Model | Image Size | Initial Weights | Public Score | Private Score |\n| --- | --- | --- | --- | --- |\n| EfficientNetB5 | 512 | ImageNet | 0.9524 | 0.9200 |\n| EfficientNetB5 | 768 | ImageNet | 0.9469 | 0.9185 |\n| EfficientNetB5 | 1024 | ImageNet | 0.9479 | 0.9285 |\n| EfficientNetB6 | 384 | ImageNet | 0.9497 | 0.9277 |\n| EfficientNetB6 | 512 | ImageNet | 0.9488 | 0.9258 |\n| EfficientNetB6 | 768 | ImageNet | 0.9502 | 0.9277 |\n| EfficientNetB6 | 1024 | ImageNet | 0.9478 | 0.9287 |\n| EfficientNetB7 | 512 | ImageNet | 0.9480 | 0.9239 |\n| EfficientNetB7 | 768 | ImageNet | 0.9379 | 0.9083 |\n| EfficientNetB5 | 512 | NoisyStudent | 0.9480 | 0.9194 |\n| EfficientNetB6 | 512 | NoisyStudent| 0.9435 | 0.9242 |\n| EfficientNetB7 | 512 | NoisyStudent| 0.9510 | 0.9218 |\n| EfficientNetB5 | 768 | NoisyStudent | 0.9543 | 0.9295 |\n| EfficientNetB6 | 768 | NoisyStudent| 0.9424 | 0.9218 |\n| ResNet152 | 512 | ImageNet | 0.9184 | 0.8901 |\n| InceptionResNetV2 | 768 | ImageNet | 0.9338 | 0.9161 |\n| Blending | --- | --- | 0.9602 | 0.9406 |\n\nBeing based on  [this idea](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147),  I mixed various models with various training image sizes. \n\n**Public submissions**\n\nWe can't ignore it since have a public submission in a silver medal zone!\nI also made two submmissions with mixing this ensemble and subs from some public kernels, like these ones \n[This one](https://www.kaggle.com/truonghoang/stacking-ensemble-on-my-submissions), because it consists of  models like VGG and SeResNext, I think they are quite uncorrelated with my effnets\n\n[This one](https://www.kaggle.com/datafan07/analysis-of-melanoma-metadata-and-effnet-ensemble), because it has good work on metadata analysis\n\nI did not use kernels kind of \"public blending\" or \"public minimax ensemble\" which were in fact combinations of others, trying to choose from public submissions these ones which correlate as little as possible with my ensemble and with each other. \n\nAt the end days of competition I fell from ~80 position to ~240 and it was nervous, but after th shakeup I, like many others, was thrown quite high (about 200 positions)! \nAnd I really not sure about reasons for it! On the one hand blending of all my models without any public submissions was about on 800 + position on Public LB but something like 140-150 on Private LB, on the other simple blending of public subs gained a silver zone.. I think this story teaches us not to trust Public LB and choose the most stable models for final sub. \n\nI think it was quite a lottery, many conducted successful experiments, but chose wrong final submissions. What about me - I think I was saved by quite an ensemble and careful use of public submissions\n\nSpecial thanks to @cdeotte for his great [kernel](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) and TFRecords datasets covering almost all possible approaches. They helped me alot throughout the competition!",
      "votes": null
    },
    {
      "id": "980137",
      "postDate": "08/21/2020 10:54:35",
      "content": "<p>Thanks for your sharing! <br>\ndid you use tensorflow or pytorch XLA to train on TPU at the end? <br>\nany secret sauce in your loss function ;D ?<br>\nthanks</p>",
      "rawMarkdown": "Thanks for your sharing! \ndid you use tensorflow or pytorch XLA to train on TPU at the end? \nany secret sauce in your loss function ;D ?\nthanks",
      "votes": null
    },
    {
      "id": "980149",
      "postDate": "08/21/2020 11:08:46",
      "content": "<p>I used TF with kaggle TPU<br>\nLoss is BCE with label smoothing 0.05 </p>",
      "rawMarkdown": "I used TF with kaggle TPU\nLoss is BCE with label smoothing 0.05",
      "votes": null
    },
    {
      "id": "980182",
      "postDate": "08/21/2020 11:34:11",
      "content": "<p>Congratulation <a href=\"https://www.kaggle.com/razumovivan\" target=\"_blank\">@razumovivan</a> that was a solid solution, my first medal was also on APTOS 😄</p>",
      "rawMarkdown": "Congratulation @razumovivan that was a solid solution, my first medal was also on APTOS 😄",
      "votes": null
    },
    {
      "id": "980191",
      "postDate": "08/21/2020 11:39:35",
      "content": "<p><a href=\"https://www.kaggle.com/dimitreoliveira\" target=\"_blank\">@dimitreoliveira</a> thanks =)  </p>",
      "rawMarkdown": "dimitreoliveira thanks =)",
      "votes": null
    },
    {
      "id": "980291",
      "postDate": "08/21/2020 13:16:36",
      "content": "<p>nice</p>",
      "rawMarkdown": "nice",
      "votes": null
    },
    {
      "id": "980802",
      "postDate": "08/21/2020 21:56:16",
      "content": "<p>Thanks for sharing your experience. Did u use TF for the blindness detection?</p>",
      "rawMarkdown": "Thanks for sharing your experience. Did u use TF for the blindness detection?",
      "votes": null
    },
    {
      "id": "980860",
      "postDate": "08/21/2020 23:58:52",
      "content": "<p>Great job Ivan. You built a great set of diverse models and then you smartly incorporated public notebooks too. Perfect execution, congrats on solo silver finish.</p>",
      "rawMarkdown": "Great job Ivan. You built a great set of diverse models and then you smartly incorporated public notebooks too. Perfect execution, congrats on solo silver finish.",
      "votes": null
    },
    {
      "id": "980921",
      "postDate": "08/22/2020 02:31:28",
      "content": "<p>No, the reason for using TF (TPU) became available a little bit later :) </p>",
      "rawMarkdown": "No, the reason for using TF (TPU) became available a little bit later :)",
      "votes": null
    },
    {
      "id": "980981",
      "postDate": "08/22/2020 04:40:23",
      "content": "<p>InceptionResNetV2    0.9338 work on 768 👍</p>\n<p>I like your solid approach-based carefulness</p>\n<p>Thank for your sharing and congrats <a href=\"https://www.kaggle.com/razumovivan\" target=\"_blank\">@razumovivan</a> </p>",
      "rawMarkdown": "InceptionResNetV2\t0.9338 work on 768 👍\n\nI like your solid approach-based carefulness\n\nThank for your sharing and congrats @razumovivan",
      "votes": null
    },
    {
      "id": "981111",
      "postDate": "08/22/2020 07:06:05",
      "content": "<p>Thanks! That was actually the main idea - build uncorrelated ensemble</p>",
      "rawMarkdown": "Thanks! That was actually the main idea - build uncorrelated ensemble",
      "votes": null
    },
    {
      "id": "981150",
      "postDate": "08/22/2020 07:57:01",
      "content": "<p><a href=\"https://www.kaggle.com/truonghoang\" target=\"_blank\">@truonghoang</a> thanks for submissions of VGG and SeResNext models :)</p>",
      "rawMarkdown": "truonghoang thanks for submissions of VGG and SeResNext models :)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 980137,
      "author_name": "fiyeroleung",
      "author_url": "",
      "post_date": "08/21/2020 10:54:35",
      "content": "<p>Thanks for your sharing! <br>\ndid you use tensorflow or pytorch XLA to train on TPU at the end? <br>\nany secret sauce in your loss function ;D ?<br>\nthanks</p>",
      "votes": null,
      "replies": [
        {
          "id": 980149,
          "author_name": "razumovivan",
          "author_url": "",
          "post_date": "08/21/2020 11:08:46",
          "content": "<p>I used TF with kaggle TPU<br>\nLoss is BCE with label smoothing 0.05 </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 980182,
      "author_name": "dimitreoliveira",
      "author_url": "",
      "post_date": "08/21/2020 11:34:11",
      "content": "<p>Congratulation <a href=\"https://www.kaggle.com/razumovivan\" target=\"_blank\">@razumovivan</a> that was a solid solution, my first medal was also on APTOS 😄</p>",
      "votes": null,
      "replies": [
        {
          "id": 980191,
          "author_name": "razumovivan",
          "author_url": "",
          "post_date": "08/21/2020 11:39:35",
          "content": "<p><a href=\"https://www.kaggle.com/dimitreoliveira\" target=\"_blank\">@dimitreoliveira</a> thanks =)  </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 980860,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "08/21/2020 23:58:52",
      "content": "<p>Great job Ivan. You built a great set of diverse models and then you smartly incorporated public notebooks too. Perfect execution, congrats on solo silver finish.</p>",
      "votes": null,
      "replies": [
        {
          "id": 981111,
          "author_name": "razumovivan",
          "author_url": "",
          "post_date": "08/22/2020 07:06:05",
          "content": "<p>Thanks! That was actually the main idea - build uncorrelated ensemble</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 980981,
      "author_name": "truonghoang",
      "author_url": "",
      "post_date": "08/22/2020 04:40:23",
      "content": "<p>InceptionResNetV2    0.9338 work on 768 👍</p>\n<p>I like your solid approach-based carefulness</p>\n<p>Thank for your sharing and congrats <a href=\"https://www.kaggle.com/razumovivan\" target=\"_blank\">@razumovivan</a> </p>",
      "votes": null,
      "replies": [
        {
          "id": 981150,
          "author_name": "razumovivan",
          "author_url": "",
          "post_date": "08/22/2020 07:57:01",
          "content": "<p><a href=\"https://www.kaggle.com/truonghoang\" target=\"_blank\">@truonghoang</a> thanks for submissions of VGG and SeResNext models :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 980291,
      "author_name": "pulkitgulati",
      "author_url": "",
      "post_date": "08/21/2020 13:16:36",
      "content": "<p>nice</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 980802,
      "author_name": "mdjawad006",
      "author_url": "",
      "post_date": "08/21/2020 21:56:16",
      "content": "<p>Thanks for sharing your experience. Did u use TF for the blindness detection?</p>",
      "votes": null,
      "replies": [
        {
          "id": 980921,
          "author_name": "razumovivan",
          "author_url": "",
          "post_date": "08/22/2020 02:31:28",
          "content": "<p>No, the reason for using TF (TPU) became available a little bit later :) </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "980117": "Hello everyone, congrats to all winners and thanks Kaggle and SIIM for this competition \n\n\nThis is my second medal awarded competiton (finally I'm an expert, yeah!!!), and my second medical image analisys competiton. My first one was  [APTOS Blindness Detection](https://www.kaggle.com/c/aptos2019-blindness-detection) last year and these are some similar problems I faced here:\n\n**Imbalanced classes**\n\nA common problem for many medical image analysis is imbalanced data distribution (few images of positive cases). There were 5 classes (or 5 degrees of disease, because the problem could be also interpreted as regression) and  2 classes strongly prevailed. In this competition it was binary classification case and the first one was much larger. The obvious way to fix classes balance was to use external data, and here is the second challenge:\n\n**Use of external data** \n\nIn both competitins we had to deal with quite a large amount of external data, you can use it to balance classes or just use all of this data. In APTOS many people pretrained their models on old external images and finetune in actual data (and so did I). Inept use of external data could decrease your score, old (external) and actual datasets were quite different. In SIIM old and actual data was much less different, so we were more free to use it.\nI used 2018+2019 external data for all training pipeline, validating on 2020 data only. \n\n\n## My approcah \n\nAt the beginning I tried various light models (like EffNet b0-b3, ResNet18 with image size 256x256) using PyTorch and various augmentation techniques both for metadata and for images, experementing with focal loss and label smoothing. But maximum I achived was 0.933 on public LB. But in discussions participants  told about experiments with sizes like  512x512 and quite large models like EffNetB6, these kind of experiments was hard for me because I used PyTorch and GPUs only\n\nAll my way in deep learning I had used PyTorch, but topics related to tensorflow and TPU appeared in discussion and public notebooks more and more, I could not ignore it anymore. Finally I found that using TPU I can train heavier models on larger images! \n\n**Making ensemble**\n\n[2019 1st place solution](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154683) is blending of various versions of EfficientNet and various augmentation techniques and image sizes, so I decided to implement something like that\n\nI used pipeline like [Tripple Stratified KFold](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords), using 2018+2019+2020 data for training and only 2020 data for validation \n\n| Model | Image Size | Initial Weights | Public Score | Private Score |\n| --- | --- | --- | --- | --- |\n| EfficientNetB5 | 512 | ImageNet | 0.9524 | 0.9200 |\n| EfficientNetB5 | 768 | ImageNet | 0.9469 | 0.9185 |\n| EfficientNetB5 | 1024 | ImageNet | 0.9479 | 0.9285 |\n| EfficientNetB6 | 384 | ImageNet | 0.9497 | 0.9277 |\n| EfficientNetB6 | 512 | ImageNet | 0.9488 | 0.9258 |\n| EfficientNetB6 | 768 | ImageNet | 0.9502 | 0.9277 |\n| EfficientNetB6 | 1024 | ImageNet | 0.9478 | 0.9287 |\n| EfficientNetB7 | 512 | ImageNet | 0.9480 | 0.9239 |\n| EfficientNetB7 | 768 | ImageNet | 0.9379 | 0.9083 |\n| EfficientNetB5 | 512 | NoisyStudent | 0.9480 | 0.9194 |\n| EfficientNetB6 | 512 | NoisyStudent| 0.9435 | 0.9242 |\n| EfficientNetB7 | 512 | NoisyStudent| 0.9510 | 0.9218 |\n| EfficientNetB5 | 768 | NoisyStudent | 0.9543 | 0.9295 |\n| EfficientNetB6 | 768 | NoisyStudent| 0.9424 | 0.9218 |\n| ResNet152 | 512 | ImageNet | 0.9184 | 0.8901 |\n| InceptionResNetV2 | 768 | ImageNet | 0.9338 | 0.9161 |\n| Blending | --- | --- | 0.9602 | 0.9406 |\n\nBeing based on  [this idea](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147),  I mixed various models with various training image sizes. \n\n**Public submissions**\n\nWe can't ignore it since have a public submission in a silver medal zone!\nI also made two submmissions with mixing this ensemble and subs from some public kernels, like these ones \n[This one](https://www.kaggle.com/truonghoang/stacking-ensemble-on-my-submissions), because it consists of  models like VGG and SeResNext, I think they are quite uncorrelated with my effnets\n\n[This one](https://www.kaggle.com/datafan07/analysis-of-melanoma-metadata-and-effnet-ensemble), because it has good work on metadata analysis\n\nI did not use kernels kind of \"public blending\" or \"public minimax ensemble\" which were in fact combinations of others, trying to choose from public submissions these ones which correlate as little as possible with my ensemble and with each other. \n\nAt the end days of competition I fell from ~80 position to ~240 and it was nervous, but after th shakeup I, like many others, was thrown quite high (about 200 positions)! \nAnd I really not sure about reasons for it! On the one hand blending of all my models without any public submissions was about on 800 + position on Public LB but something like 140-150 on Private LB, on the other simple blending of public subs gained a silver zone.. I think this story teaches us not to trust Public LB and choose the most stable models for final sub. \n\nI think it was quite a lottery, many conducted successful experiments, but chose wrong final submissions. What about me - I think I was saved by quite an ensemble and careful use of public submissions\n\nSpecial thanks to @cdeotte for his great [kernel](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) and TFRecords datasets covering almost all possible approaches. They helped me alot throughout the competition!",
    "980137": "Thanks for your sharing! \ndid you use tensorflow or pytorch XLA to train on TPU at the end? \nany secret sauce in your loss function ;D ?\nthanks",
    "980149": "I used TF with kaggle TPU\nLoss is BCE with label smoothing 0.05",
    "980182": "Congratulation @razumovivan that was a solid solution, my first medal was also on APTOS 😄",
    "980191": "dimitreoliveira thanks =)",
    "980291": "nice",
    "980802": "Thanks for sharing your experience. Did u use TF for the blindness detection?",
    "980860": "Great job Ivan. You built a great set of diverse models and then you smartly incorporated public notebooks too. Perfect execution, congrats on solo silver finish.",
    "980921": "No, the reason for using TF (TPU) became available a little bit later :)",
    "980981": "InceptionResNetV2\t0.9338 work on 768 👍\n\nI like your solid approach-based carefulness\n\nThank for your sharing and congrats @razumovivan",
    "981111": "Thanks! That was actually the main idea - build uncorrelated ensemble",
    "981150": "truonghoang thanks for submissions of VGG and SeResNext models :)"
  },
  "source": "meta"
}