{
  "id": 226663,
  "title": "40th Place: A UNet only solution",
  "url": "/competitions/ranzcr-clip-catheter-line-classification/writeups/datasaurus-40th-place-a-unet-only-solution",
  "author_name": "",
  "post_date": "2021-03-17T11:03:16.243Z",
  "votes": 36,
  "comment_count": 13,
  "views": 0,
  "content": "<p>Congratulations to all the winners and also to the hosts for such a successful competition which will hopefully benefit medical professionals in their work going forward</p>\n<p>My solution uses only UNets in a 2-stage training process to leverage the annotations and help the backbone attend to the salient parts of the X-ray.</p>\n<h1>CV Strategy</h1>\n<p>I applied GroupKFold (k=5) separately in the annotated images and the unannotated images and concatenated the two sets</p>\n<h1>Pre-processing</h1>\n<p>I generated 5 channel masks for each annotated image by grouping the abnormal/borderline/normal classes and used OpenCV to convert the annotations into lines.</p>\n<ul>\n<li>Channel 1: ETT</li>\n<li>Channel 2: NGT</li>\n<li>Channel 3: CVC</li>\n<li>Channel 4: Swan Ganz Catheter</li>\n<li>Channel 5: <a href=\"https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207183\" target=\"_blank\">Lung masks</a> from <a href=\"https://www.kaggle.com/raddar\" target=\"_blank\">@raddar</a> </li>\n</ul>\n<h1>Stage 1: UNet Teacher</h1>\n<p>Stage 1 was a UNet that would predict the 5 channel mask, and also had a classification head from the bottleneck to predict the 11 classes. This was trained only on the annotated images:</p>\n<ul>\n<li>AdamW with 0.1 weight decay</li>\n<li>Cosine Annealing LR with Tmax=60 epochs</li>\n<li>Early stopping with patience=5</li>\n<li>LR = 0.00025 with the batch size around 5-7 depending on the backbone</li>\n<li>Heavy augmentation similar to what was seen in kernels/discussions</li>\n<li>BCEWithLogitsLoss for segmentation</li>\n<li>0.01 * BCEWithLogitsLoss for classification</li>\n</ul>\n<p>I didn’t generally submit these models, but for a ResNet-200D UNet, this was good for 0.953 on public/0.961 private</p>\n<h1>Stage 2: UNet Student</h1>\n<p>I then trained a second UNet (same architecture) but using the full class labels and the masks generated from the teacher UNet, using essentially the same hyperparameters but with lighter augmentation. These pseudo masks were generated on the fly to better benefit from augmentations and passed through a sigmoid activation before passing to BCE. I didn't use a threshold to create binary masks in an attempt to convey teacher uncertainty.</p>\n<table>\n<thead>\n<tr>\n<th>Backbone</th>\n<th>Image Size</th>\n<th>CV</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>ResNet200D</td>\n<td>512</td>\n<td>0.9539</td>\n<td>0.964</td>\n<td>0.968</td>\n</tr>\n<tr>\n<td>EfficientNet-B4</td>\n<td>512</td>\n<td>0.9453</td>\n<td>0.962</td>\n<td>0.965</td>\n</tr>\n<tr>\n<td>ResNet200D</td>\n<td>640</td>\n<td>0.9568</td>\n<td>0.966</td>\n<td>0.970</td>\n</tr>\n<tr>\n<td>EfficientNet-B4</td>\n<td>640</td>\n<td>0.9528</td>\n<td>TBC</td>\n<td>TBC</td>\n</tr>\n<tr>\n<td>SEResNet152d</td>\n<td>640</td>\n<td>0.9556</td>\n<td>0.966</td>\n<td>0.971</td>\n</tr>\n</tbody>\n</table>\n<p>To use ResNet200D &amp; SEResNet152d as UNet encoders, I forked <a href=\"https://www.kaggle.com/pavel92\" target=\"_blank\">@pavel92</a>'s excellent <a href=\"https://github.com/qubvel/segmentation_models.pytorch\" target=\"_blank\">segmentation models</a> and added these encoders from timm (thanks <a href=\"https://www.kaggle.com/rwightman\" target=\"_blank\">@rwightman</a>)</p>\n<h1>Final submission</h1>\n<p>I used 2x TTA (identity &amp; hflip) and a hillclimb ensemble for my final submission of 0.968 public/ 0.972 private</p>\n<h1>Stuff that didn't work</h1>\n<ul>\n<li>Pseudo label hardening/temperature - I found that a temperature (i.e. a number to divide the pseudo label logits by) of 1.0 worked fine. Tried 0.1 but didn't see any significant gain.</li>\n<li>I had the idea of using distillation using the NIH dataset using my stage 2 model as the teacher, and then fine-tune that distilled model using the competition data. However, due to time and leakage, I couldn't quite get this to work. Looking forward to seeing how the top teams leveraged this</li>\n<li>MixUp - not sure why that didn't work in this competition</li>\n<li>Heavy backbones (B7, NFNets etc) - this was probably due to batch size limitations</li>\n</ul>",
  "messages": [
    {
      "id": "1241840",
      "postDate": "03/17/2021 08:29:09",
      "content": "<p>Congratulations to all the winners and also to the hosts for such a successful competition which will hopefully benefit medical professionals in their work going forward</p>\n<p>My solution uses only UNets in a 2-stage training process to leverage the annotations and help the backbone attend to the salient parts of the X-ray.</p>\n<h1>CV Strategy</h1>\n<p>I applied GroupKFold (k=5) separately in the annotated images and the unannotated images and concatenated the two sets</p>\n<h1>Pre-processing</h1>\n<p>I generated 5 channel masks for each annotated image by grouping the abnormal/borderline/normal classes and used OpenCV to convert the annotations into lines.</p>\n<ul>\n<li>Channel 1: ETT</li>\n<li>Channel 2: NGT</li>\n<li>Channel 3: CVC</li>\n<li>Channel 4: Swan Ganz Catheter</li>\n<li>Channel 5: <a href=\"https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207183\" target=\"_blank\">Lung masks</a> from <a href=\"https://www.kaggle.com/raddar\" target=\"_blank\">@raddar</a> </li>\n</ul>\n<h1>Stage 1: UNet Teacher</h1>\n<p>Stage 1 was a UNet that would predict the 5 channel mask, and also had a classification head from the bottleneck to predict the 11 classes. This was trained only on the annotated images:</p>\n<ul>\n<li>AdamW with 0.1 weight decay</li>\n<li>Cosine Annealing LR with Tmax=60 epochs</li>\n<li>Early stopping with patience=5</li>\n<li>LR = 0.00025 with the batch size around 5-7 depending on the backbone</li>\n<li>Heavy augmentation similar to what was seen in kernels/discussions</li>\n<li>BCEWithLogitsLoss for segmentation</li>\n<li>0.01 * BCEWithLogitsLoss for classification</li>\n</ul>\n<p>I didn’t generally submit these models, but for a ResNet-200D UNet, this was good for 0.953 on public/0.961 private</p>\n<h1>Stage 2: UNet Student</h1>\n<p>I then trained a second UNet (same architecture) but using the full class labels and the masks generated from the teacher UNet, using essentially the same hyperparameters but with lighter augmentation. These pseudo masks were generated on the fly to better benefit from augmentations and passed through a sigmoid activation before passing to BCE. I didn't use a threshold to create binary masks in an attempt to convey teacher uncertainty.</p>\n<table>\n<thead>\n<tr>\n<th>Backbone</th>\n<th>Image Size</th>\n<th>CV</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>ResNet200D</td>\n<td>512</td>\n<td>0.9539</td>\n<td>0.964</td>\n<td>0.968</td>\n</tr>\n<tr>\n<td>EfficientNet-B4</td>\n<td>512</td>\n<td>0.9453</td>\n<td>0.962</td>\n<td>0.965</td>\n</tr>\n<tr>\n<td>ResNet200D</td>\n<td>640</td>\n<td>0.9568</td>\n<td>0.966</td>\n<td>0.970</td>\n</tr>\n<tr>\n<td>EfficientNet-B4</td>\n<td>640</td>\n<td>0.9528</td>\n<td>TBC</td>\n<td>TBC</td>\n</tr>\n<tr>\n<td>SEResNet152d</td>\n<td>640</td>\n<td>0.9556</td>\n<td>0.966</td>\n<td>0.971</td>\n</tr>\n</tbody>\n</table>\n<p>To use ResNet200D &amp; SEResNet152d as UNet encoders, I forked <a href=\"https://www.kaggle.com/pavel92\" target=\"_blank\">@pavel92</a>'s excellent <a href=\"https://github.com/qubvel/segmentation_models.pytorch\" target=\"_blank\">segmentation models</a> and added these encoders from timm (thanks <a href=\"https://www.kaggle.com/rwightman\" target=\"_blank\">@rwightman</a>)</p>\n<h1>Final submission</h1>\n<p>I used 2x TTA (identity &amp; hflip) and a hillclimb ensemble for my final submission of 0.968 public/ 0.972 private</p>\n<h1>Stuff that didn't work</h1>\n<ul>\n<li>Pseudo label hardening/temperature - I found that a temperature (i.e. a number to divide the pseudo label logits by) of 1.0 worked fine. Tried 0.1 but didn't see any significant gain.</li>\n<li>I had the idea of using distillation using the NIH dataset using my stage 2 model as the teacher, and then fine-tune that distilled model using the competition data. However, due to time and leakage, I couldn't quite get this to work. Looking forward to seeing how the top teams leveraged this</li>\n<li>MixUp - not sure why that didn't work in this competition</li>\n<li>Heavy backbones (B7, NFNets etc) - this was probably due to batch size limitations</li>\n</ul>",
      "rawMarkdown": "Congratulations to all the winners and also to the hosts for such a successful competition which will hopefully benefit medical professionals in their work going forward\n\nMy solution uses only UNets in a 2-stage training process to leverage the annotations and help the backbone attend to the salient parts of the X-ray.\n\n# CV Strategy\nI applied GroupKFold (k=5) separately in the annotated images and the unannotated images and concatenated the two sets\n\n# Pre-processing\nI generated 5 channel masks for each annotated image by grouping the abnormal/borderline/normal classes and used OpenCV to convert the annotations into lines.\n* Channel 1: ETT\n* Channel 2: NGT\n* Channel 3: CVC\n* Channel 4: Swan Ganz Catheter\n* Channel 5: [Lung masks](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207183) from @raddar \n\n# Stage 1: UNet Teacher\nStage 1 was a UNet that would predict the 5 channel mask, and also had a classification head from the bottleneck to predict the 11 classes. This was trained only on the annotated images:\n* AdamW with 0.1 weight decay\n* Cosine Annealing LR with Tmax=60 epochs\n* Early stopping with patience=5\n* LR = 0.00025 with the batch size around 5-7 depending on the backbone\n* Heavy augmentation similar to what was seen in kernels/discussions\n* BCEWithLogitsLoss for segmentation\n* 0.01 * BCEWithLogitsLoss for classification\n\nI didn’t generally submit these models, but for a ResNet-200D UNet, this was good for 0.953 on public/0.961 private\n\n# Stage 2: UNet Student\nI then trained a second UNet (same architecture) but using the full class labels and the masks generated from the teacher UNet, using essentially the same hyperparameters but with lighter augmentation. These pseudo masks were generated on the fly to better benefit from augmentations and passed through a sigmoid activation before passing to BCE. I didn't use a threshold to create binary masks in an attempt to convey teacher uncertainty.\n\n| Backbone        | Image Size |   CV   | Public | Private |\n| --------------- | ---------- | ------ | ------ | ------- |\n| ResNet200D      | 512        | 0.9539 | 0.964  | 0.968   |\n| EfficientNet-B4 | 512        | 0.9453 | 0.962  | 0.965   |\n| ResNet200D      | 640        | 0.9568 | 0.966  | 0.970   |\n| EfficientNet-B4 | 640        | 0.9528 | TBC    | TBC     |\n| SEResNet152d    | 640        | 0.9556 | 0.966  | 0.971   |\n\nTo use ResNet200D & SEResNet152d as UNet encoders, I forked @pavel92's excellent [segmentation models](https://github.com/qubvel/segmentation_models.pytorch) and added these encoders from timm (thanks @rwightman)\n\n# Final submission\nI used 2x TTA (identity & hflip) and a hillclimb ensemble for my final submission of 0.968 public/ 0.972 private\n\n# Stuff that didn't work\n* Pseudo label hardening/temperature - I found that a temperature (i.e. a number to divide the pseudo label logits by) of 1.0 worked fine. Tried 0.1 but didn't see any significant gain.\n* I had the idea of using distillation using the NIH dataset using my stage 2 model as the teacher, and then fine-tune that distilled model using the competition data. However, due to time and leakage, I couldn't quite get this to work. Looking forward to seeing how the top teams leveraged this\n* MixUp - not sure why that didn't work in this competition\n* Heavy backbones (B7, NFNets etc) - this was probably due to batch size limitations",
      "votes": null
    },
    {
      "id": "1241910",
      "postDate": "03/17/2021 09:17:58",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/anjum48\" target=\"_blank\">@anjum48</a> and great writeup on UNet solution!  Really thought there would have been more on UNet in this competition and like the 5 channel mask idea.  Saw something similar in mouse brain segmentation using a brain atlas annotation mask + channel masks which made me wonder if that would be useful here.  So interesting to read what you have done with this concept!      </p>",
      "rawMarkdown": "Congratulations @anjum48 and great writeup on UNet solution!  Really thought there would have been more on UNet in this competition and like the 5 channel mask idea.  Saw something similar in mouse brain segmentation using a brain atlas annotation mask + channel masks which made me wonder if that would be useful here.  So interesting to read what you have done with this concept!",
      "votes": null
    },
    {
      "id": "1241929",
      "postDate": "03/17/2021 09:31:52",
      "content": "<p>Good score with UNet segmentation 2 stage model <a href=\"https://www.kaggle.com/anjum48\" target=\"_blank\">@anjum48</a> congrats on 40th place</p>",
      "rawMarkdown": "Good score with UNet segmentation 2 stage model @anjum48 congrats on 40th place",
      "votes": null
    },
    {
      "id": "1242017",
      "postDate": "03/17/2021 10:49:34",
      "content": "<p><a href=\"https://www.kaggle.com/anjum48\" target=\"_blank\">@anjum48</a> Congratulations on Silver Finish and great writeup</p>",
      "rawMarkdown": "anjum48 Congratulations on Silver Finish and great writeup",
      "votes": null
    },
    {
      "id": "1242031",
      "postDate": "03/17/2021 10:59:33",
      "content": "<p>Thank you! I'll have to check out the mouse brain segmentation models, sounds very interesting!</p>",
      "rawMarkdown": "Thank you! I'll have to check out the mouse brain segmentation models, sounds very interesting!",
      "votes": null
    },
    {
      "id": "1242032",
      "postDate": "03/17/2021 10:59:47",
      "content": "<p>Thank you! :)</p>",
      "rawMarkdown": "Thank you! :)",
      "votes": null
    },
    {
      "id": "1242033",
      "postDate": "03/17/2021 10:59:56",
      "content": "<p>Thanks!! :)</p>",
      "rawMarkdown": "Thanks!! :)",
      "votes": null
    },
    {
      "id": "1242090",
      "postDate": "03/17/2021 11:51:36",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/anjum48\" target=\"_blank\">@anjum48</a> on silver. Learned a lot from this write up. Thank you for sharing</p>",
      "rawMarkdown": "Congratulations @anjum48 on silver. Learned a lot from this write up. Thank you for sharing",
      "votes": null
    },
    {
      "id": "1242121",
      "postDate": "03/17/2021 12:14:58",
      "content": "<p>Wow, only unet solution. Great performance. I also wonder what would be your performance with the NIH Dataset as well.</p>\n<p>Anyways, great write up! Congratss!!</p>",
      "rawMarkdown": "Wow, only unet solution. Great performance. I also wonder what would be your performance with the NIH Dataset as well.\n\nAnyways, great write up! Congratss!!",
      "votes": null
    },
    {
      "id": "1242567",
      "postDate": "03/17/2021 17:29:50",
      "content": "<p>Great solution. Congrats on Silver finish👍</p>",
      "rawMarkdown": "Great solution. Congrats on Silver finish👍",
      "votes": null
    },
    {
      "id": "1242631",
      "postDate": "03/17/2021 18:11:49",
      "content": "<p>Congratulations. Learn a lot from you about UNet and the segmentation model. Can you share any of your kernels? Thanks in advance. </p>",
      "rawMarkdown": "Congratulations. Learn a lot from you about UNet and the segmentation model. Can you share any of your kernels? Thanks in advance.",
      "votes": null
    },
    {
      "id": "1242813",
      "postDate": "03/17/2021 20:27:01",
      "content": "<p>hi <a href=\"https://www.kaggle.com/anjum48\" target=\"_blank\">@anjum48</a>,</p>\n<p>This is a very lucid writeup. Thank you for providing such an in-depth insight to your solution.<br>\nCongratulations on the Medal!!!</p>",
      "rawMarkdown": "hi @anjum48,\n\nThis is a very lucid writeup. Thank you for providing such an in-depth insight to your solution.\nCongratulations on the Medal!!!",
      "votes": null
    },
    {
      "id": "1243126",
      "postDate": "03/18/2021 03:22:56",
      "content": "<p>Here is the link -<br>\n<a href=\"https://ieeexplore.ieee.org/document/8759226\" target=\"_blank\">https://ieeexplore.ieee.org/document/8759226</a><br>\n One-Shot Learning for Function-Specific Region Segmentation in Mouse Brain</p>",
      "rawMarkdown": "Here is the link -\nhttps://ieeexplore.ieee.org/document/8759226\n One-Shot Learning for Function-Specific Region Segmentation in Mouse Brain",
      "votes": null
    },
    {
      "id": "1505021",
      "postDate": "09/06/2021 20:53:04",
      "content": "<p><a href=\"https://www.kaggle.com/anjum48\" target=\"_blank\">@anjum48</a> thanks for your sharing! Perhaps this is a big ask - but would you share your code by any chance?</p>",
      "rawMarkdown": "anjum48 thanks for your sharing! Perhaps this is a big ask - but would you share your code by any chance?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1241910,
      "author_name": "something4kag",
      "author_url": "",
      "post_date": "03/17/2021 09:17:58",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/anjum48\" target=\"_blank\">@anjum48</a> and great writeup on UNet solution!  Really thought there would have been more on UNet in this competition and like the 5 channel mask idea.  Saw something similar in mouse brain segmentation using a brain atlas annotation mask + channel masks which made me wonder if that would be useful here.  So interesting to read what you have done with this concept!      </p>",
      "votes": null,
      "replies": [
        {
          "id": 1242031,
          "author_name": "anjum48",
          "author_url": "",
          "post_date": "03/17/2021 10:59:33",
          "content": "<p>Thank you! I'll have to check out the mouse brain segmentation models, sounds very interesting!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1243126,
          "author_name": "something4kag",
          "author_url": "",
          "post_date": "03/18/2021 03:22:56",
          "content": "<p>Here is the link -<br>\n<a href=\"https://ieeexplore.ieee.org/document/8759226\" target=\"_blank\">https://ieeexplore.ieee.org/document/8759226</a><br>\n One-Shot Learning for Function-Specific Region Segmentation in Mouse Brain</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1241929,
      "author_name": "duykhanh99",
      "author_url": "",
      "post_date": "03/17/2021 09:31:52",
      "content": "<p>Good score with UNet segmentation 2 stage model <a href=\"https://www.kaggle.com/anjum48\" target=\"_blank\">@anjum48</a> congrats on 40th place</p>",
      "votes": null,
      "replies": [
        {
          "id": 1242033,
          "author_name": "anjum48",
          "author_url": "",
          "post_date": "03/17/2021 10:59:56",
          "content": "<p>Thanks!! :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1242017,
      "author_name": "usharengaraju",
      "author_url": "",
      "post_date": "03/17/2021 10:49:34",
      "content": "<p><a href=\"https://www.kaggle.com/anjum48\" target=\"_blank\">@anjum48</a> Congratulations on Silver Finish and great writeup</p>",
      "votes": null,
      "replies": [
        {
          "id": 1242032,
          "author_name": "anjum48",
          "author_url": "",
          "post_date": "03/17/2021 10:59:47",
          "content": "<p>Thank you! :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1242090,
      "author_name": "thrineshduvvuru",
      "author_url": "",
      "post_date": "03/17/2021 11:51:36",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/anjum48\" target=\"_blank\">@anjum48</a> on silver. Learned a lot from this write up. Thank you for sharing</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1242121,
      "author_name": "snnclsr",
      "author_url": "",
      "post_date": "03/17/2021 12:14:58",
      "content": "<p>Wow, only unet solution. Great performance. I also wonder what would be your performance with the NIH Dataset as well.</p>\n<p>Anyways, great write up! Congratss!!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1242567,
      "author_name": "jagadish13",
      "author_url": "",
      "post_date": "03/17/2021 17:29:50",
      "content": "<p>Great solution. Congrats on Silver finish👍</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1242631,
      "author_name": "durbin164",
      "author_url": "",
      "post_date": "03/17/2021 18:11:49",
      "content": "<p>Congratulations. Learn a lot from you about UNet and the segmentation model. Can you share any of your kernels? Thanks in advance. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1242813,
      "author_name": "supplejade",
      "author_url": "",
      "post_date": "03/17/2021 20:27:01",
      "content": "<p>hi <a href=\"https://www.kaggle.com/anjum48\" target=\"_blank\">@anjum48</a>,</p>\n<p>This is a very lucid writeup. Thank you for providing such an in-depth insight to your solution.<br>\nCongratulations on the Medal!!!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1505021,
      "author_name": "pukkinming",
      "author_url": "",
      "post_date": "09/06/2021 20:53:04",
      "content": "<p><a href=\"https://www.kaggle.com/anjum48\" target=\"_blank\">@anjum48</a> thanks for your sharing! Perhaps this is a big ask - but would you share your code by any chance?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1241840": "Congratulations to all the winners and also to the hosts for such a successful competition which will hopefully benefit medical professionals in their work going forward\n\nMy solution uses only UNets in a 2-stage training process to leverage the annotations and help the backbone attend to the salient parts of the X-ray.\n\n# CV Strategy\nI applied GroupKFold (k=5) separately in the annotated images and the unannotated images and concatenated the two sets\n\n# Pre-processing\nI generated 5 channel masks for each annotated image by grouping the abnormal/borderline/normal classes and used OpenCV to convert the annotations into lines.\n* Channel 1: ETT\n* Channel 2: NGT\n* Channel 3: CVC\n* Channel 4: Swan Ganz Catheter\n* Channel 5: [Lung masks](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207183) from @raddar \n\n# Stage 1: UNet Teacher\nStage 1 was a UNet that would predict the 5 channel mask, and also had a classification head from the bottleneck to predict the 11 classes. This was trained only on the annotated images:\n* AdamW with 0.1 weight decay\n* Cosine Annealing LR with Tmax=60 epochs\n* Early stopping with patience=5\n* LR = 0.00025 with the batch size around 5-7 depending on the backbone\n* Heavy augmentation similar to what was seen in kernels/discussions\n* BCEWithLogitsLoss for segmentation\n* 0.01 * BCEWithLogitsLoss for classification\n\nI didn’t generally submit these models, but for a ResNet-200D UNet, this was good for 0.953 on public/0.961 private\n\n# Stage 2: UNet Student\nI then trained a second UNet (same architecture) but using the full class labels and the masks generated from the teacher UNet, using essentially the same hyperparameters but with lighter augmentation. These pseudo masks were generated on the fly to better benefit from augmentations and passed through a sigmoid activation before passing to BCE. I didn't use a threshold to create binary masks in an attempt to convey teacher uncertainty.\n\n| Backbone        | Image Size |   CV   | Public | Private |\n| --------------- | ---------- | ------ | ------ | ------- |\n| ResNet200D      | 512        | 0.9539 | 0.964  | 0.968   |\n| EfficientNet-B4 | 512        | 0.9453 | 0.962  | 0.965   |\n| ResNet200D      | 640        | 0.9568 | 0.966  | 0.970   |\n| EfficientNet-B4 | 640        | 0.9528 | TBC    | TBC     |\n| SEResNet152d    | 640        | 0.9556 | 0.966  | 0.971   |\n\nTo use ResNet200D & SEResNet152d as UNet encoders, I forked @pavel92's excellent [segmentation models](https://github.com/qubvel/segmentation_models.pytorch) and added these encoders from timm (thanks @rwightman)\n\n# Final submission\nI used 2x TTA (identity & hflip) and a hillclimb ensemble for my final submission of 0.968 public/ 0.972 private\n\n# Stuff that didn't work\n* Pseudo label hardening/temperature - I found that a temperature (i.e. a number to divide the pseudo label logits by) of 1.0 worked fine. Tried 0.1 but didn't see any significant gain.\n* I had the idea of using distillation using the NIH dataset using my stage 2 model as the teacher, and then fine-tune that distilled model using the competition data. However, due to time and leakage, I couldn't quite get this to work. Looking forward to seeing how the top teams leveraged this\n* MixUp - not sure why that didn't work in this competition\n* Heavy backbones (B7, NFNets etc) - this was probably due to batch size limitations",
    "1241910": "Congratulations @anjum48 and great writeup on UNet solution!  Really thought there would have been more on UNet in this competition and like the 5 channel mask idea.  Saw something similar in mouse brain segmentation using a brain atlas annotation mask + channel masks which made me wonder if that would be useful here.  So interesting to read what you have done with this concept!",
    "1241929": "Good score with UNet segmentation 2 stage model @anjum48 congrats on 40th place",
    "1242017": "anjum48 Congratulations on Silver Finish and great writeup",
    "1242031": "Thank you! I'll have to check out the mouse brain segmentation models, sounds very interesting!",
    "1242032": "Thank you! :)",
    "1242033": "Thanks!! :)",
    "1242090": "Congratulations @anjum48 on silver. Learned a lot from this write up. Thank you for sharing",
    "1242121": "Wow, only unet solution. Great performance. I also wonder what would be your performance with the NIH Dataset as well.\n\nAnyways, great write up! Congratss!!",
    "1242567": "Great solution. Congrats on Silver finish👍",
    "1242631": "Congratulations. Learn a lot from you about UNet and the segmentation model. Can you share any of your kernels? Thanks in advance.",
    "1242813": "hi @anjum48,\n\nThis is a very lucid writeup. Thank you for providing such an in-depth insight to your solution.\nCongratulations on the Medal!!!",
    "1243126": "Here is the link -\nhttps://ieeexplore.ieee.org/document/8759226\n One-Shot Learning for Function-Specific Region Segmentation in Mouse Brain",
    "1505021": "anjum48 thanks for your sharing! Perhaps this is a big ask - but would you share your code by any chance?"
  },
  "source": "meta"
}