{
  "id": 77322,
  "title": "15th place solution with Deep Supervision",
  "url": "/competitions/human-protein-atlas-image-classification/writeups/a-team-15th-place-solution-with-deep-supervision",
  "author_name": "",
  "post_date": "2019-01-11T17:23:09.087Z",
  "votes": 21,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Here is a writeup on my team's approach to this challenge</p>\n\n<p>Data: Initially we split the Kaggle dataset into 6 folds. After the external data was discovered, we incorporated it directly into the train part in every fold (so we had roughly 100k train images and 5k valid. images). We trained and predicted using resized 512x512 RGB/ RGBY images.</p>\n\n<p>Models:  SE_ResNext50/101 and Inceptionv3/v4\nWe realized that multi-scale predictions are really crucial since the trace of a protein may appear in a very small region of the image. We solved this problem by applying the deep supervision trick. The chosen networks consist of multiple blocks of gradually decreasing resolutions; on top of these blocks, a small auxiliary classification network (Global Pooling + BN + Dense) was built. The final loss is an equal sum of these auxiliary losses. At inference time, we averaged all the auxiliary blocks' predictions and the final Dense layer predictions. For Inception networks, we picked 6 mixed blocks.  For ResNext networks, block 3, 4, 5 and a feature pyramid network outputs were used.</p>\n\n<p>Loss: Pure binary cross entropy, no upsampling/downsampling rare classes. Focal loss didn't work well for us.</p>\n\n<p>Threshold: 0.5 for several popular classes, rest 0.2</p>\n\n<p>Final submission is a weighted average of 4 above models. Stacking gave a strong 5% boost on  local F1 score yet sucked both in public and private LB. My teammate also did extra postprocessing on submission file but we didn't go with it at the end since it gave slightly lower public LB score. Sadly, this post-processed submission would have given us another gold medal XD</p>\n\n<p>Finally, huge congrats to all the winning teams and gold medalists XD</p>",
  "messages": [
    {
      "id": "454366",
      "postDate": "01/11/2019 14:27:17",
      "content": "<p>Here is a writeup on my team's approach to this challenge</p>\n\n<p>Data: Initially we split the Kaggle dataset into 6 folds. After the external data was discovered, we incorporated it directly into the train part in every fold (so we had roughly 100k train images and 5k valid. images). We trained and predicted using resized 512x512 RGB/ RGBY images.</p>\n\n<p>Models:  SE_ResNext50/101 and Inceptionv3/v4\nWe realized that multi-scale predictions are really crucial since the trace of a protein may appear in a very small region of the image. We solved this problem by applying the deep supervision trick. The chosen networks consist of multiple blocks of gradually decreasing resolutions; on top of these blocks, a small auxiliary classification network (Global Pooling + BN + Dense) was built. The final loss is an equal sum of these auxiliary losses. At inference time, we averaged all the auxiliary blocks' predictions and the final Dense layer predictions. For Inception networks, we picked 6 mixed blocks.  For ResNext networks, block 3, 4, 5 and a feature pyramid network outputs were used.</p>\n\n<p>Loss: Pure binary cross entropy, no upsampling/downsampling rare classes. Focal loss didn't work well for us.</p>\n\n<p>Threshold: 0.5 for several popular classes, rest 0.2</p>\n\n<p>Final submission is a weighted average of 4 above models. Stacking gave a strong 5% boost on  local F1 score yet sucked both in public and private LB. My teammate also did extra postprocessing on submission file but we didn't go with it at the end since it gave slightly lower public LB score. Sadly, this post-processed submission would have given us another gold medal XD</p>\n\n<p>Finally, huge congrats to all the winning teams and gold medalists XD</p>",
      "rawMarkdown": "Here is a writeup on my team's approach to this challenge\n\nData: Initially we split the Kaggle dataset into 6 folds. After the external data was discovered, we incorporated it directly into the train part in every fold (so we had roughly 100k train images and 5k valid. images). We trained and predicted using resized 512x512 RGB/ RGBY images.\n\nModels:  SE_ResNext50/101 and Inceptionv3/v4\nWe realized that multi-scale predictions are really crucial since the trace of a protein may appear in a very small region of the image. We solved this problem by applying the deep supervision trick. The chosen networks consist of multiple blocks of gradually decreasing resolutions; on top of these blocks, a small auxiliary classification network (Global Pooling + BN + Dense) was built. The final loss is an equal sum of these auxiliary losses. At inference time, we averaged all the auxiliary blocks' predictions and the final Dense layer predictions. For Inception networks, we picked 6 mixed blocks.  For ResNext networks, block 3, 4, 5 and a feature pyramid network outputs were used.\n\nLoss: Pure binary cross entropy, no upsampling/downsampling rare classes. Focal loss didn't work well for us.\n\nThreshold: 0.5 for several popular classes, rest 0.2\n\nFinal submission is a weighted average of 4 above models. Stacking gave a strong 5% boost on  local F1 score yet sucked both in public and private LB. My teammate also did extra postprocessing on submission file but we didn't go with it at the end since it gave slightly lower public LB score. Sadly, this post-processed submission would have given us another gold medal XD\n\nFinally, huge congrats to all the winning teams and gold medalists XD",
      "votes": null
    },
    {
      "id": "454377",
      "postDate": "01/11/2019 14:53:57",
      "content": "<p>Congratulations, good job!</p>",
      "rawMarkdown": "Congratulations, good job!",
      "votes": null
    },
    {
      "id": "454405",
      "postDate": "01/11/2019 15:39:17",
      "content": "<p>Congratulations to you and your teammates! I tried to do something similar with resnet50. I added an \"attention\" layer comprising of Global Pooling, and a convolution downscaling and upscaling at the end of the bottleneck. But I could not get it work as I wanted.  Congratulations on getting your model to work! It is this kind of ingenuity (also want to mention @Dieter's GAPNet implementation) that makes the competition more interesting. </p>",
      "rawMarkdown": "Congratulations to you and your teammates! I tried to do something similar with resnet50. I added an \"attention\" layer comprising of Global Pooling, and a convolution downscaling and upscaling at the end of the bottleneck. But I could not get it work as I wanted.  Congratulations on getting your model to work! It is this kind of ingenuity (also want to mention @Dieter's GAPNet implementation) that makes the competition more interesting.",
      "votes": null
    },
    {
      "id": "458817",
      "postDate": "01/20/2019 14:37:45",
      "content": "<p>Congratulations!</p>",
      "rawMarkdown": "Congratulations!",
      "votes": null
    },
    {
      "id": "460258",
      "postDate": "01/23/2019 10:06:47",
      "content": "<p>Congratulations and Thanks for sharing.</p>",
      "rawMarkdown": "Congratulations and Thanks for sharing.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 454377,
      "author_name": "arnaurm",
      "author_url": "",
      "post_date": "01/11/2019 14:53:57",
      "content": "<p>Congratulations, good job!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 454405,
      "author_name": "watts2",
      "author_url": "",
      "post_date": "01/11/2019 15:39:17",
      "content": "<p>Congratulations to you and your teammates! I tried to do something similar with resnet50. I added an \"attention\" layer comprising of Global Pooling, and a convolution downscaling and upscaling at the end of the bottleneck. But I could not get it work as I wanted.  Congratulations on getting your model to work! It is this kind of ingenuity (also want to mention @Dieter's GAPNet implementation) that makes the competition more interesting. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 458817,
      "author_name": "meatloaf1",
      "author_url": "",
      "post_date": "01/20/2019 14:37:45",
      "content": "<p>Congratulations!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 460258,
      "author_name": "",
      "author_url": "",
      "post_date": "01/23/2019 10:06:47",
      "content": "<p>Congratulations and Thanks for sharing.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "454366": "Here is a writeup on my team's approach to this challenge\n\nData: Initially we split the Kaggle dataset into 6 folds. After the external data was discovered, we incorporated it directly into the train part in every fold (so we had roughly 100k train images and 5k valid. images). We trained and predicted using resized 512x512 RGB/ RGBY images.\n\nModels:  SE_ResNext50/101 and Inceptionv3/v4\nWe realized that multi-scale predictions are really crucial since the trace of a protein may appear in a very small region of the image. We solved this problem by applying the deep supervision trick. The chosen networks consist of multiple blocks of gradually decreasing resolutions; on top of these blocks, a small auxiliary classification network (Global Pooling + BN + Dense) was built. The final loss is an equal sum of these auxiliary losses. At inference time, we averaged all the auxiliary blocks' predictions and the final Dense layer predictions. For Inception networks, we picked 6 mixed blocks.  For ResNext networks, block 3, 4, 5 and a feature pyramid network outputs were used.\n\nLoss: Pure binary cross entropy, no upsampling/downsampling rare classes. Focal loss didn't work well for us.\n\nThreshold: 0.5 for several popular classes, rest 0.2\n\nFinal submission is a weighted average of 4 above models. Stacking gave a strong 5% boost on  local F1 score yet sucked both in public and private LB. My teammate also did extra postprocessing on submission file but we didn't go with it at the end since it gave slightly lower public LB score. Sadly, this post-processed submission would have given us another gold medal XD\n\nFinally, huge congrats to all the winning teams and gold medalists XD",
    "454377": "Congratulations, good job!",
    "454405": "Congratulations to you and your teammates! I tried to do something similar with resnet50. I added an \"attention\" layer comprising of Global Pooling, and a convolution downscaling and upscaling at the end of the bottleneck. But I could not get it work as I wanted.  Congratulations on getting your model to work! It is this kind of ingenuity (also want to mention @Dieter's GAPNet implementation) that makes the competition more interesting.",
    "458817": "Congratulations!",
    "460258": "Congratulations and Thanks for sharing."
  },
  "source": "meta"
}