{
  "id": 227407,
  "title": "Dual-Head Model with 4-stage Training, 2nd Place Solution",
  "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/227407",
  "author_name": "sheep",
  "post_date": "2021-03-20T11:55:15.728000",
  "votes": 34,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Congrats to all the winners, well done! I am grateful to my teammates <a href=\"https://www.kaggle.com/nvnnghia\" target=\"_blank\">@nvnnghia</a> <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> and <a href=\"https://www.kaggle.com/underwearfitting\" target=\"_blank\">@underwearfitting</a> you all did a very good job and finally we finished in 2nd place.<br>\nI would like to thank <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> for your Unet code at hubmap competition and the wonderful idea of combination of the label and the annotations. Also thanks to <a href=\"https://www.kaggle.com/yasufuminakama\" target=\"_blank\">@yasufuminakama</a> 's 3-stage training implementation.<br>\nAnd congratulations to VinBigData Group, their employees won 3 solo golds in this competition! <a href=\"https://www.kaggle.com/moewie94\" target=\"_blank\">@moewie94</a> <a href=\"https://www.kaggle.com/andy2709\" target=\"_blank\">@andy2709</a> <a href=\"https://www.kaggle.com/nguyenbadung\" target=\"_blank\">@nguyenbadung</a>. Looking forward to read your solutions</p>\n<h2>Data</h2>\n<ul>\n<li>For training, we only used RANZCR's training set and the official annotation data. We have tried other annotations like lung mask and key point, but none of them worked well at our setup.</li>\n<li>We tried pseudo labeling with ChestX dataset we were affected by severe leaking so we gave up pseudo labeling.</li>\n<li>We used lighter augmentation based on sin's public notebook, cutmix didn’t improve CV and LB  but introduced diversity to our final ensemble.</li>\n<li>We failed to use image size larger than 1024, even the performance of 1024 image is somehow bad. Our best model is a <code>Efficientnetb7-unet</code> model trained at 768 size. We will present it later. </li>\n<li>Mask generation: we generated masks via plot the line between the annotation points. After training, we generated a full training set pseudo mask by averaging 5 fold's output. Since we use pseudo masks (trainset w/o annotations) as a supervisor, we ignore the potential leakage here. You can refer to this notebook.</li>\n</ul>\n<h2>Model</h2>\n<p>Our model is a dual-head Unet model. with both classification and segmentation output. Unlike other teams, we only used masks as a supervisor, for two reasons: first, we wanted the model to learn the correct pattern with segmentation loss; secondly,  our data is grouped and collected from different hospitals or patients, which may lead to different quality imaging and mask output. Directly inputting these masks to classification may reduce the generalization performance of our model.<br>\nOur final model used <code>Resnet200d</code>, <code>efficientnet-b5</code> and <code>efficientnet-b7</code> as backbones, unet-decoder part is reduced in order to train with decent VRAM usage.<br>\nWe also tried adding consistency loss to our model but this has slowed the training procedure a lot.</p>\n<p><a href=\"https://ibb.co/F0HYpCT\"><img src=\"https://i.ibb.co/rcdp9Nr/image1.png\" alt=\"image1\"></a></p>\n<h2>Training</h2>\n<p>We borrowed stage1 and stage2 from the public notebook, only modifying a few training parameters for stage2.</p>\n<h3>Mask Generation</h3>\n<ul>\n<li>stage3: trained <code>resnet200d-unet</code> at 512 and train_annotation generated mask, if an image input without annotations, we calculate the classification loss only. Predicted full train set pseudo mask v1</li>\n<li>stage4: trained <code>resnet200d-unet</code> at 768 with v1 mask and scaled up to 768 size. Generate pseudo mask version v2.<br>\n<img src=\"https://i.ibb.co/8MLZg37/image2.png\" alt=\"image2\"></li>\n</ul>\n<h3>Model training</h3>\n<ul>\n<li>stage3: trained with <code>efficientnet-b5-unet</code>, <code>efficient-b7-unet</code>, <code>resnet200d-unet</code> at 768 resolution</li>\n<li>stage4[optional]: scaled up training for <code>resnet200d-unet</code> at 1024 resolution<br>\n<a href=\"https://ibb.co/Z2PDZbs\"><img src=\"https://i.ibb.co/7k5m90c/stage3-2.png\" alt=\"stage3-2\"></a></li>\n</ul>\n<p>Our best performing model is <code>efficientnet-b7-unet</code> which achieved <code>0.9695</code> CV w/ hflip tta at image size 768, <code>resnet-200d-unet</code> achieve <code>0.9685</code> cv and <code>0.972</code> Public LB w/ hflp tta. A single <code>resnet-200d-unet</code> can achieve <code>0.974</code> on Private LB and win a gold medal. After ensemble these models, three model ensembles achieve <code>0.972</code> CV, <code>0.97383</code> Public LB and <code>0.97599</code> Private LB.<br>\nSix models ensemble achieved <code>0.973</code> CV, <code>0.97412</code> Public LB and <code>0.97641</code> Private LB.</p>\n<h2>Inference</h2>\n<p>Our model design enables us to inference with backbone and classification head. We added one 1024 <code>resnet200d-unet</code> model, one 768 image size <code>efficientnet-b5-unet</code> model and an <code>efficientnet-b7-unet</code> model to the inference notebook and it finished in about 8 hours and 50 minutes. We also found an image retrieval trick to reduce the inference time.</p>\n<p>Roughly we do the things below:</p>\n<ul>\n<li>Pseudo label chestx dataset using existing models</li>\n<li>Train a metric learning model with target of patient ID</li>\n<li>Use a KNN model to retrieve the top-1 similar image from chestx dataset and submit the pseudo label.<br>\n<img src=\"https://i.ibb.co/r6WCnQw/image4.png\" alt=\"image4\"></li>\n</ul>\n<p>Using this method, we improved private LB to 0.97641</p>\n<h2>Code</h2>\n<p>We are organizing our code and will release our code on github later.</p>",
  "messages": [
    {
      "id": 1245969,
      "postDate": "2021-03-20T11:55:15.727Z",
      "content": "<p>Congrats to all the winners, well done! I am grateful to my teammates <a href=\"https://www.kaggle.com/nvnnghia\" target=\"_blank\">@nvnnghia</a> <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> and <a href=\"https://www.kaggle.com/underwearfitting\" target=\"_blank\">@underwearfitting</a> you all did a very good job and finally we finished in 2nd place.<br>\nI would like to thank <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> for your Unet code at hubmap competition and the wonderful idea of combination of the label and the annotations. Also thanks to <a href=\"https://www.kaggle.com/yasufuminakama\" target=\"_blank\">@yasufuminakama</a> 's 3-stage training implementation.<br>\nAnd congratulations to VinBigData Group, their employees won 3 solo golds in this competition! <a href=\"https://www.kaggle.com/moewie94\" target=\"_blank\">@moewie94</a> <a href=\"https://www.kaggle.com/andy2709\" target=\"_blank\">@andy2709</a> <a href=\"https://www.kaggle.com/nguyenbadung\" target=\"_blank\">@nguyenbadung</a>. Looking forward to read your solutions</p>\n<h2>Data</h2>\n<ul>\n<li>For training, we only used RANZCR's training set and the official annotation data. We have tried other annotations like lung mask and key point, but none of them worked well at our setup.</li>\n<li>We tried pseudo labeling with ChestX dataset we were affected by severe leaking so we gave up pseudo labeling.</li>\n<li>We used lighter augmentation based on sin's public notebook, cutmix didn’t improve CV and LB  but introduced diversity to our final ensemble.</li>\n<li>We failed to use image size larger than 1024, even the performance of 1024 image is somehow bad. Our best model is a <code>Efficientnetb7-unet</code> model trained at 768 size. We will present it later. </li>\n<li>Mask generation: we generated masks via plot the line between the annotation points. After training, we generated a full training set pseudo mask by averaging 5 fold's output. Since we use pseudo masks (trainset w/o annotations) as a supervisor, we ignore the potential leakage here. You can refer to this notebook.</li>\n</ul>\n<h2>Model</h2>\n<p>Our model is a dual-head Unet model. with both classification and segmentation output. Unlike other teams, we only used masks as a supervisor, for two reasons: first, we wanted the model to learn the correct pattern with segmentation loss; secondly,  our data is grouped and collected from different hospitals or patients, which may lead to different quality imaging and mask output. Directly inputting these masks to classification may reduce the generalization performance of our model.<br>\nOur final model used <code>Resnet200d</code>, <code>efficientnet-b5</code> and <code>efficientnet-b7</code> as backbones, unet-decoder part is reduced in order to train with decent VRAM usage.<br>\nWe also tried adding consistency loss to our model but this has slowed the training procedure a lot.</p>\n<p><a href=\"https://ibb.co/F0HYpCT\"><img src=\"https://i.ibb.co/rcdp9Nr/image1.png\" alt=\"image1\"></a></p>\n<h2>Training</h2>\n<p>We borrowed stage1 and stage2 from the public notebook, only modifying a few training parameters for stage2.</p>\n<h3>Mask Generation</h3>\n<ul>\n<li>stage3: trained <code>resnet200d-unet</code> at 512 and train_annotation generated mask, if an image input without annotations, we calculate the classification loss only. Predicted full train set pseudo mask v1</li>\n<li>stage4: trained <code>resnet200d-unet</code> at 768 with v1 mask and scaled up to 768 size. Generate pseudo mask version v2.<br>\n<img src=\"https://i.ibb.co/8MLZg37/image2.png\" alt=\"image2\"></li>\n</ul>\n<h3>Model training</h3>\n<ul>\n<li>stage3: trained with <code>efficientnet-b5-unet</code>, <code>efficient-b7-unet</code>, <code>resnet200d-unet</code> at 768 resolution</li>\n<li>stage4[optional]: scaled up training for <code>resnet200d-unet</code> at 1024 resolution<br>\n<a href=\"https://ibb.co/Z2PDZbs\"><img src=\"https://i.ibb.co/7k5m90c/stage3-2.png\" alt=\"stage3-2\"></a></li>\n</ul>\n<p>Our best performing model is <code>efficientnet-b7-unet</code> which achieved <code>0.9695</code> CV w/ hflip tta at image size 768, <code>resnet-200d-unet</code> achieve <code>0.9685</code> cv and <code>0.972</code> Public LB w/ hflp tta. A single <code>resnet-200d-unet</code> can achieve <code>0.974</code> on Private LB and win a gold medal. After ensemble these models, three model ensembles achieve <code>0.972</code> CV, <code>0.97383</code> Public LB and <code>0.97599</code> Private LB.<br>\nSix models ensemble achieved <code>0.973</code> CV, <code>0.97412</code> Public LB and <code>0.97641</code> Private LB.</p>\n<h2>Inference</h2>\n<p>Our model design enables us to inference with backbone and classification head. We added one 1024 <code>resnet200d-unet</code> model, one 768 image size <code>efficientnet-b5-unet</code> model and an <code>efficientnet-b7-unet</code> model to the inference notebook and it finished in about 8 hours and 50 minutes. We also found an image retrieval trick to reduce the inference time.</p>\n<p>Roughly we do the things below:</p>\n<ul>\n<li>Pseudo label chestx dataset using existing models</li>\n<li>Train a metric learning model with target of patient ID</li>\n<li>Use a KNN model to retrieve the top-1 similar image from chestx dataset and submit the pseudo label.<br>\n<img src=\"https://i.ibb.co/r6WCnQw/image4.png\" alt=\"image4\"></li>\n</ul>\n<p>Using this method, we improved private LB to 0.97641</p>\n<h2>Code</h2>\n<p>We are organizing our code and will release our code on github later.</p>",
      "rawMarkdown": "Congrats to all the winners, well done! I am grateful to my teammates @nvnnghia @cdeotte and @underwearfitting you all did a very good job and finally we finished in 2nd place.\nI would like to thank @hengck23 for your Unet code at hubmap competition and the wonderful idea of combination of the label and the annotations. Also thanks to @yasufuminakama 's 3-stage training implementation.\nAnd congratulations to VinBigData Group, their employees won 3 solo golds in this competition! @moewie94 @andy2709 @nguyenbadung. Looking forward to read your solutions\n\n## Data\n* For training, we only used RANZCR's training set and the official annotation data. We have tried other annotations like lung mask and key point, but none of them worked well at our setup.\n* We tried pseudo labeling with ChestX dataset we were affected by severe leaking so we gave up pseudo labeling.\n* We used lighter augmentation based on sin's public notebook, cutmix didn’t improve CV and LB  but introduced diversity to our final ensemble.\n* We failed to use image size larger than 1024, even the performance of 1024 image is somehow bad. Our best model is a `Efficientnetb7-unet` model trained at 768 size. We will present it later. \n* Mask generation: we generated masks via plot the line between the annotation points. After training, we generated a full training set pseudo mask by averaging 5 fold's output. Since we use pseudo masks (trainset w/o annotations) as a supervisor, we ignore the potential leakage here. You can refer to this notebook.\n\n## Model\nOur model is a dual-head Unet model. with both classification and segmentation output. Unlike other teams, we only used masks as a supervisor, for two reasons: first, we wanted the model to learn the correct pattern with segmentation loss; secondly,  our data is grouped and collected from different hospitals or patients, which may lead to different quality imaging and mask output. Directly inputting these masks to classification may reduce the generalization performance of our model.\nOur final model used `Resnet200d`, `efficientnet-b5` and `efficientnet-b7` as backbones, unet-decoder part is reduced in order to train with decent VRAM usage.\nWe also tried adding consistency loss to our model but this has slowed the training procedure a lot.\n\n<a href=\"https://ibb.co/F0HYpCT\"><img src=\"https://i.ibb.co/rcdp9Nr/image1.png\" alt=\"image1\" border=\"0\"></a>\n## Training\nWe borrowed stage1 and stage2 from the public notebook, only modifying a few training parameters for stage2.\n### Mask Generation\n* stage3: trained `resnet200d-unet` at 512 and train_annotation generated mask, if an image input without annotations, we calculate the classification loss only. Predicted full train set pseudo mask v1\n* stage4: trained `resnet200d-unet` at 768 with v1 mask and scaled up to 768 size. Generate pseudo mask version v2.\n<img src=\"https://i.ibb.co/8MLZg37/image2.png\" alt=\"image2\" border=\"0\">\n### Model training\n* stage3: trained with `efficientnet-b5-unet`, `efficient-b7-unet`, `resnet200d-unet` at 768 resolution\n* stage4[optional]: scaled up training for `resnet200d-unet` at 1024 resolution\n<a href=\"https://ibb.co/Z2PDZbs\"><img src=\"https://i.ibb.co/7k5m90c/stage3-2.png\" alt=\"stage3-2\" border=\"0\"></a>\n\nOur best performing model is `efficientnet-b7-unet` which achieved `0.9695` CV w/ hflip tta at image size 768, `resnet-200d-unet` achieve `0.9685` cv and `0.972` Public LB w/ hflp tta. A single `resnet-200d-unet` can achieve `0.974` on Private LB and win a gold medal. After ensemble these models, three model ensembles achieve `0.972` CV, `0.97383` Public LB and `0.97599` Private LB.\nSix models ensemble achieved `0.973` CV, `0.97412` Public LB and `0.97641` Private LB.\n\n## Inference\nOur model design enables us to inference with backbone and classification head. We added one 1024 `resnet200d-unet` model, one 768 image size `efficientnet-b5-unet` model and an `efficientnet-b7-unet` model to the inference notebook and it finished in about 8 hours and 50 minutes. We also found an image retrieval trick to reduce the inference time.\n\nRoughly we do the things below:\n* Pseudo label chestx dataset using existing models\n* Train a metric learning model with target of patient ID\n* Use a KNN model to retrieve the top-1 similar image from chestx dataset and submit the pseudo label.\n<img src=\"https://i.ibb.co/r6WCnQw/image4.png\" alt=\"image4\" border=\"0\">\n\nUsing this method, we improved private LB to 0.97641\n\n## Code\nWe are organizing our code and will release our code on github later.\n\n\n\n",
      "votes": 34
    },
    {
      "id": 1246831,
      "postDate": "2021-03-21T06:29:31.237Z",
      "content": "<p>Great solution! Thanks for sharing details solution <a href=\"https://www.kaggle.com/steamedsheep\" target=\"_blank\">@steamedsheep</a>. Congratulations again <a href=\"https://www.kaggle.com/underwearfitting\" target=\"_blank\">@underwearfitting</a> <a href=\"https://www.kaggle.com/nvnnghia\" target=\"_blank\">@nvnnghia</a> <a href=\"https://www.kaggle.com/steamedsheep\" target=\"_blank\">@steamedsheep</a> and <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> </p>",
      "rawMarkdown": "Great solution! Thanks for sharing details solution @steamedsheep. Congratulations again @underwearfitting @nvnnghia @steamedsheep and @cdeotte ",
      "votes": 7
    },
    {
      "id": 1247181,
      "postDate": "2021-03-21T13:57:48.903Z",
      "content": "<p>Congratulations on 2nd place! Thank you for your solutions.</p>",
      "rawMarkdown": "Congratulations on 2nd place! Thank you for your solutions.",
      "votes": 2
    },
    {
      "id": 1246345,
      "postDate": "2021-03-20T16:55:41.420Z",
      "content": "<p>Congratulations on #2 win!</p>",
      "rawMarkdown": "Congratulations on #2 win!",
      "votes": 2
    },
    {
      "id": 1246928,
      "postDate": "2021-03-21T09:01:47.803Z",
      "content": "<p><a href=\"https://www.kaggle.com/steamedsheep\" target=\"_blank\">@steamedsheep</a> , Great solution! Thanks for sharing. I added it to my collection in <a href=\"https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques\" target=\"_blank\">\"Data Science with DL &amp; NLP: Advanced Techniques\"</a>, section \"Prize Competition Winners: notebooks (kernels) and posts with Magic\".</p>",
      "rawMarkdown": "@steamedsheep , Great solution! Thanks for sharing. I added it to my collection in [\"Data Science with DL & NLP: Advanced Techniques\"](https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques), section \"Prize Competition Winners: notebooks (kernels) and posts with Magic\"."
    },
    {
      "id": 1506184,
      "postDate": "2021-09-08T01:45:00.047Z",
      "content": "<p><a href=\"https://www.kaggle.com/steamedsheep\" target=\"_blank\">@steamedsheep</a>, this is great write-up. Thank you and congrats for the wonderful win! Look forward to your github code!</p>",
      "rawMarkdown": "@steamedsheep, this is great write-up. Thank you and congrats for the wonderful win! Look forward to your github code!"
    },
    {
      "id": 1394779,
      "postDate": "2021-07-20T15:40:04.803Z",
      "content": "<p><a href=\"https://www.kaggle.com/steamedsheep\" target=\"_blank\">@steamedsheep</a> Great writeup! Did you share the code somewhere?</p>",
      "rawMarkdown": "@steamedsheep Great writeup! Did you share the code somewhere?"
    },
    {
      "id": 1246816,
      "postDate": "2021-03-21T06:17:38.643Z",
      "rawMarkdown": "",
      "votes": 3,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1246831,
      "author_name": "KhanhVD",
      "author_url": "",
      "post_date": "2021-03-21T06:29:31.237000",
      "content": "<p>Great solution! Thanks for sharing details solution <a href=\"https://www.kaggle.com/steamedsheep\" target=\"_blank\">@steamedsheep</a>. Congratulations again <a href=\"https://www.kaggle.com/underwearfitting\" target=\"_blank\">@underwearfitting</a> <a href=\"https://www.kaggle.com/nvnnghia\" target=\"_blank\">@nvnnghia</a> <a href=\"https://www.kaggle.com/steamedsheep\" target=\"_blank\">@steamedsheep</a> and <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> </p>",
      "votes": 7,
      "replies": []
    },
    {
      "id": 1247181,
      "author_name": "Miyatti",
      "author_url": "",
      "post_date": "2021-03-21T13:57:48.903000",
      "content": "<p>Congratulations on 2nd place! Thank you for your solutions.</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 1246345,
      "author_name": "Mohammed Rizin V K",
      "author_url": "",
      "post_date": "2021-03-20T16:55:41.420000",
      "content": "<p>Congratulations on #2 win!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 1246928,
      "author_name": "Vitalii Mokin",
      "author_url": "",
      "post_date": "2021-03-21T09:01:47.803000",
      "content": "<p><a href=\"https://www.kaggle.com/steamedsheep\" target=\"_blank\">@steamedsheep</a> , Great solution! Thanks for sharing. I added it to my collection in <a href=\"https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques\" target=\"_blank\">\"Data Science with DL &amp; NLP: Advanced Techniques\"</a>, section \"Prize Competition Winners: notebooks (kernels) and posts with Magic\".</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1506184,
      "author_name": "FP",
      "author_url": "",
      "post_date": "2021-09-08T01:45:00.047000",
      "content": "<p><a href=\"https://www.kaggle.com/steamedsheep\" target=\"_blank\">@steamedsheep</a>, this is great write-up. Thank you and congrats for the wonderful win! Look forward to your github code!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1394779,
      "author_name": "hrunic",
      "author_url": "",
      "post_date": "2021-07-20T15:40:04.803000",
      "content": "<p><a href=\"https://www.kaggle.com/steamedsheep\" target=\"_blank\">@steamedsheep</a> Great writeup! Did you share the code somewhere?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1246816,
      "author_name": "",
      "author_url": "",
      "post_date": "2021-03-21T06:17:38.643000",
      "content": "",
      "votes": 3,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1245969": "Congrats to all the winners, well done! I am grateful to my teammates @nvnnghia @cdeotte and @underwearfitting you all did a very good job and finally we finished in 2nd place.\nI would like to thank @hengck23 for your Unet code at hubmap competition and the wonderful idea of combination of the label and the annotations. Also thanks to @yasufuminakama 's 3-stage training implementation.\nAnd congratulations to VinBigData Group, their employees won 3 solo golds in this competition! @moewie94 @andy2709 @nguyenbadung. Looking forward to read your solutions\n\n## Data\n* For training, we only used RANZCR's training set and the official annotation data. We have tried other annotations like lung mask and key point, but none of them worked well at our setup.\n* We tried pseudo labeling with ChestX dataset we were affected by severe leaking so we gave up pseudo labeling.\n* We used lighter augmentation based on sin's public notebook, cutmix didn’t improve CV and LB  but introduced diversity to our final ensemble.\n* We failed to use image size larger than 1024, even the performance of 1024 image is somehow bad. Our best model is a `Efficientnetb7-unet` model trained at 768 size. We will present it later. \n* Mask generation: we generated masks via plot the line between the annotation points. After training, we generated a full training set pseudo mask by averaging 5 fold's output. Since we use pseudo masks (trainset w/o annotations) as a supervisor, we ignore the potential leakage here. You can refer to this notebook.\n\n## Model\nOur model is a dual-head Unet model. with both classification and segmentation output. Unlike other teams, we only used masks as a supervisor, for two reasons: first, we wanted the model to learn the correct pattern with segmentation loss; secondly,  our data is grouped and collected from different hospitals or patients, which may lead to different quality imaging and mask output. Directly inputting these masks to classification may reduce the generalization performance of our model.\nOur final model used `Resnet200d`, `efficientnet-b5` and `efficientnet-b7` as backbones, unet-decoder part is reduced in order to train with decent VRAM usage.\nWe also tried adding consistency loss to our model but this has slowed the training procedure a lot.\n\n<a href=\"https://ibb.co/F0HYpCT\"><img src=\"https://i.ibb.co/rcdp9Nr/image1.png\" alt=\"image1\" border=\"0\"></a>\n## Training\nWe borrowed stage1 and stage2 from the public notebook, only modifying a few training parameters for stage2.\n### Mask Generation\n* stage3: trained `resnet200d-unet` at 512 and train_annotation generated mask, if an image input without annotations, we calculate the classification loss only. Predicted full train set pseudo mask v1\n* stage4: trained `resnet200d-unet` at 768 with v1 mask and scaled up to 768 size. Generate pseudo mask version v2.\n<img src=\"https://i.ibb.co/8MLZg37/image2.png\" alt=\"image2\" border=\"0\">\n### Model training\n* stage3: trained with `efficientnet-b5-unet`, `efficient-b7-unet`, `resnet200d-unet` at 768 resolution\n* stage4[optional]: scaled up training for `resnet200d-unet` at 1024 resolution\n<a href=\"https://ibb.co/Z2PDZbs\"><img src=\"https://i.ibb.co/7k5m90c/stage3-2.png\" alt=\"stage3-2\" border=\"0\"></a>\n\nOur best performing model is `efficientnet-b7-unet` which achieved `0.9695` CV w/ hflip tta at image size 768, `resnet-200d-unet` achieve `0.9685` cv and `0.972` Public LB w/ hflp tta. A single `resnet-200d-unet` can achieve `0.974` on Private LB and win a gold medal. After ensemble these models, three model ensembles achieve `0.972` CV, `0.97383` Public LB and `0.97599` Private LB.\nSix models ensemble achieved `0.973` CV, `0.97412` Public LB and `0.97641` Private LB.\n\n## Inference\nOur model design enables us to inference with backbone and classification head. We added one 1024 `resnet200d-unet` model, one 768 image size `efficientnet-b5-unet` model and an `efficientnet-b7-unet` model to the inference notebook and it finished in about 8 hours and 50 minutes. We also found an image retrieval trick to reduce the inference time.\n\nRoughly we do the things below:\n* Pseudo label chestx dataset using existing models\n* Train a metric learning model with target of patient ID\n* Use a KNN model to retrieve the top-1 similar image from chestx dataset and submit the pseudo label.\n<img src=\"https://i.ibb.co/r6WCnQw/image4.png\" alt=\"image4\" border=\"0\">\n\nUsing this method, we improved private LB to 0.97641\n\n## Code\nWe are organizing our code and will release our code on github later.\n\n\n\n",
    "1246831": "Great solution! Thanks for sharing details solution @steamedsheep. Congratulations again @underwearfitting @nvnnghia @steamedsheep and @cdeotte ",
    "1247181": "Congratulations on 2nd place! Thank you for your solutions.",
    "1246345": "Congratulations on #2 win!",
    "1246928": "@steamedsheep , Great solution! Thanks for sharing. I added it to my collection in [\"Data Science with DL & NLP: Advanced Techniques\"](https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques), section \"Prize Competition Winners: notebooks (kernels) and posts with Magic\".",
    "1506184": "@steamedsheep, this is great write-up. Thank you and congrats for the wonderful win! Look forward to your github code!",
    "1394779": "@steamedsheep Great writeup! Did you share the code somewhere?",
    "1246816": ""
  }
}