{
  "id": 226622,
  "title": "21st Place Solution : So close Yet So Far",
  "url": "/competitions/ranzcr-clip-catheter-line-classification/writeups/to-be-continued-21st-place-solution-so-close-yet-s",
  "author_name": "",
  "post_date": "2021-03-17T10:35:39.653Z",
  "votes": 33,
  "comment_count": 15,
  "views": 0,
  "content": "<p>Hi all , first of all a big congratulations to the winners , a big thank you to kaggle for organizing this competition . The competition was full of ups and downs for us , we toiled hard , sometimes we thought we could be there at the top and other times we were all discouraged because of the CV-LB gap . I would like to thank <a href=\"https://www.kaggle.com/ttahara\" target=\"_blank\">@ttahara</a> and <a href=\"https://www.kaggle.com/yasufuminakama\" target=\"_blank\">@yasufuminakama</a> for their starting baselines , there was so much new to learn from them and also <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> for sharing his ideas time to time almost all of which I tried .</p>\n<p>Throughout this competition we tried a lot of ambitious and simple ideas and all four of us were training models till the last day . Below are the things that we tried , but didn't work :</p>\n<ul>\n<li>Training 2nd Stage with self attention</li>\n<li>Training 2nd Stage with Transformer attention from teacher</li>\n<li>Training Multi-head model with Spatial attention at every head and self attention before the head and output of the last layer </li>\n<li>Training 3 stages Multi-Head model </li>\n<li>Training 2nd Stage with all 30k images with annotations generated by UNET</li>\n<li>Training 3rd stage with heavy augs by <a href=\"https://www.kaggle.com/underwearfitting\" target=\"_blank\">@underwearfitting</a></li>\n<li>TTA</li>\n<li>Cropping black area preprocess<br>\nand many more</li>\n</ul>\n<p>All of the above techniques didn't provide as a significant boost on lb while our cv improved , at last we decided to stick to simple models</p>\n<h1>Our Strategy</h1>\n<p>We used multi-head models with spatial attention fine tuned on <em>pseudo labelled NIH data, Kaggle SIM competition data and test data</em> and also four stage resnet200d in our ensemble</p>\n<h1>Models</h1>\n<ul>\n<li><p>Backbone</p>\n<ul>\n<li>Resnet200d</li>\n<li>Effnet B5</li>\n<li>Seresnet152d</li></ul></li>\n<li><p>Image SIze</p>\n<ul>\n<li>640</li>\n<li>768</li></ul></li>\n<li><p>Head</p>\n<ul>\n<li>Multi-head</li>\n<li>Multi-stage</li></ul></li>\n<li><p>Pseudo</p>\n<ul>\n<li>NIH Data</li>\n<li>SIIM Data</li>\n<li>Test data (3500 images)</li></ul></li>\n</ul>\n<p>CV strategy : we had used the same folds as given by <a href=\"https://www.kaggle.com/underwearfitting\" target=\"_blank\">@underwearfitting</a> . We found it to every stable and in sync with lb untill we introduced a leakage</p>\n<h1>Ensembling</h1>\n<p>We have used an ensemble of 8 models for our final submission with optuna and oofs to optimize the weights given to each model in our ensemble .</p>\n<h1>Few things that we missed</h1>\n<ul>\n<li>Training on High Resolution as it would have brought diversity in our ensemble and also would have given better lb score but we had limited hardware resources</li>\n<li>Training four stage with our own models instead of using public pretrained models ,might have given us a big shake but we were lucky due to blending of large models we didnt shake</li>\n</ul>\n<p>At the end I would like to thank all my teammates <a href=\"https://www.kaggle.com/nischaydnk\" target=\"_blank\">@nischaydnk</a> , <a href=\"https://www.kaggle.com/adg1822\" target=\"_blank\">@adg1822</a> and  <a href=\"https://www.kaggle.com/shivamcyborg\" target=\"_blank\">@shivamcyborg</a><br>\nThis was my first team with all Indian Members and I have enjoyed a lot , besides learning and getting upset together we have developed a special bond .</p>\n<p>Thanks for reading</p>",
  "messages": [
    {
      "id": "1241591",
      "postDate": "03/17/2021 05:39:30",
      "content": "<p>Hi all , first of all a big congratulations to the winners , a big thank you to kaggle for organizing this competition . The competition was full of ups and downs for us , we toiled hard , sometimes we thought we could be there at the top and other times we were all discouraged because of the CV-LB gap . I would like to thank <a href=\"https://www.kaggle.com/ttahara\" target=\"_blank\">@ttahara</a> and <a href=\"https://www.kaggle.com/yasufuminakama\" target=\"_blank\">@yasufuminakama</a> for their starting baselines , there was so much new to learn from them and also <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> for sharing his ideas time to time almost all of which I tried .</p>\n<p>Throughout this competition we tried a lot of ambitious and simple ideas and all four of us were training models till the last day . Below are the things that we tried , but didn't work :</p>\n<ul>\n<li>Training 2nd Stage with self attention</li>\n<li>Training 2nd Stage with Transformer attention from teacher</li>\n<li>Training Multi-head model with Spatial attention at every head and self attention before the head and output of the last layer </li>\n<li>Training 3 stages Multi-Head model </li>\n<li>Training 2nd Stage with all 30k images with annotations generated by UNET</li>\n<li>Training 3rd stage with heavy augs by <a href=\"https://www.kaggle.com/underwearfitting\" target=\"_blank\">@underwearfitting</a></li>\n<li>TTA</li>\n<li>Cropping black area preprocess<br>\nand many more</li>\n</ul>\n<p>All of the above techniques didn't provide as a significant boost on lb while our cv improved , at last we decided to stick to simple models</p>\n<h1>Our Strategy</h1>\n<p>We used multi-head models with spatial attention fine tuned on <em>pseudo labelled NIH data, Kaggle SIM competition data and test data</em> and also four stage resnet200d in our ensemble</p>\n<h1>Models</h1>\n<ul>\n<li><p>Backbone</p>\n<ul>\n<li>Resnet200d</li>\n<li>Effnet B5</li>\n<li>Seresnet152d</li></ul></li>\n<li><p>Image SIze</p>\n<ul>\n<li>640</li>\n<li>768</li></ul></li>\n<li><p>Head</p>\n<ul>\n<li>Multi-head</li>\n<li>Multi-stage</li></ul></li>\n<li><p>Pseudo</p>\n<ul>\n<li>NIH Data</li>\n<li>SIIM Data</li>\n<li>Test data (3500 images)</li></ul></li>\n</ul>\n<p>CV strategy : we had used the same folds as given by <a href=\"https://www.kaggle.com/underwearfitting\" target=\"_blank\">@underwearfitting</a> . We found it to every stable and in sync with lb untill we introduced a leakage</p>\n<h1>Ensembling</h1>\n<p>We have used an ensemble of 8 models for our final submission with optuna and oofs to optimize the weights given to each model in our ensemble .</p>\n<h1>Few things that we missed</h1>\n<ul>\n<li>Training on High Resolution as it would have brought diversity in our ensemble and also would have given better lb score but we had limited hardware resources</li>\n<li>Training four stage with our own models instead of using public pretrained models ,might have given us a big shake but we were lucky due to blending of large models we didnt shake</li>\n</ul>\n<p>At the end I would like to thank all my teammates <a href=\"https://www.kaggle.com/nischaydnk\" target=\"_blank\">@nischaydnk</a> , <a href=\"https://www.kaggle.com/adg1822\" target=\"_blank\">@adg1822</a> and  <a href=\"https://www.kaggle.com/shivamcyborg\" target=\"_blank\">@shivamcyborg</a><br>\nThis was my first team with all Indian Members and I have enjoyed a lot , besides learning and getting upset together we have developed a special bond .</p>\n<p>Thanks for reading</p>",
      "rawMarkdown": "Hi all , first of all a big congratulations to the winners , a big thank you to kaggle for organizing this competition . The competition was full of ups and downs for us , we toiled hard , sometimes we thought we could be there at the top and other times we were all discouraged because of the CV-LB gap . I would like to thank @ttahara and @yasufuminakama for their starting baselines , there was so much new to learn from them and also @hengck23 for sharing his ideas time to time almost all of which I tried .\n\nThroughout this competition we tried a lot of ambitious and simple ideas and all four of us were training models till the last day . Below are the things that we tried , but didn't work :\n\n* Training 2nd Stage with self attention\n* Training 2nd Stage with Transformer attention from teacher\n* Training Multi-head model with Spatial attention at every head and self attention before the head and output of the last layer \n* Training 3 stages Multi-Head model \n* Training 2nd Stage with all 30k images with annotations generated by UNET\n* Training 3rd stage with heavy augs by @underwearfitting\n* TTA\n* Cropping black area preprocess\nand many more\n\nAll of the above techniques didn't provide as a significant boost on lb while our cv improved , at last we decided to stick to simple models\n\n# Our Strategy\n\nWe used multi-head models with spatial attention fine tuned on *pseudo labelled NIH data, Kaggle SIM competition data and test data* and also four stage resnet200d in our ensemble\n\n# Models\n\n* Backbone\n   * Resnet200d\n   * Effnet B5\n   * Seresnet152d\n\n* Image SIze\n  * 640\n  * 768\n\n* Head\n  * Multi-head\n  * Multi-stage\n\n* Pseudo\n  * NIH Data\n  * SIIM Data\n  * Test data (3500 images)\n\nCV strategy : we had used the same folds as given by @underwearfitting . We found it to every stable and in sync with lb untill we introduced a leakage\n\n# Ensembling\n\nWe have used an ensemble of 8 models for our final submission with optuna and oofs to optimize the weights given to each model in our ensemble .\n\n# Few things that we missed\n\n* Training on High Resolution as it would have brought diversity in our ensemble and also would have given better lb score but we had limited hardware resources\n* Training four stage with our own models instead of using public pretrained models ,might have given us a big shake but we were lucky due to blending of large models we didnt shake\n\n\nAt the end I would like to thank all my teammates @nischaydnk , @adg1822 and  @shivamcyborg\nThis was my first team with all Indian Members and I have enjoyed a lot , besides learning and getting upset together we have developed a special bond .\n\nThanks for reading",
      "votes": null
    },
    {
      "id": "1241604",
      "postDate": "03/17/2021 05:50:34",
      "content": "<p>Can you share How segmentation affected your CV?</p>",
      "rawMarkdown": "Can you share How segmentation affected your CV?",
      "votes": null
    },
    {
      "id": "1241658",
      "postDate": "03/17/2021 06:24:15",
      "content": "<p>Nice! Hungry for gold v3 lezzgo </p>",
      "rawMarkdown": "Nice! Hungry for gold v3 lezzgo",
      "votes": null
    },
    {
      "id": "1241821",
      "postDate": "03/17/2021 08:08:49",
      "content": "<p>We are still hungry.😪</p>",
      "rawMarkdown": "We are still hungry.😪",
      "votes": null
    },
    {
      "id": "1241828",
      "postDate": "03/17/2021 08:13:02",
      "content": "<p>Tried it on 3 stage training, cv wasn't too great, around 0.002 decrement. Segmented images weren't that great, maybe that could be a reason. </p>",
      "rawMarkdown": "Tried it on 3 stage training, cv wasn't too great, around 0.002 decrement. Segmented images weren't that great, maybe that could be a reason.",
      "votes": null
    },
    {
      "id": "1241853",
      "postDate": "03/17/2021 08:45:20",
      "content": "<p>Thank you for sharing.</p>\n<p>I'm glad that my baseline was helpful to you 😃</p>",
      "rawMarkdown": "Thank you for sharing.\n\nI'm glad that my baseline was helpful to you 😃",
      "votes": null
    },
    {
      "id": "1241970",
      "postDate": "03/17/2021 10:04:08",
      "content": "<p>Congratulations. Enjoined the reading thanks for sharing ))</p>",
      "rawMarkdown": "Congratulations. Enjoined the reading thanks for sharing ))",
      "votes": null
    },
    {
      "id": "1242653",
      "postDate": "03/17/2021 18:20:31",
      "content": "<p>Congratulations. And Thanks a lot for sharing your team's great idea. </p>",
      "rawMarkdown": "Congratulations. And Thanks a lot for sharing your team's great idea.",
      "votes": null
    },
    {
      "id": "1242724",
      "postDate": "03/17/2021 18:59:22",
      "content": "<p>Thanks for all your contributions in this competition . I hope we meet again</p>",
      "rawMarkdown": "Thanks for all your contributions in this competition . I hope we meet again",
      "votes": null
    },
    {
      "id": "1242829",
      "postDate": "03/17/2021 20:58:59",
      "content": "<p>Congratulations to you and your team <a href=\"https://www.kaggle.com/tanulsingh077\" target=\"_blank\">@tanulsingh077</a> on securing the 21st position. Your strategy cleared what was lacking with my team's approach. Also, one request, can you elaborate a little more on your ensembling plan.</p>",
      "rawMarkdown": "Congratulations to you and your team @tanulsingh077 on securing the 21st position. Your strategy cleared what was lacking with my team's approach. Also, one request, can you elaborate a little more on your ensembling plan.",
      "votes": null
    },
    {
      "id": "1243462",
      "postDate": "03/18/2021 08:28:44",
      "content": "<p>Thanks for sharing 😊 </p>",
      "rawMarkdown": "Thanks for sharing 😊",
      "votes": null
    },
    {
      "id": "1243523",
      "postDate": "03/18/2021 09:31:29",
      "content": "<p>Thank you for sharing your approach. <br>\nIt's a matter of time before you achieve that gold <a href=\"https://www.kaggle.com/tanulsingh077\" target=\"_blank\">@tanulsingh077</a> , I'm sure of it </p>",
      "rawMarkdown": "Thank you for sharing your approach. \nIt's a matter of time before you achieve that gold @tanulsingh077 , I'm sure of it",
      "votes": null
    },
    {
      "id": "1243626",
      "postDate": "03/18/2021 11:28:01",
      "content": "<p>Thanks <a href=\"https://www.kaggle.com/sanchitvj\" target=\"_blank\">@sanchitvj</a> , I am glad you liked our approach</p>",
      "rawMarkdown": "Thanks @sanchitvj , I am glad you liked our approach",
      "votes": null
    },
    {
      "id": "1243627",
      "postDate": "03/18/2021 11:28:18",
      "content": "<p>Thanks <a href=\"https://www.kaggle.com/nyleve\" target=\"_blank\">@nyleve</a> for the kind words</p>",
      "rawMarkdown": "Thanks @nyleve for the kind words",
      "votes": null
    },
    {
      "id": "1245122",
      "postDate": "03/19/2021 13:29:24",
      "content": "<p><a href=\"https://www.kaggle.com/tanulsingh077\" target=\"_blank\">@tanulsingh077</a> Thanks for sharing your solution here</p>",
      "rawMarkdown": "tanulsingh077 Thanks for sharing your solution here",
      "votes": null
    },
    {
      "id": "1246200",
      "postDate": "03/20/2021 14:34:48",
      "content": "<p>Thanks for sharing your insights. If I may ask, what type of hardware have you used? </p>",
      "rawMarkdown": "Thanks for sharing your insights. If I may ask, what type of hardware have you used?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1241604,
      "author_name": "morizin",
      "author_url": "",
      "post_date": "03/17/2021 05:50:34",
      "content": "<p>Can you share How segmentation affected your CV?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1241828,
          "author_name": "nischaydnk",
          "author_url": "",
          "post_date": "03/17/2021 08:13:02",
          "content": "<p>Tried it on 3 stage training, cv wasn't too great, around 0.002 decrement. Segmented images weren't that great, maybe that could be a reason. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1241658,
      "author_name": "reighns",
      "author_url": "",
      "post_date": "03/17/2021 06:24:15",
      "content": "<p>Nice! Hungry for gold v3 lezzgo </p>",
      "votes": null,
      "replies": [
        {
          "id": 1241821,
          "author_name": "nischaydnk",
          "author_url": "",
          "post_date": "03/17/2021 08:08:49",
          "content": "<p>We are still hungry.😪</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1241853,
      "author_name": "ttahara",
      "author_url": "",
      "post_date": "03/17/2021 08:45:20",
      "content": "<p>Thank you for sharing.</p>\n<p>I'm glad that my baseline was helpful to you 😃</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1241970,
      "author_name": "ammarali32",
      "author_url": "",
      "post_date": "03/17/2021 10:04:08",
      "content": "<p>Congratulations. Enjoined the reading thanks for sharing ))</p>",
      "votes": null,
      "replies": [
        {
          "id": 1242724,
          "author_name": "tanulsingh077",
          "author_url": "",
          "post_date": "03/17/2021 18:59:22",
          "content": "<p>Thanks for all your contributions in this competition . I hope we meet again</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1242653,
      "author_name": "durbin164",
      "author_url": "",
      "post_date": "03/17/2021 18:20:31",
      "content": "<p>Congratulations. And Thanks a lot for sharing your team's great idea. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1242829,
      "author_name": "sanchitvj",
      "author_url": "",
      "post_date": "03/17/2021 20:58:59",
      "content": "<p>Congratulations to you and your team <a href=\"https://www.kaggle.com/tanulsingh077\" target=\"_blank\">@tanulsingh077</a> on securing the 21st position. Your strategy cleared what was lacking with my team's approach. Also, one request, can you elaborate a little more on your ensembling plan.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1243626,
          "author_name": "tanulsingh077",
          "author_url": "",
          "post_date": "03/18/2021 11:28:01",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/sanchitvj\" target=\"_blank\">@sanchitvj</a> , I am glad you liked our approach</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1243462,
      "author_name": "milobele",
      "author_url": "",
      "post_date": "03/18/2021 08:28:44",
      "content": "<p>Thanks for sharing 😊 </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1243523,
      "author_name": "nyleve",
      "author_url": "",
      "post_date": "03/18/2021 09:31:29",
      "content": "<p>Thank you for sharing your approach. <br>\nIt's a matter of time before you achieve that gold <a href=\"https://www.kaggle.com/tanulsingh077\" target=\"_blank\">@tanulsingh077</a> , I'm sure of it </p>",
      "votes": null,
      "replies": [
        {
          "id": 1243627,
          "author_name": "tanulsingh077",
          "author_url": "",
          "post_date": "03/18/2021 11:28:18",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/nyleve\" target=\"_blank\">@nyleve</a> for the kind words</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1245122,
      "author_name": "digvijayyadav",
      "author_url": "",
      "post_date": "03/19/2021 13:29:24",
      "content": "<p><a href=\"https://www.kaggle.com/tanulsingh077\" target=\"_blank\">@tanulsingh077</a> Thanks for sharing your solution here</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1246200,
      "author_name": "yassinealouini",
      "author_url": "",
      "post_date": "03/20/2021 14:34:48",
      "content": "<p>Thanks for sharing your insights. If I may ask, what type of hardware have you used? </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1241591": "Hi all , first of all a big congratulations to the winners , a big thank you to kaggle for organizing this competition . The competition was full of ups and downs for us , we toiled hard , sometimes we thought we could be there at the top and other times we were all discouraged because of the CV-LB gap . I would like to thank @ttahara and @yasufuminakama for their starting baselines , there was so much new to learn from them and also @hengck23 for sharing his ideas time to time almost all of which I tried .\n\nThroughout this competition we tried a lot of ambitious and simple ideas and all four of us were training models till the last day . Below are the things that we tried , but didn't work :\n\n* Training 2nd Stage with self attention\n* Training 2nd Stage with Transformer attention from teacher\n* Training Multi-head model with Spatial attention at every head and self attention before the head and output of the last layer \n* Training 3 stages Multi-Head model \n* Training 2nd Stage with all 30k images with annotations generated by UNET\n* Training 3rd stage with heavy augs by @underwearfitting\n* TTA\n* Cropping black area preprocess\nand many more\n\nAll of the above techniques didn't provide as a significant boost on lb while our cv improved , at last we decided to stick to simple models\n\n# Our Strategy\n\nWe used multi-head models with spatial attention fine tuned on *pseudo labelled NIH data, Kaggle SIM competition data and test data* and also four stage resnet200d in our ensemble\n\n# Models\n\n* Backbone\n   * Resnet200d\n   * Effnet B5\n   * Seresnet152d\n\n* Image SIze\n  * 640\n  * 768\n\n* Head\n  * Multi-head\n  * Multi-stage\n\n* Pseudo\n  * NIH Data\n  * SIIM Data\n  * Test data (3500 images)\n\nCV strategy : we had used the same folds as given by @underwearfitting . We found it to every stable and in sync with lb untill we introduced a leakage\n\n# Ensembling\n\nWe have used an ensemble of 8 models for our final submission with optuna and oofs to optimize the weights given to each model in our ensemble .\n\n# Few things that we missed\n\n* Training on High Resolution as it would have brought diversity in our ensemble and also would have given better lb score but we had limited hardware resources\n* Training four stage with our own models instead of using public pretrained models ,might have given us a big shake but we were lucky due to blending of large models we didnt shake\n\n\nAt the end I would like to thank all my teammates @nischaydnk , @adg1822 and  @shivamcyborg\nThis was my first team with all Indian Members and I have enjoyed a lot , besides learning and getting upset together we have developed a special bond .\n\nThanks for reading",
    "1241604": "Can you share How segmentation affected your CV?",
    "1241658": "Nice! Hungry for gold v3 lezzgo",
    "1241821": "We are still hungry.😪",
    "1241828": "Tried it on 3 stage training, cv wasn't too great, around 0.002 decrement. Segmented images weren't that great, maybe that could be a reason.",
    "1241853": "Thank you for sharing.\n\nI'm glad that my baseline was helpful to you 😃",
    "1241970": "Congratulations. Enjoined the reading thanks for sharing ))",
    "1242653": "Congratulations. And Thanks a lot for sharing your team's great idea.",
    "1242724": "Thanks for all your contributions in this competition . I hope we meet again",
    "1242829": "Congratulations to you and your team @tanulsingh077 on securing the 21st position. Your strategy cleared what was lacking with my team's approach. Also, one request, can you elaborate a little more on your ensembling plan.",
    "1243462": "Thanks for sharing 😊",
    "1243523": "Thank you for sharing your approach. \nIt's a matter of time before you achieve that gold @tanulsingh077 , I'm sure of it",
    "1243626": "Thanks @sanchitvj , I am glad you liked our approach",
    "1243627": "Thanks @nyleve for the kind words",
    "1245122": "tanulsingh077 Thanks for sharing your solution here",
    "1246200": "Thanks for sharing your insights. If I may ask, what type of hardware have you used?"
  },
  "source": "meta"
}