{
  "id": 226702,
  "title": "19th Place Solution - First Time With NFNets",
  "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/226702",
  "author_name": "Ertuğrul Demir",
  "post_date": "2021-03-17T11:10:17.781000",
  "votes": 29,
  "comment_count": 19,
  "views": 0,
  "content": "<p>Hey all, congrats to all for their hard work. Before I start I'd like to thank my teammates <a href=\"https://www.kaggle.com/pukkinming\" target=\"_blank\">@pukkinming</a> , <a href=\"https://www.kaggle.com/frtgnn\" target=\"_blank\">@frtgnn</a> and <a href=\"https://www.kaggle.com/divyansh22\" target=\"_blank\">@divyansh22</a> for their hard work and commitment. The competition was really interesting for us and learned a lot on the way.</p>\n<p>Also would like to thank all the competitors where they shared their codes, ideas, discussions, datasets etc.</p>\n<p>Along the way we tried many different ideas, both simple or complex, but I'm going to share key points taken to keep things simple:</p>\n<h1>Preprocessing</h1>\n<ul>\n<li>We went with 640x image size, tried couple other resolutions but didn't get expected scores or we couldn't go with them because of performance or timing reasons.</li>\n<li>For augmentations we usually went something medicore aprroach.</li>\n<li>We tried removing black boxes around but didn't give us better results.</li>\n</ul>\n<h1>Models</h1>\n<p>We tried several architectures but we got CV success mainly on two of them:</p>\n<ul>\n<li>NfNet F3,</li>\n<li>ResNet 200d</li>\n</ul>\n<h1>Training</h1>\n<p>After getting stable CV/LB correlation we tried to get our CV high as possible, first we tried ImageNet pretrained models on training images themselves to test limits of these with different losses, schedulers, optimizer combinations.</p>\n<ul>\n<li>We found out that Ranger optimizer converges faster while SAM optimizer generalizes better. So in the light of that we decided to use some kind of distillation of model using Ranger first then fine tuning to find minima with SAM optimizer.</li>\n</ul>\n<h1>Pseudo Labeling</h1>\n<ul>\n<li><p>With the findings from distillation we decided to train model on bigger dataset and transfer learning to main model itself, so we start working on external datasets like NIH. </p></li>\n<li><p>We pseudo labeled NIH dataset with our best single models and then retrained these models with the created labels.</p></li>\n<li><p>This method increased our CV considerably, so we kept trying different approaches for increasing the CV with this method.</p></li>\n</ul>\n<h1>Ensemble</h1>\n<ul>\n<li>After getting satisfactory results with CV's we went with ensembling our best CV models. </li>\n<li>Weighting were based on CV's with simple averaging while NfNet model taking huge weight on this followed by ResNet200d.</li>\n</ul>\n<h1>Post Processing</h1>\n<ul>\n<li>We tried several approaches based on train/test distributions but no avail.</li>\n</ul>\n<h1>Trusting your CV</h1>\n<p>In last hours we were deciding to choose which submission would be to best selection. At the end we went with highest CV ensemble and found out it was actually our best private submission.</p>\n<h1>Side Note About NfNets:</h1>\n<p>I'd say our NfNet F3 model gave really robust result and made us survive the shakeup. At some point we were almost giving up on that architecture and move on because of the performance/time trade. I want to thank my teammates again for their patience with my obsession about the model and enduring my endless improvement ideas about them :)</p>\n<p>I actually shared one of the earlier versions as public notebook, our private model was little bit modified/pretrained version with heavier approach. You can check the code here:</p>\n<p><a href=\"https://www.kaggle.com/datafan07/ranzcr-nfnets-tutorial-single-fold-training\" target=\"_blank\">https://www.kaggle.com/datafan07/ranzcr-nfnets-tutorial-single-fold-training</a></p>",
  "messages": [
    {
      "id": 1242048,
      "postDate": "2021-03-17T11:10:17.780Z",
      "content": "<p>Hey all, congrats to all for their hard work. Before I start I'd like to thank my teammates <a href=\"https://www.kaggle.com/pukkinming\" target=\"_blank\">@pukkinming</a> , <a href=\"https://www.kaggle.com/frtgnn\" target=\"_blank\">@frtgnn</a> and <a href=\"https://www.kaggle.com/divyansh22\" target=\"_blank\">@divyansh22</a> for their hard work and commitment. The competition was really interesting for us and learned a lot on the way.</p>\n<p>Also would like to thank all the competitors where they shared their codes, ideas, discussions, datasets etc.</p>\n<p>Along the way we tried many different ideas, both simple or complex, but I'm going to share key points taken to keep things simple:</p>\n<h1>Preprocessing</h1>\n<ul>\n<li>We went with 640x image size, tried couple other resolutions but didn't get expected scores or we couldn't go with them because of performance or timing reasons.</li>\n<li>For augmentations we usually went something medicore aprroach.</li>\n<li>We tried removing black boxes around but didn't give us better results.</li>\n</ul>\n<h1>Models</h1>\n<p>We tried several architectures but we got CV success mainly on two of them:</p>\n<ul>\n<li>NfNet F3,</li>\n<li>ResNet 200d</li>\n</ul>\n<h1>Training</h1>\n<p>After getting stable CV/LB correlation we tried to get our CV high as possible, first we tried ImageNet pretrained models on training images themselves to test limits of these with different losses, schedulers, optimizer combinations.</p>\n<ul>\n<li>We found out that Ranger optimizer converges faster while SAM optimizer generalizes better. So in the light of that we decided to use some kind of distillation of model using Ranger first then fine tuning to find minima with SAM optimizer.</li>\n</ul>\n<h1>Pseudo Labeling</h1>\n<ul>\n<li><p>With the findings from distillation we decided to train model on bigger dataset and transfer learning to main model itself, so we start working on external datasets like NIH. </p></li>\n<li><p>We pseudo labeled NIH dataset with our best single models and then retrained these models with the created labels.</p></li>\n<li><p>This method increased our CV considerably, so we kept trying different approaches for increasing the CV with this method.</p></li>\n</ul>\n<h1>Ensemble</h1>\n<ul>\n<li>After getting satisfactory results with CV's we went with ensembling our best CV models. </li>\n<li>Weighting were based on CV's with simple averaging while NfNet model taking huge weight on this followed by ResNet200d.</li>\n</ul>\n<h1>Post Processing</h1>\n<ul>\n<li>We tried several approaches based on train/test distributions but no avail.</li>\n</ul>\n<h1>Trusting your CV</h1>\n<p>In last hours we were deciding to choose which submission would be to best selection. At the end we went with highest CV ensemble and found out it was actually our best private submission.</p>\n<h1>Side Note About NfNets:</h1>\n<p>I'd say our NfNet F3 model gave really robust result and made us survive the shakeup. At some point we were almost giving up on that architecture and move on because of the performance/time trade. I want to thank my teammates again for their patience with my obsession about the model and enduring my endless improvement ideas about them :)</p>\n<p>I actually shared one of the earlier versions as public notebook, our private model was little bit modified/pretrained version with heavier approach. You can check the code here:</p>\n<p><a href=\"https://www.kaggle.com/datafan07/ranzcr-nfnets-tutorial-single-fold-training\" target=\"_blank\">https://www.kaggle.com/datafan07/ranzcr-nfnets-tutorial-single-fold-training</a></p>",
      "rawMarkdown": "Hey all, congrats to all for their hard work. Before I start I'd like to thank my teammates @pukkinming , @frtgnn and @divyansh22 for their hard work and commitment. The competition was really interesting for us and learned a lot on the way.\n\nAlso would like to thank all the competitors where they shared their codes, ideas, discussions, datasets etc.\n\nAlong the way we tried many different ideas, both simple or complex, but I'm going to share key points taken to keep things simple:\n\n# Preprocessing\n\n\n- We went with 640x image size, tried couple other resolutions but didn't get expected scores or we couldn't go with them because of performance or timing reasons.\n- For augmentations we usually went something medicore aprroach.\n- We tried removing black boxes around but didn't give us better results.\n\n# Models\n\nWe tried several architectures but we got CV success mainly on two of them:\n\n- NfNet F3,\n- ResNet 200d\n\n# Training\n\nAfter getting stable CV/LB correlation we tried to get our CV high as possible, first we tried ImageNet pretrained models on training images themselves to test limits of these with different losses, schedulers, optimizer combinations.\n\n- We found out that Ranger optimizer converges faster while SAM optimizer generalizes better. So in the light of that we decided to use some kind of distillation of model using Ranger first then fine tuning to find minima with SAM optimizer.\n\n# Pseudo Labeling\n\n- With the findings from distillation we decided to train model on bigger dataset and transfer learning to main model itself, so we start working on external datasets like NIH. \n\n- We pseudo labeled NIH dataset with our best single models and then retrained these models with the created labels.\n- This method increased our CV considerably, so we kept trying different approaches for increasing the CV with this method.\n\n# Ensemble\n\n- After getting satisfactory results with CV's we went with ensembling our best CV models. \n- Weighting were based on CV's with simple averaging while NfNet model taking huge weight on this followed by ResNet200d.\n\n# Post Processing\n\n- We tried several approaches based on train/test distributions but no avail.\n\n# Trusting your CV\n\nIn last hours we were deciding to choose which submission would be to best selection. At the end we went with highest CV ensemble and found out it was actually our best private submission.\n\n# Side Note About NfNets:\n\nI'd say our NfNet F3 model gave really robust result and made us survive the shakeup. At some point we were almost giving up on that architecture and move on because of the performance/time trade. I want to thank my teammates again for their patience with my obsession about the model and enduring my endless improvement ideas about them :)\n\nI actually shared one of the earlier versions as public notebook, our private model was little bit modified/pretrained version with heavier approach. You can check the code here:\n\nhttps://www.kaggle.com/datafan07/ranzcr-nfnets-tutorial-single-fold-training",
      "votes": 29
    },
    {
      "id": 1242962,
      "postDate": "2021-03-17T23:35:47.810Z",
      "content": "<p>Congrats Ertuğrul and team. It's exciting to see cutting edge NfNets in a top solution.</p>",
      "rawMarkdown": "Congrats Ertuğrul and team. It's exciting to see cutting edge NfNets in a top solution.",
      "votes": 3,
      "replies": [
        {
          "id": 1243527,
          "postDate": "2021-03-18T09:37:44.230Z",
          "content": "<p>Thanks Chris, congrats to you and your team too, I really liked your teams predicting/finding similar images from nih solution!</p>",
          "rawMarkdown": "Thanks Chris, congrats to you and your team too, I really liked your teams predicting/finding similar images from nih solution!",
          "votes": 1
        }
      ]
    },
    {
      "id": 1242093,
      "postDate": "2021-03-17T11:54:12.913Z",
      "content": "<p>Congrats team, great approach!! Sticking on your intuitions and achieving a success is priceless. Thanks for the detailed and very informative write up!</p>\n<p>Also congrats on becoming a Competition Master as well 💪 </p>",
      "rawMarkdown": "Congrats team, great approach!! Sticking on your intuitions and achieving a success is priceless. Thanks for the detailed and very informative write up!\n\nAlso congrats on becoming a Competition Master as well 💪 ",
      "votes": 3
    },
    {
      "id": 1242191,
      "postDate": "2021-03-17T13:15:07.727Z",
      "content": "<p>Me and <a href=\"https://www.kaggle.com/khoongweihao\" target=\"_blank\">@khoongweihao</a> gave up on NFNET halfway! It’s so hard to train on and the cv always quite low </p>",
      "rawMarkdown": "Me and @khoongweihao gave up on NFNET halfway! It’s so hard to train on and the cv always quite low ",
      "votes": 4,
      "replies": [
        {
          "id": 1242211,
          "postDate": "2021-03-17T13:31:59.367Z",
          "content": "<p>If you use pretrained weights with NIH external dataset to train Nfnet, CV can be pretty good.</p>",
          "rawMarkdown": "If you use pretrained weights with NIH external dataset to train Nfnet, CV can be pretty good.",
          "votes": 2
        },
        {
          "id": 1242213,
          "postDate": "2021-03-17T13:32:16.833Z",
          "content": "<p>And yes, Nfnet is GPU memory hungry. </p>",
          "rawMarkdown": "And yes, Nfnet is GPU memory hungry. ",
          "votes": 1
        },
        {
          "id": 1242313,
          "postDate": "2021-03-17T14:40:29.007Z",
          "content": "<p>Yea man, I didn’t thought of using the NIH external data. It really seems to be quite hard to converge. I’ll try it out again. </p>",
          "rawMarkdown": "Yea man, I didn’t thought of using the NIH external data. It really seems to be quite hard to converge. I’ll try it out again. ",
          "votes": 1
        },
        {
          "id": 1242315,
          "postDate": "2021-03-17T14:41:13.833Z",
          "content": "<p>The GPU consumption is insane… We could not go beyond 0.902 public LB with NFNets, which came from NFNet-F5. The GPU part made us give up early on NFNets so we couldn't tune the hyperparameters much :( But interesting to know that NFNet does indeed work well for some teams! Congrats 👍</p>",
          "rawMarkdown": "The GPU consumption is insane... We could not go beyond 0.902 public LB with NFNets, which came from NFNet-F5. The GPU part made us give up early on NFNets so we couldn't tune the hyperparameters much :( But interesting to know that NFNet does indeed work well for some teams! Congrats 👍",
          "votes": 1
        },
        {
          "id": 1242403,
          "postDate": "2021-03-17T15:35:41.673Z",
          "content": "<p><a href=\"https://www.kaggle.com/reighns\" target=\"_blank\">@reighns</a> we experience the convergence issue too so we just started with the basic - lowering initial LR and lighter augmentation and things worked again.</p>",
          "rawMarkdown": "@reighns we experience the convergence issue too so we just started with the basic - lowering initial LR and lighter augmentation and things worked again.",
          "votes": 1
        }
      ]
    },
    {
      "id": 1242495,
      "postDate": "2021-03-17T16:39:04.773Z",
      "content": "<p>Great work!</p>",
      "rawMarkdown": "Great work!",
      "votes": 1
    },
    {
      "id": 1242153,
      "postDate": "2021-03-17T12:45:45.123Z",
      "content": "<p>Great work !</p>",
      "rawMarkdown": "Great work !",
      "votes": 1
    },
    {
      "id": 1242295,
      "postDate": "2021-03-17T14:26:04.347Z",
      "content": "<p>Congrats <a href=\"https://www.kaggle.com/datafan07\" target=\"_blank\">@datafan07</a> and team on the results, NFNet is better on private LB</p>",
      "rawMarkdown": "Congrats @datafan07 and team on the results, NFNet is better on private LB",
      "votes": 2
    },
    {
      "id": 1242104,
      "postDate": "2021-03-17T11:58:45.507Z",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/datafan07\" target=\"_blank\">@datafan07</a> for becoming competition Master :)<br>\nWhat is the public score of your best private? The gap between my public and private is consistent around +0.002 (Pub=0.969, Pri=0.971), yours should be minimum +0.004.<br>\nDo you think NfNet F3 contributed to this large gap?</p>",
      "rawMarkdown": "Congratulations @datafan07 for becoming competition Master :)\nWhat is the public score of your best private? The gap between my public and private is consistent around +0.002 (Pub=0.969, Pri=0.971), yours should be minimum +0.004.\nDo you think NfNet F3 contributed to this large gap?",
      "votes": 2,
      "replies": [
        {
          "id": 1242122,
          "postDate": "2021-03-17T12:15:52.430Z",
          "content": "<p>I think so, it's +0.005 with nfnet and +0.003 without nfnet if I remember correctly.</p>",
          "rawMarkdown": "I think so, it's +0.005 with nfnet and +0.003 without nfnet if I remember correctly.",
          "votes": 1
        }
      ]
    },
    {
      "id": 1242086,
      "postDate": "2021-03-17T11:48:56.783Z",
      "content": "<p><a href=\"https://www.kaggle.com/datafan07\" target=\"_blank\">@datafan07</a> Congratulations on Silver Finish . Thanks for sharing great writeup</p>",
      "rawMarkdown": "@datafan07 Congratulations on Silver Finish . Thanks for sharing great writeup",
      "votes": 2
    },
    {
      "id": 1242699,
      "postDate": "2021-03-17T18:41:13.123Z",
      "content": "<p>Congratz ! Would you say NFNets are actually viable ? As you're saying, they seem really long to train and I'd probably consider some big resnet/efficientnet arch at that point ^^</p>",
      "rawMarkdown": "Congratz ! Would you say NFNets are actually viable ? As you're saying, they seem really long to train and I'd probably consider some big resnet/efficientnet arch at that point ^^",
      "replies": [
        {
          "id": 1243529,
          "postDate": "2021-03-18T09:40:43.560Z",
          "content": "<p>I think they will be in time. <a href=\"https://www.kaggle.com/rwightman\" target=\"_blank\">@rwightman</a> still trying to get them more practical in his great library:</p>\n<p><a href=\"https://twitter.com/wightmanr/status/1372292319634882560\" target=\"_blank\">https://twitter.com/wightmanr/status/1372292319634882560</a></p>",
          "rawMarkdown": "I think they will be in time. @rwightman still trying to get them more practical in his great library:\n\nhttps://twitter.com/wightmanr/status/1372292319634882560",
          "votes": 2
        }
      ]
    },
    {
      "id": 1242695,
      "postDate": "2021-03-17T18:39:09.853Z",
      "content": "<p>Congratulations and Thanks for sharing with us. Can you share your sudo-label create Kernal or sudo-label csv file. Thanks in Advance. </p>",
      "rawMarkdown": "Congratulations and Thanks for sharing with us. Can you share your sudo-label create Kernal or sudo-label csv file. Thanks in Advance. "
    },
    {
      "id": 1246824,
      "postDate": "2021-03-21T06:21:43.273Z",
      "rawMarkdown": "",
      "votes": 1,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1242962,
      "author_name": "Chris Deotte",
      "author_url": "",
      "post_date": "2021-03-17T23:35:47.810000",
      "content": "<p>Congrats Ertuğrul and team. It's exciting to see cutting edge NfNets in a top solution.</p>",
      "votes": 3,
      "replies": [
        {
          "id": 1243527,
          "author_name": "Ertuğrul Demir",
          "author_url": "",
          "post_date": "2021-03-18T09:37:44.230000",
          "content": "<p>Thanks Chris, congrats to you and your team too, I really liked your teams predicting/finding similar images from nih solution!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1242093,
      "author_name": "Sinan Calisir",
      "author_url": "",
      "post_date": "2021-03-17T11:54:12.913000",
      "content": "<p>Congrats team, great approach!! Sticking on your intuitions and achieving a success is priceless. Thanks for the detailed and very informative write up!</p>\n<p>Also congrats on becoming a Competition Master as well 💪 </p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 1242191,
      "author_name": "gao-hongnan",
      "author_url": "",
      "post_date": "2021-03-17T13:15:07.727000",
      "content": "<p>Me and <a href=\"https://www.kaggle.com/khoongweihao\" target=\"_blank\">@khoongweihao</a> gave up on NFNET halfway! It’s so hard to train on and the cv always quite low </p>",
      "votes": 4,
      "replies": [
        {
          "id": 1242211,
          "author_name": "FP",
          "author_url": "",
          "post_date": "2021-03-17T13:31:59.367000",
          "content": "<p>If you use pretrained weights with NIH external dataset to train Nfnet, CV can be pretty good.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1242213,
          "author_name": "FP",
          "author_url": "",
          "post_date": "2021-03-17T13:32:16.833000",
          "content": "<p>And yes, Nfnet is GPU memory hungry. </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1242313,
          "author_name": "gao-hongnan",
          "author_url": "",
          "post_date": "2021-03-17T14:40:29.007000",
          "content": "<p>Yea man, I didn’t thought of using the NIH external data. It really seems to be quite hard to converge. I’ll try it out again. </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1242315,
          "author_name": "Wei Hao Khoong",
          "author_url": "",
          "post_date": "2021-03-17T14:41:13.833000",
          "content": "<p>The GPU consumption is insane… We could not go beyond 0.902 public LB with NFNets, which came from NFNet-F5. The GPU part made us give up early on NFNets so we couldn't tune the hyperparameters much :( But interesting to know that NFNet does indeed work well for some teams! Congrats 👍</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1242403,
          "author_name": "FP",
          "author_url": "",
          "post_date": "2021-03-17T15:35:41.673000",
          "content": "<p><a href=\"https://www.kaggle.com/reighns\" target=\"_blank\">@reighns</a> we experience the convergence issue too so we just started with the basic - lowering initial LR and lighter augmentation and things worked again.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1242495,
      "author_name": "Miyatti",
      "author_url": "",
      "post_date": "2021-03-17T16:39:04.773000",
      "content": "<p>Great work!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1242153,
      "author_name": "Emir Koçak",
      "author_url": "",
      "post_date": "2021-03-17T12:45:45.123000",
      "content": "<p>Great work !</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1242295,
      "author_name": "KhanhVD",
      "author_url": "",
      "post_date": "2021-03-17T14:26:04.347000",
      "content": "<p>Congrats <a href=\"https://www.kaggle.com/datafan07\" target=\"_blank\">@datafan07</a> and team on the results, NFNet is better on private LB</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 1242104,
      "author_name": "Amin",
      "author_url": "",
      "post_date": "2021-03-17T11:58:45.507000",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/datafan07\" target=\"_blank\">@datafan07</a> for becoming competition Master :)<br>\nWhat is the public score of your best private? The gap between my public and private is consistent around +0.002 (Pub=0.969, Pri=0.971), yours should be minimum +0.004.<br>\nDo you think NfNet F3 contributed to this large gap?</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1242122,
          "author_name": "Ertuğrul Demir",
          "author_url": "",
          "post_date": "2021-03-17T12:15:52.430000",
          "content": "<p>I think so, it's +0.005 with nfnet and +0.003 without nfnet if I remember correctly.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1242086,
      "author_name": "Tensor Girl",
      "author_url": "",
      "post_date": "2021-03-17T11:48:56.783000",
      "content": "<p><a href=\"https://www.kaggle.com/datafan07\" target=\"_blank\">@datafan07</a> Congratulations on Silver Finish . Thanks for sharing great writeup</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 1242699,
      "author_name": "Theo Viel",
      "author_url": "",
      "post_date": "2021-03-17T18:41:13.123000",
      "content": "<p>Congratz ! Would you say NFNets are actually viable ? As you're saying, they seem really long to train and I'd probably consider some big resnet/efficientnet arch at that point ^^</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1243529,
          "author_name": "Ertuğrul Demir",
          "author_url": "",
          "post_date": "2021-03-18T09:40:43.560000",
          "content": "<p>I think they will be in time. <a href=\"https://www.kaggle.com/rwightman\" target=\"_blank\">@rwightman</a> still trying to get them more practical in his great library:</p>\n<p><a href=\"https://twitter.com/wightmanr/status/1372292319634882560\" target=\"_blank\">https://twitter.com/wightmanr/status/1372292319634882560</a></p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1242695,
      "author_name": "Md. Masud Rana",
      "author_url": "",
      "post_date": "2021-03-17T18:39:09.853000",
      "content": "<p>Congratulations and Thanks for sharing with us. Can you share your sudo-label create Kernal or sudo-label csv file. Thanks in Advance. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1246824,
      "author_name": "",
      "author_url": "",
      "post_date": "2021-03-21T06:21:43.273000",
      "content": "",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1242048": "Hey all, congrats to all for their hard work. Before I start I'd like to thank my teammates @pukkinming , @frtgnn and @divyansh22 for their hard work and commitment. The competition was really interesting for us and learned a lot on the way.\n\nAlso would like to thank all the competitors where they shared their codes, ideas, discussions, datasets etc.\n\nAlong the way we tried many different ideas, both simple or complex, but I'm going to share key points taken to keep things simple:\n\n# Preprocessing\n\n\n- We went with 640x image size, tried couple other resolutions but didn't get expected scores or we couldn't go with them because of performance or timing reasons.\n- For augmentations we usually went something medicore aprroach.\n- We tried removing black boxes around but didn't give us better results.\n\n# Models\n\nWe tried several architectures but we got CV success mainly on two of them:\n\n- NfNet F3,\n- ResNet 200d\n\n# Training\n\nAfter getting stable CV/LB correlation we tried to get our CV high as possible, first we tried ImageNet pretrained models on training images themselves to test limits of these with different losses, schedulers, optimizer combinations.\n\n- We found out that Ranger optimizer converges faster while SAM optimizer generalizes better. So in the light of that we decided to use some kind of distillation of model using Ranger first then fine tuning to find minima with SAM optimizer.\n\n# Pseudo Labeling\n\n- With the findings from distillation we decided to train model on bigger dataset and transfer learning to main model itself, so we start working on external datasets like NIH. \n\n- We pseudo labeled NIH dataset with our best single models and then retrained these models with the created labels.\n- This method increased our CV considerably, so we kept trying different approaches for increasing the CV with this method.\n\n# Ensemble\n\n- After getting satisfactory results with CV's we went with ensembling our best CV models. \n- Weighting were based on CV's with simple averaging while NfNet model taking huge weight on this followed by ResNet200d.\n\n# Post Processing\n\n- We tried several approaches based on train/test distributions but no avail.\n\n# Trusting your CV\n\nIn last hours we were deciding to choose which submission would be to best selection. At the end we went with highest CV ensemble and found out it was actually our best private submission.\n\n# Side Note About NfNets:\n\nI'd say our NfNet F3 model gave really robust result and made us survive the shakeup. At some point we were almost giving up on that architecture and move on because of the performance/time trade. I want to thank my teammates again for their patience with my obsession about the model and enduring my endless improvement ideas about them :)\n\nI actually shared one of the earlier versions as public notebook, our private model was little bit modified/pretrained version with heavier approach. You can check the code here:\n\nhttps://www.kaggle.com/datafan07/ranzcr-nfnets-tutorial-single-fold-training",
    "1242962": "Congrats Ertuğrul and team. It's exciting to see cutting edge NfNets in a top solution.",
    "1242093": "Congrats team, great approach!! Sticking on your intuitions and achieving a success is priceless. Thanks for the detailed and very informative write up!\n\nAlso congrats on becoming a Competition Master as well 💪 ",
    "1242191": "Me and @khoongweihao gave up on NFNET halfway! It’s so hard to train on and the cv always quite low ",
    "1242495": "Great work!",
    "1242153": "Great work !",
    "1242295": "Congrats @datafan07 and team on the results, NFNet is better on private LB",
    "1242104": "Congratulations @datafan07 for becoming competition Master :)\nWhat is the public score of your best private? The gap between my public and private is consistent around +0.002 (Pub=0.969, Pri=0.971), yours should be minimum +0.004.\nDo you think NfNet F3 contributed to this large gap?",
    "1242086": "@datafan07 Congratulations on Silver Finish . Thanks for sharing great writeup",
    "1242699": "Congratz ! Would you say NFNets are actually viable ? As you're saying, they seem really long to train and I'd probably consider some big resnet/efficientnet arch at that point ^^",
    "1242695": "Congratulations and Thanks for sharing with us. Can you share your sudo-label create Kernal or sudo-label csv file. Thanks in Advance. ",
    "1246824": ""
  }
}