{
  "id": 110434,
  "title": "8th place solution summary",
  "url": "/competitions/recursion-cellular-image-classification/writeups/larko-s-darkos-8th-place-solution-summary",
  "author_name": "",
  "post_date": "2019-09-27T19:12:10.997Z",
  "votes": 39,
  "comment_count": 9,
  "views": 0,
  "content": "<p>Congrats to all on a great competition, and especially to top3. Please do share the NeurIPS content you present. </p>\n\n<p>Shout out to Catalyst team <code>Reproducible and fast DL &amp; RL</code> \n<a href=\"https://catalyst-team.github.io/catalyst\">Catalyst</a>\nHuge thanks to Albumentations:\n<a href=\"https://github.com/albu/albumentations\">Albumentations</a>\nAnd kudos to pytorch-toolbelt, for TTA on GPU:\n<a href=\"https://github.com/BloodAxe/pytorch-toolbelt\">PyTorch-toolbelt</a></p>\n\n<h3>Main points are …</h3>\n\nPreprocessing\n\n<ul>\n<li>Concat 6 channels</li>\n<li>Controls from train and test added as additional classes</li>\n<li>Treat each sirna sit separately for loading</li>\n<li>Normalize images by experiment/plate/channel</li>\n<li>Augmentations from albumentation - filp, rotate, transpose, cutout holes, shiftscale rotate</li>\n<li>Apply mixup/cutout by batch (big help)</li>\n<li>5-fold CV</li>\n<li>Built 256x256 and 512x512 based models</li>\n</ul>\n\nModelling\n\n<ul>\n<li>Denesent121 was workhorse, also used EfficientNet-B1, SE-Restnet101 and Densenet169</li>\n<li>Cosine LR policy and sawtooth policy worked pretty well, incl. Ralamb + Lookahead.</li>\n<li>Apex mixed precision helped speed up training</li>\n<li>Pseudo labels for test</li>\n<li>Label smoothing of ~ 0.1</li>\n<li>Continue training per individual experiment group (only worked without pseudo)</li>\n</ul>\n\nPost processing\n\n<ul>\n<li>No arcface - simply average train LOGITs on experiment level, and get cosine similarity of test LOGITs to nearest sirna</li>\n<li>TTA -  flip, rotate, transpose</li>\n<li>Apply Plate Leak</li>\n<li>Prediction balancing from Doodle @Pavel <a href=\"https://github.com/PavelOstyakov/predictions_balancing\">https://github.com/PavelOstyakov/predictions_balancing</a></li>\n</ul>",
  "messages": [
    {
      "id": "635522",
      "postDate": "09/27/2019 18:51:02",
      "content": "<p>Congrats to all on a great competition, and especially to top3. Please do share the NeurIPS content you present. </p>\n\n<p>Shout out to Catalyst team <code>Reproducible and fast DL &amp; RL</code> \n<a href=\"https://catalyst-team.github.io/catalyst\">Catalyst</a>\nHuge thanks to Albumentations:\n<a href=\"https://github.com/albu/albumentations\">Albumentations</a>\nAnd kudos to pytorch-toolbelt, for TTA on GPU:\n<a href=\"https://github.com/BloodAxe/pytorch-toolbelt\">PyTorch-toolbelt</a></p>\n\n<h3>Main points are …</h3>\n\nPreprocessing\n\n<ul>\n<li>Concat 6 channels</li>\n<li>Controls from train and test added as additional classes</li>\n<li>Treat each sirna sit separately for loading</li>\n<li>Normalize images by experiment/plate/channel</li>\n<li>Augmentations from albumentation - filp, rotate, transpose, cutout holes, shiftscale rotate</li>\n<li>Apply mixup/cutout by batch (big help)</li>\n<li>5-fold CV</li>\n<li>Built 256x256 and 512x512 based models</li>\n</ul>\n\nModelling\n\n<ul>\n<li>Denesent121 was workhorse, also used EfficientNet-B1, SE-Restnet101 and Densenet169</li>\n<li>Cosine LR policy and sawtooth policy worked pretty well, incl. Ralamb + Lookahead.</li>\n<li>Apex mixed precision helped speed up training</li>\n<li>Pseudo labels for test</li>\n<li>Label smoothing of ~ 0.1</li>\n<li>Continue training per individual experiment group (only worked without pseudo)</li>\n</ul>\n\nPost processing\n\n<ul>\n<li>No arcface - simply average train LOGITs on experiment level, and get cosine similarity of test LOGITs to nearest sirna</li>\n<li>TTA -  flip, rotate, transpose</li>\n<li>Apply Plate Leak</li>\n<li>Prediction balancing from Doodle @Pavel <a href=\"https://github.com/PavelOstyakov/predictions_balancing\">https://github.com/PavelOstyakov/predictions_balancing</a></li>\n</ul>",
      "rawMarkdown": "Congrats to all on a great competition, and especially to top3. Please do share the NeurIPS content you present. \n\nShout out to Catalyst team `Reproducible and fast DL &amp; RL` \n[Catalyst](https://catalyst-team.github.io/catalyst)\nHuge thanks to Albumentations:\n[Albumentations](https://github.com/albu/albumentations)\nAnd kudos to pytorch-toolbelt, for TTA on GPU:\n[PyTorch-toolbelt](https://github.com/BloodAxe/pytorch-toolbelt)\n\n### Main points are … \n#### Preprocessing\n- Concat 6 channels\n- Controls from train and test added as additional classes\n- Treat each sirna sit separately for loading\n- Normalize images by experiment/plate/channel\n- Augmentations from albumentation - filp, rotate, transpose, cutout holes, shiftscale rotate\n- Apply mixup/cutout by batch (big help)\n- 5-fold CV\n- Built 256x256 and 512x512 based models\n\n#### Modelling\n- Denesent121 was workhorse, also used EfficientNet-B1, SE-Restnet101 and Densenet169\n- Cosine LR policy and sawtooth policy worked pretty well, incl. Ralamb + Lookahead.\n- Apex mixed precision helped speed up training\n- Pseudo labels for test\n- Label smoothing of ~ 0.1\n- Continue training per individual experiment group (only worked without pseudo)\n\n#### Post processing\n- No arcface - simply average train LOGITs on experiment level, and get cosine similarity of test LOGITs to nearest sirna\n- TTA -  flip, rotate, transpose\n- Apply Plate Leak\n- Prediction balancing from Doodle @Pavel https://github.com/PavelOstyakov/predictions_balancing",
      "votes": null
    },
    {
      "id": "635529",
      "postDate": "09/27/2019 19:03:53",
      "content": "<p>Nice. Would help if you can share the code as well. </p>",
      "rawMarkdown": "Nice. Would help if you can share the code as well.",
      "votes": null
    },
    {
      "id": "635530",
      "postDate": "09/27/2019 19:05:07",
      "content": "<p>Code will be shared later in this discussion thread.</p>",
      "rawMarkdown": "Code will be shared later in this discussion thread.",
      "votes": null
    },
    {
      "id": "635552",
      "postDate": "09/27/2019 19:57:55",
      "content": "<p>Congratulations! Thank you for sharing the details!</p>\n\n<p>Have you tried larger EfficientNet models? If so, why wouldn't they work better than B1 on this dataset?</p>\n\n<p>Curious to see the cosine LR policy, pseudo-labelling and plate leak code!</p>",
      "rawMarkdown": "Congratulations! Thank you for sharing the details!\n\nHave you tried larger EfficientNet models? If so, why wouldn't they work better than B1 on this dataset?\n\nCurious to see the cosine LR policy, pseudo-labelling and plate leak code!",
      "votes": null
    },
    {
      "id": "635588",
      "postDate": "09/27/2019 22:01:53",
      "content": "<p>We have not tried larger models in general, since they seemed to have too much capacity. DenseNet169 seemed to work better than DenseNet121 initially, but DenseNet201 was not performing as well. Later on, when we started using pseudolables, DenseNet121 ended up being better than DenseNet169, so we figured that in general models with more capacity would not be too helpful. </p>",
      "rawMarkdown": "We have not tried larger models in general, since they seemed to have too much capacity. DenseNet169 seemed to work better than DenseNet121 initially, but DenseNet201 was not performing as well. Later on, when we started using pseudolables, DenseNet121 ended up being better than DenseNet169, so we figured that in general models with more capacity would not be too helpful.",
      "votes": null
    },
    {
      "id": "635594",
      "postDate": "09/27/2019 22:19:45",
      "content": "<p>Interesting! Thanks for clearing it up!</p>",
      "rawMarkdown": "Interesting! Thanks for clearing it up!",
      "votes": null
    },
    {
      "id": "635789",
      "postDate": "09/28/2019 08:01:28",
      "content": "<p>Congratulations\nGreat Write-Up\nThanks for Sharing your Approach &amp; Insights…!! <a href=\"/dmitrylarko\">@dmitrylarko</a> </p>",
      "rawMarkdown": "Congratulations\nGreat Write-Up\nThanks for Sharing your Approach &amp; Insights…!! @dmitrylarko",
      "votes": null
    },
    {
      "id": "636203",
      "postDate": "09/29/2019 02:51:26",
      "content": "<p>Congrats</p>",
      "rawMarkdown": "Congrats",
      "votes": null
    },
    {
      "id": "643004",
      "postDate": "10/06/2019 23:24:59",
      "content": "<p>Congrats, thank you for sharing.</p>",
      "rawMarkdown": "Congrats, thank you for sharing.",
      "votes": null
    },
    {
      "id": "652017",
      "postDate": "10/18/2019 08:25:37",
      "content": "<p>any plans to share your code solutions especially on how to utilize those useful packages (catalyst, PyTorch-toolbelt, etc)?</p>",
      "rawMarkdown": "any plans to share your code solutions especially on how to utilize those useful packages (catalyst, PyTorch-toolbelt, etc)?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 635529,
      "author_name": "akashram",
      "author_url": "",
      "post_date": "09/27/2019 19:03:53",
      "content": "<p>Nice. Would help if you can share the code as well. </p>",
      "votes": null,
      "replies": [
        {
          "id": 635530,
          "author_name": "dmitrylarko",
          "author_url": "",
          "post_date": "09/27/2019 19:05:07",
          "content": "<p>Code will be shared later in this discussion thread.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 652017,
          "author_name": "projdev",
          "author_url": "",
          "post_date": "10/18/2019 08:25:37",
          "content": "<p>any plans to share your code solutions especially on how to utilize those useful packages (catalyst, PyTorch-toolbelt, etc)?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 635552,
      "author_name": "carlolepelaars",
      "author_url": "",
      "post_date": "09/27/2019 19:57:55",
      "content": "<p>Congratulations! Thank you for sharing the details!</p>\n\n<p>Have you tried larger EfficientNet models? If so, why wouldn't they work better than B1 on this dataset?</p>\n\n<p>Curious to see the cosine LR policy, pseudo-labelling and plate leak code!</p>",
      "votes": null,
      "replies": [
        {
          "id": 635588,
          "author_name": "tunguz",
          "author_url": "",
          "post_date": "09/27/2019 22:01:53",
          "content": "<p>We have not tried larger models in general, since they seemed to have too much capacity. DenseNet169 seemed to work better than DenseNet121 initially, but DenseNet201 was not performing as well. Later on, when we started using pseudolables, DenseNet121 ended up being better than DenseNet169, so we figured that in general models with more capacity would not be too helpful. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 635594,
          "author_name": "carlolepelaars",
          "author_url": "",
          "post_date": "09/27/2019 22:19:45",
          "content": "<p>Interesting! Thanks for clearing it up!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 635789,
      "author_name": "veeralakrishna",
      "author_url": "",
      "post_date": "09/28/2019 08:01:28",
      "content": "<p>Congratulations\nGreat Write-Up\nThanks for Sharing your Approach &amp; Insights…!! <a href=\"/dmitrylarko\">@dmitrylarko</a> </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 636203,
      "author_name": "schiffer98",
      "author_url": "",
      "post_date": "09/29/2019 02:51:26",
      "content": "<p>Congrats</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 643004,
      "author_name": "corochann",
      "author_url": "",
      "post_date": "10/06/2019 23:24:59",
      "content": "<p>Congrats, thank you for sharing.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "635522": "Congrats to all on a great competition, and especially to top3. Please do share the NeurIPS content you present. \n\nShout out to Catalyst team `Reproducible and fast DL &amp; RL` \n[Catalyst](https://catalyst-team.github.io/catalyst)\nHuge thanks to Albumentations:\n[Albumentations](https://github.com/albu/albumentations)\nAnd kudos to pytorch-toolbelt, for TTA on GPU:\n[PyTorch-toolbelt](https://github.com/BloodAxe/pytorch-toolbelt)\n\n### Main points are … \n#### Preprocessing\n- Concat 6 channels\n- Controls from train and test added as additional classes\n- Treat each sirna sit separately for loading\n- Normalize images by experiment/plate/channel\n- Augmentations from albumentation - filp, rotate, transpose, cutout holes, shiftscale rotate\n- Apply mixup/cutout by batch (big help)\n- 5-fold CV\n- Built 256x256 and 512x512 based models\n\n#### Modelling\n- Denesent121 was workhorse, also used EfficientNet-B1, SE-Restnet101 and Densenet169\n- Cosine LR policy and sawtooth policy worked pretty well, incl. Ralamb + Lookahead.\n- Apex mixed precision helped speed up training\n- Pseudo labels for test\n- Label smoothing of ~ 0.1\n- Continue training per individual experiment group (only worked without pseudo)\n\n#### Post processing\n- No arcface - simply average train LOGITs on experiment level, and get cosine similarity of test LOGITs to nearest sirna\n- TTA -  flip, rotate, transpose\n- Apply Plate Leak\n- Prediction balancing from Doodle @Pavel https://github.com/PavelOstyakov/predictions_balancing",
    "635529": "Nice. Would help if you can share the code as well.",
    "635530": "Code will be shared later in this discussion thread.",
    "635552": "Congratulations! Thank you for sharing the details!\n\nHave you tried larger EfficientNet models? If so, why wouldn't they work better than B1 on this dataset?\n\nCurious to see the cosine LR policy, pseudo-labelling and plate leak code!",
    "635588": "We have not tried larger models in general, since they seemed to have too much capacity. DenseNet169 seemed to work better than DenseNet121 initially, but DenseNet201 was not performing as well. Later on, when we started using pseudolables, DenseNet121 ended up being better than DenseNet169, so we figured that in general models with more capacity would not be too helpful.",
    "635594": "Interesting! Thanks for clearing it up!",
    "635789": "Congratulations\nGreat Write-Up\nThanks for Sharing your Approach &amp; Insights…!! @dmitrylarko",
    "636203": "Congrats",
    "643004": "Congrats, thank you for sharing.",
    "652017": "any plans to share your code solutions especially on how to utilize those useful packages (catalyst, PyTorch-toolbelt, etc)?"
  },
  "source": "meta"
}