{
  "id": 107996,
  "title": "142nd place solution",
  "url": "/competitions/aptos2019-blindness-detection/discussion/107996",
  "author_name": "NAIN",
  "post_date": "2019-09-08T10:58:17.124000",
  "votes": 17,
  "comment_count": 3,
  "views": 0,
  "content": "<p>First of all, congratulations to all the winners. I am gonna describe all the things that we tried and what worked for us and what didn't. </p>\n\n<p><code>What didn't work in our case?</code>\n1.  We experimented with a lot of different models. Starting from ResNet-50 to get a baseline, we tried all variants of ResNets including ResNeXt but none of them gave us a good score on the leaderboard.\n2. We approached the problem from both ends. I tried solving it as a classification problem while <a href=\"https://www.kaggle.com/konradb\">Konrad</a> and <a href=\"https://www.kaggle.com/abhishek\">Abhishek</a> approached it as a regression problem. After a while, I realized that treating it as a classification problem isn't providing a boost to the score.\n3. Ben's preprocessing: In fact, all models where we used this resulted in a very low score.\n4. Label smoothing didn't work very well for classification models</p>\n\n<p><code>What worked for us?</code>\n1. For preprocessing, we found out that cropping out the black area in the original images is good enough.\n2. Heavy augmentation: Random horizontal and vertical flips weren't enough.\n3. <code>EfficientNets</code>, especially <code>B4</code> and <code>B5</code>, proved more effective than anything else. We ended up using <code>B4</code> only\n4. We used both <code>old competition</code> data as well as <code>new competition</code> data but not blindly. Both the datasets have huge <code>class imbalance</code> with classes <code>0</code> and <code>2</code> as majority classes. We tried to balance the classes as much as possible. </p>\n\n<p><strong>One crazy idea</strong>\nFor the last attempt, we tried a crazy idea regarding <code>sampling</code> of the classes from the old data. We were positive that it would work but we weren't very sure at any point in time.  We randomly sampled the old dataset in a fashion such that the class distribution for old dataset looked like this:\n <code>\n3    2087\n4    1914\n2    1000\n1    1000\n0    1000\n</code>\nWe didn't want to throw away any data point in the new dataset. So, we <code>undersampled</code> the classes that were in <code>majority</code> in the new dataset. Doing this would induce a bias in the model during <code>pretraining</code> phase towards the classes <code>3</code> and <code>4</code>. Once we switch to <code>fine-tuning</code> the model on the new dataset, the distribution is reversed now and the model should be able to correct the bias. With this distribution and heavy augmentation, our best model, <code>EfficientNet-B4</code>, with 5-folds TTA scored <code>0.920</code> on the final leaderboard. </p>\n\n<p><code>What we didn't try?</code>\n1. We didn't try pseudo labelling. \n2. We didn't ensemble our models, especially all variants of <code>EfficientNets</code>.</p>\n\n<p>We didn't try these things because during the last month because we all three were having too much work on our end. I was diagnosed with <code>dengue</code> and couldn't contribute fully in between. Nevertheless, we are happy that with a <code>single model</code> we made a jump of <strong>655</strong> positions on the leaderboard and didn't end badly. Thanks to my amazing teammates @konradb and @abhishek </p>",
  "messages": [
    {
      "id": 621230,
      "postDate": "2019-09-08T10:58:17.123Z",
      "content": "<p>First of all, congratulations to all the winners. I am gonna describe all the things that we tried and what worked for us and what didn't. </p>\n\n<p><code>What didn't work in our case?</code>\n1.  We experimented with a lot of different models. Starting from ResNet-50 to get a baseline, we tried all variants of ResNets including ResNeXt but none of them gave us a good score on the leaderboard.\n2. We approached the problem from both ends. I tried solving it as a classification problem while <a href=\"https://www.kaggle.com/konradb\">Konrad</a> and <a href=\"https://www.kaggle.com/abhishek\">Abhishek</a> approached it as a regression problem. After a while, I realized that treating it as a classification problem isn't providing a boost to the score.\n3. Ben's preprocessing: In fact, all models where we used this resulted in a very low score.\n4. Label smoothing didn't work very well for classification models</p>\n\n<p><code>What worked for us?</code>\n1. For preprocessing, we found out that cropping out the black area in the original images is good enough.\n2. Heavy augmentation: Random horizontal and vertical flips weren't enough.\n3. <code>EfficientNets</code>, especially <code>B4</code> and <code>B5</code>, proved more effective than anything else. We ended up using <code>B4</code> only\n4. We used both <code>old competition</code> data as well as <code>new competition</code> data but not blindly. Both the datasets have huge <code>class imbalance</code> with classes <code>0</code> and <code>2</code> as majority classes. We tried to balance the classes as much as possible. </p>\n\n<p><strong>One crazy idea</strong>\nFor the last attempt, we tried a crazy idea regarding <code>sampling</code> of the classes from the old data. We were positive that it would work but we weren't very sure at any point in time.  We randomly sampled the old dataset in a fashion such that the class distribution for old dataset looked like this:\n <code>\n3    2087\n4    1914\n2    1000\n1    1000\n0    1000\n</code>\nWe didn't want to throw away any data point in the new dataset. So, we <code>undersampled</code> the classes that were in <code>majority</code> in the new dataset. Doing this would induce a bias in the model during <code>pretraining</code> phase towards the classes <code>3</code> and <code>4</code>. Once we switch to <code>fine-tuning</code> the model on the new dataset, the distribution is reversed now and the model should be able to correct the bias. With this distribution and heavy augmentation, our best model, <code>EfficientNet-B4</code>, with 5-folds TTA scored <code>0.920</code> on the final leaderboard. </p>\n\n<p><code>What we didn't try?</code>\n1. We didn't try pseudo labelling. \n2. We didn't ensemble our models, especially all variants of <code>EfficientNets</code>.</p>\n\n<p>We didn't try these things because during the last month because we all three were having too much work on our end. I was diagnosed with <code>dengue</code> and couldn't contribute fully in between. Nevertheless, we are happy that with a <code>single model</code> we made a jump of <strong>655</strong> positions on the leaderboard and didn't end badly. Thanks to my amazing teammates @konradb and @abhishek </p>",
      "rawMarkdown": "First of all, congratulations to all the winners. I am gonna describe all the things that we tried and what worked for us and what didn't. \n\n`What didn't work in our case?`\n1.  We experimented with a lot of different models. Starting from ResNet-50 to get a baseline, we tried all variants of ResNets including ResNeXt but none of them gave us a good score on the leaderboard.\n2. We approached the problem from both ends. I tried solving it as a classification problem while [Konrad](https://www.kaggle.com/konradb) and [Abhishek](https://www.kaggle.com/abhishek) approached it as a regression problem. After a while, I realized that treating it as a classification problem isn't providing a boost to the score.\n3. Ben's preprocessing: In fact, all models where we used this resulted in a very low score.\n4. Label smoothing didn't work very well for classification models\n\n`What worked for us?`\n1. For preprocessing, we found out that cropping out the black area in the original images is good enough.\n2. Heavy augmentation: Random horizontal and vertical flips weren't enough.\n3. `EfficientNets`, especially `B4` and `B5`, proved more effective than anything else. We ended up using `B4` only\n4. We used both `old competition` data as well as `new competition` data but not blindly. Both the datasets have huge `class imbalance` with classes `0` and `2` as majority classes. We tried to balance the classes as much as possible. \n\n**One crazy idea**\nFor the last attempt, we tried a crazy idea regarding `sampling` of the classes from the old data. We were positive that it would work but we weren't very sure at any point in time.  We randomly sampled the old dataset in a fashion such that the class distribution for old dataset looked like this:\n ```\n3    2087\n4    1914\n2    1000\n1    1000\n0    1000\n```\nWe didn't want to throw away any data point in the new dataset. So, we `undersampled` the classes that were in `majority` in the new dataset. Doing this would induce a bias in the model during `pretraining` phase towards the classes `3` and `4`. Once we switch to `fine-tuning` the model on the new dataset, the distribution is reversed now and the model should be able to correct the bias. With this distribution and heavy augmentation, our best model, `EfficientNet-B4`, with 5-folds TTA scored `0.920` on the final leaderboard. \n\n`What we didn't try?`\n1. We didn't try pseudo labelling. \n2. We didn't ensemble our models, especially all variants of `EfficientNets`.\n\n  \nWe didn't try these things because during the last month because we all three were having too much work on our end. I was diagnosed with `dengue` and couldn't contribute fully in between. Nevertheless, we are happy that with a `single model` we made a jump of **655** positions on the leaderboard and didn't end badly. Thanks to my amazing teammates @konradb and @abhishek ",
      "votes": 17
    },
    {
      "id": 624347,
      "postDate": "2019-09-12T02:15:39.740Z",
      "content": "<p>Congrats on your medal guys, well deserved. <a href=\"/aakashnain\">@aakashnain</a> , hope you recovered completely💪   </p>",
      "rawMarkdown": "Congrats on your medal guys, well deserved. @aakashnain , hope you recovered completely💪   "
    },
    {
      "id": 621942,
      "postDate": "2019-09-09T05:35:48.863Z",
      "rawMarkdown": "",
      "isDeleted": true,
      "replies": [
        {
          "id": 622001,
          "postDate": "2019-09-09T06:38:31.350Z",
          "content": "<p>Thank you.</p>",
          "rawMarkdown": "Thank you."
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 624347,
      "author_name": "Abhinand",
      "author_url": "",
      "post_date": "2019-09-12T02:15:39.740000",
      "content": "<p>Congrats on your medal guys, well deserved. <a href=\"/aakashnain\">@aakashnain</a> , hope you recovered completely💪   </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 621942,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-09-09T05:35:48.863000",
      "content": "",
      "votes": 0,
      "replies": [
        {
          "id": 622001,
          "author_name": "NAIN",
          "author_url": "",
          "post_date": "2019-09-09T06:38:31.350000",
          "content": "<p>Thank you.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "621230": "First of all, congratulations to all the winners. I am gonna describe all the things that we tried and what worked for us and what didn't. \n\n`What didn't work in our case?`\n1.  We experimented with a lot of different models. Starting from ResNet-50 to get a baseline, we tried all variants of ResNets including ResNeXt but none of them gave us a good score on the leaderboard.\n2. We approached the problem from both ends. I tried solving it as a classification problem while [Konrad](https://www.kaggle.com/konradb) and [Abhishek](https://www.kaggle.com/abhishek) approached it as a regression problem. After a while, I realized that treating it as a classification problem isn't providing a boost to the score.\n3. Ben's preprocessing: In fact, all models where we used this resulted in a very low score.\n4. Label smoothing didn't work very well for classification models\n\n`What worked for us?`\n1. For preprocessing, we found out that cropping out the black area in the original images is good enough.\n2. Heavy augmentation: Random horizontal and vertical flips weren't enough.\n3. `EfficientNets`, especially `B4` and `B5`, proved more effective than anything else. We ended up using `B4` only\n4. We used both `old competition` data as well as `new competition` data but not blindly. Both the datasets have huge `class imbalance` with classes `0` and `2` as majority classes. We tried to balance the classes as much as possible. \n\n**One crazy idea**\nFor the last attempt, we tried a crazy idea regarding `sampling` of the classes from the old data. We were positive that it would work but we weren't very sure at any point in time.  We randomly sampled the old dataset in a fashion such that the class distribution for old dataset looked like this:\n ```\n3    2087\n4    1914\n2    1000\n1    1000\n0    1000\n```\nWe didn't want to throw away any data point in the new dataset. So, we `undersampled` the classes that were in `majority` in the new dataset. Doing this would induce a bias in the model during `pretraining` phase towards the classes `3` and `4`. Once we switch to `fine-tuning` the model on the new dataset, the distribution is reversed now and the model should be able to correct the bias. With this distribution and heavy augmentation, our best model, `EfficientNet-B4`, with 5-folds TTA scored `0.920` on the final leaderboard. \n\n`What we didn't try?`\n1. We didn't try pseudo labelling. \n2. We didn't ensemble our models, especially all variants of `EfficientNets`.\n\n  \nWe didn't try these things because during the last month because we all three were having too much work on our end. I was diagnosed with `dengue` and couldn't contribute fully in between. Nevertheless, we are happy that with a `single model` we made a jump of **655** positions on the leaderboard and didn't end badly. Thanks to my amazing teammates @konradb and @abhishek ",
    "624347": "Congrats on your medal guys, well deserved. @aakashnain , hope you recovered completely💪   ",
    "621942": ""
  }
}