{
  "id": 108057,
  "title": "76nd place solution with code",
  "url": "/competitions/aptos2019-blindness-detection/discussion/108057",
  "author_name": "Insaf Ashrapov",
  "post_date": "2019-09-08T19:07:07.050000",
  "votes": 12,
  "comment_count": 2,
  "views": 0,
  "content": "<p>### Team</p>\n\n<p>Mamat Shamshiev (<a href=\"/mamatml\">@mamatml</a>), Insaf Ashrapov (<a href=\"/insaff\">@insaff</a>), Mishunyayev Nikita (<a href=\"/mnikita\">@mnikita</a>)</p>\n\n<h3>Data</h3>\n\n<p>2015 competition data was used for pretraining all our models. Without it out models performed much worse. We used different techniques: first train on old data, then finetuning on the new train, another technique train on both data, the finetune on new train data. Besides, starting finetuning with freezing all layers and training only last FC layer gave us more stable results.</p>\n\n<h3>Models and Preprocessing</h3>\n\n<p>From the beginning, efficientnet outperformed other models. Using fp16 (available in kaggle kernels) allowed to use bigger batch size - speeded up training and inference.\nModels used in the final submission:</p>\n\n<p>1) EfficientNet-B5 (best single model): 224x224 (tta with Hflip, preprocessing - crop_from_gray, circle_crop, ben_preprocess=10)\n2) EfficientNet-B4: 256x256 (tta with Hflip, preprocessing - crop_from_gray, circle_crop, ben_preprocess=20)\n3) EfficientNet-B5: 256x256 (tta with Hflip, preprocessing - crop_from_gray, circle_crop, ben_preprocess=30)\n4) EfficientNet-B5: (256x256) without specific preprocess, two models with different augmentations. </p>\n\n<p>We tried bigger image sizes but it gave worse results. EfficientNet-B2, B3, B6 gave worse results as well or didn't improve our blend.</p>\n\n<h3>Augmentations</h3>\n\n<p>From <a href=\"https://github.com/albu/albumentations\">Albumentations </a>library:\n<em>Hflip, VFlip,  RandomScale, CenterCrop,  RandomBrightnessContrast, ShiftScaleRotate, RandomGamma, RandomGamma, JpegCompression, HueSaturationValue, RGBShift, ChannelShuffle, ToGray, Cutout</em></p>\n\n<h3>Training</h3>\n\n<ul>\n<li>First 3 models were trained using <a href=\"https://github.com/catalyst-team/catalyst\">Catalyst</a> library and the last one with FastAi, both of them work on top of Pytorch.</li>\n<li>We used both ordinal regression and regression. Models with classification tasks weren't well enough to use them.</li>\n<li>Adam with OneCycle was used for training. WarmUp helped to get more stable results. RAdam, Label smoothing didn't help to improve the score.</li>\n<li>We tried to use leak investigated <a href=\"https://www.kaggle.com/miklgr500/leakage-detection-about-8-test-dataset\">here </a> and <a href=\"https://www.kaggle.com/konradb/adversarial-validation-quick-fast-ai-approach\">here</a> by fixing output results. Almost 10% of the public test data were part of the train. Results dropped significantly, which means training data annotation were pretty bad.</li>\n<li>We tried kappa coefficient optimization, it didn't give reliable improvement on public, but could help us on private almost +0.003 score.</li>\n</ul>\n\n<h3>Hardware</h3>\n\n<p>We used 1x*2080, 1x* Tesla v40, 1x*1070ti, kaggle kernels</p>\n\n<h3>Ensembling</h3>\n\n<p>Weighted average based on public LB gave .823 on the private LB.\nUsing more models with different image size, augmentation gave a lower result on public LB, but could push us above 40 places. Unfortunately, we didn't pick it as two final submissions.</p>\n\n<h3>Link to solution</h3>\n\n<p><a href=\"https://github.com/MamatShamshiev/Kaggle-APTOS-2019-Blindness-Detection\">https://github.com/MamatShamshiev/Kaggle-APTOS-2019-Blindness-Detection</a>\n<a href=\"https://github.com/Mishunyayev-Nikita/Kaggle-Aptos-Blindness-Detection-2019-solution\">https://github.com/Mishunyayev-Nikita/Kaggle-Aptos-Blindness-Detection-2019-solution</a></p>",
  "messages": [
    {
      "id": 621632,
      "postDate": "2019-09-08T19:07:07.050Z",
      "content": "<p>### Team</p>\n\n<p>Mamat Shamshiev (<a href=\"/mamatml\">@mamatml</a>), Insaf Ashrapov (<a href=\"/insaff\">@insaff</a>), Mishunyayev Nikita (<a href=\"/mnikita\">@mnikita</a>)</p>\n\n<h3>Data</h3>\n\n<p>2015 competition data was used for pretraining all our models. Without it out models performed much worse. We used different techniques: first train on old data, then finetuning on the new train, another technique train on both data, the finetune on new train data. Besides, starting finetuning with freezing all layers and training only last FC layer gave us more stable results.</p>\n\n<h3>Models and Preprocessing</h3>\n\n<p>From the beginning, efficientnet outperformed other models. Using fp16 (available in kaggle kernels) allowed to use bigger batch size - speeded up training and inference.\nModels used in the final submission:</p>\n\n<p>1) EfficientNet-B5 (best single model): 224x224 (tta with Hflip, preprocessing - crop_from_gray, circle_crop, ben_preprocess=10)\n2) EfficientNet-B4: 256x256 (tta with Hflip, preprocessing - crop_from_gray, circle_crop, ben_preprocess=20)\n3) EfficientNet-B5: 256x256 (tta with Hflip, preprocessing - crop_from_gray, circle_crop, ben_preprocess=30)\n4) EfficientNet-B5: (256x256) without specific preprocess, two models with different augmentations. </p>\n\n<p>We tried bigger image sizes but it gave worse results. EfficientNet-B2, B3, B6 gave worse results as well or didn't improve our blend.</p>\n\n<h3>Augmentations</h3>\n\n<p>From <a href=\"https://github.com/albu/albumentations\">Albumentations </a>library:\n<em>Hflip, VFlip,  RandomScale, CenterCrop,  RandomBrightnessContrast, ShiftScaleRotate, RandomGamma, RandomGamma, JpegCompression, HueSaturationValue, RGBShift, ChannelShuffle, ToGray, Cutout</em></p>\n\n<h3>Training</h3>\n\n<ul>\n<li>First 3 models were trained using <a href=\"https://github.com/catalyst-team/catalyst\">Catalyst</a> library and the last one with FastAi, both of them work on top of Pytorch.</li>\n<li>We used both ordinal regression and regression. Models with classification tasks weren't well enough to use them.</li>\n<li>Adam with OneCycle was used for training. WarmUp helped to get more stable results. RAdam, Label smoothing didn't help to improve the score.</li>\n<li>We tried to use leak investigated <a href=\"https://www.kaggle.com/miklgr500/leakage-detection-about-8-test-dataset\">here </a> and <a href=\"https://www.kaggle.com/konradb/adversarial-validation-quick-fast-ai-approach\">here</a> by fixing output results. Almost 10% of the public test data were part of the train. Results dropped significantly, which means training data annotation were pretty bad.</li>\n<li>We tried kappa coefficient optimization, it didn't give reliable improvement on public, but could help us on private almost +0.003 score.</li>\n</ul>\n\n<h3>Hardware</h3>\n\n<p>We used 1x*2080, 1x* Tesla v40, 1x*1070ti, kaggle kernels</p>\n\n<h3>Ensembling</h3>\n\n<p>Weighted average based on public LB gave .823 on the private LB.\nUsing more models with different image size, augmentation gave a lower result on public LB, but could push us above 40 places. Unfortunately, we didn't pick it as two final submissions.</p>\n\n<h3>Link to solution</h3>\n\n<p><a href=\"https://github.com/MamatShamshiev/Kaggle-APTOS-2019-Blindness-Detection\">https://github.com/MamatShamshiev/Kaggle-APTOS-2019-Blindness-Detection</a>\n<a href=\"https://github.com/Mishunyayev-Nikita/Kaggle-Aptos-Blindness-Detection-2019-solution\">https://github.com/Mishunyayev-Nikita/Kaggle-Aptos-Blindness-Detection-2019-solution</a></p>",
      "rawMarkdown": " ### Team\n\nMamat Shamshiev (@mamatml), Insaf Ashrapov (@insaff), Mishunyayev Nikita (@mnikita)\n\n### Data\n2015 competition data was used for pretraining all our models. Without it out models performed much worse. We used different techniques: first train on old data, then finetuning on the new train, another technique train on both data, the finetune on new train data. Besides, starting finetuning with freezing all layers and training only last FC layer gave us more stable results.\n\n### Models and Preprocessing\nFrom the beginning, efficientnet outperformed other models. Using fp16 (available in kaggle kernels) allowed to use bigger batch size - speeded up training and inference.\nModels used in the final submission:\n\n1) EfficientNet-B5 (best single model): 224x224 (tta with Hflip, preprocessing - crop_from_gray, circle_crop, ben_preprocess=10)\n2) EfficientNet-B4: 256x256 (tta with Hflip, preprocessing - crop_from_gray, circle_crop, ben_preprocess=20)\n3) EfficientNet-B5: 256x256 (tta with Hflip, preprocessing - crop_from_gray, circle_crop, ben_preprocess=30)\n4) EfficientNet-B5: (256x256) without specific preprocess, two models with different augmentations. \n\nWe tried bigger image sizes but it gave worse results. EfficientNet-B2, B3, B6 gave worse results as well or didn't improve our blend.\n\n### Augmentations\nFrom [Albumentations ](https://github.com/albu/albumentations)library:\n*Hflip, VFlip,  RandomScale, CenterCrop,  RandomBrightnessContrast, ShiftScaleRotate, RandomGamma, RandomGamma, JpegCompression, HueSaturationValue, RGBShift, ChannelShuffle, ToGray, Cutout*\n\n### Training\n* First 3 models were trained using [Catalyst](https://github.com/catalyst-team/catalyst) library and the last one with FastAi, both of them work on top of Pytorch.\n* We used both ordinal regression and regression. Models with classification tasks weren't well enough to use them.\n* Adam with OneCycle was used for training. WarmUp helped to get more stable results. RAdam, Label smoothing didn't help to improve the score.\n* We tried to use leak investigated [here ](https://www.kaggle.com/miklgr500/leakage-detection-about-8-test-dataset) and [here](https://www.kaggle.com/konradb/adversarial-validation-quick-fast-ai-approach) by fixing output results. Almost 10% of the public test data were part of the train. Results dropped significantly, which means training data annotation were pretty bad.\n* We tried kappa coefficient optimization, it didn't give reliable improvement on public, but could help us on private almost +0.003 score.\n\n### Hardware\nWe used 1x*2080, 1x* Tesla v40, 1x*1070ti, kaggle kernels\n\n### Ensembling\nWeighted average based on public LB gave .823 on the private LB.\nUsing more models with different image size, augmentation gave a lower result on public LB, but could push us above 40 places. Unfortunately, we didn't pick it as two final submissions.\n\n### Link to solution \nhttps://github.com/MamatShamshiev/Kaggle-APTOS-2019-Blindness-Detection\nhttps://github.com/Mishunyayev-Nikita/Kaggle-Aptos-Blindness-Detection-2019-solution",
      "votes": 12
    },
    {
      "id": 622319,
      "postDate": "2019-09-09T13:54:56.740Z",
      "content": "<p>Congratulations.\nI loved your code, simple organized and effective. </p>",
      "rawMarkdown": "Congratulations.\nI loved your code, simple organized and effective. "
    },
    {
      "id": 621899,
      "postDate": "2019-09-09T04:43:25.683Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 622319,
      "author_name": "JM100",
      "author_url": "",
      "post_date": "2019-09-09T13:54:56.740000",
      "content": "<p>Congratulations.\nI loved your code, simple organized and effective. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 621899,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-09-09T04:43:25.683000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "621632": " ### Team\n\nMamat Shamshiev (@mamatml), Insaf Ashrapov (@insaff), Mishunyayev Nikita (@mnikita)\n\n### Data\n2015 competition data was used for pretraining all our models. Without it out models performed much worse. We used different techniques: first train on old data, then finetuning on the new train, another technique train on both data, the finetune on new train data. Besides, starting finetuning with freezing all layers and training only last FC layer gave us more stable results.\n\n### Models and Preprocessing\nFrom the beginning, efficientnet outperformed other models. Using fp16 (available in kaggle kernels) allowed to use bigger batch size - speeded up training and inference.\nModels used in the final submission:\n\n1) EfficientNet-B5 (best single model): 224x224 (tta with Hflip, preprocessing - crop_from_gray, circle_crop, ben_preprocess=10)\n2) EfficientNet-B4: 256x256 (tta with Hflip, preprocessing - crop_from_gray, circle_crop, ben_preprocess=20)\n3) EfficientNet-B5: 256x256 (tta with Hflip, preprocessing - crop_from_gray, circle_crop, ben_preprocess=30)\n4) EfficientNet-B5: (256x256) without specific preprocess, two models with different augmentations. \n\nWe tried bigger image sizes but it gave worse results. EfficientNet-B2, B3, B6 gave worse results as well or didn't improve our blend.\n\n### Augmentations\nFrom [Albumentations ](https://github.com/albu/albumentations)library:\n*Hflip, VFlip,  RandomScale, CenterCrop,  RandomBrightnessContrast, ShiftScaleRotate, RandomGamma, RandomGamma, JpegCompression, HueSaturationValue, RGBShift, ChannelShuffle, ToGray, Cutout*\n\n### Training\n* First 3 models were trained using [Catalyst](https://github.com/catalyst-team/catalyst) library and the last one with FastAi, both of them work on top of Pytorch.\n* We used both ordinal regression and regression. Models with classification tasks weren't well enough to use them.\n* Adam with OneCycle was used for training. WarmUp helped to get more stable results. RAdam, Label smoothing didn't help to improve the score.\n* We tried to use leak investigated [here ](https://www.kaggle.com/miklgr500/leakage-detection-about-8-test-dataset) and [here](https://www.kaggle.com/konradb/adversarial-validation-quick-fast-ai-approach) by fixing output results. Almost 10% of the public test data were part of the train. Results dropped significantly, which means training data annotation were pretty bad.\n* We tried kappa coefficient optimization, it didn't give reliable improvement on public, but could help us on private almost +0.003 score.\n\n### Hardware\nWe used 1x*2080, 1x* Tesla v40, 1x*1070ti, kaggle kernels\n\n### Ensembling\nWeighted average based on public LB gave .823 on the private LB.\nUsing more models with different image size, augmentation gave a lower result on public LB, but could push us above 40 places. Unfortunately, we didn't pick it as two final submissions.\n\n### Link to solution \nhttps://github.com/MamatShamshiev/Kaggle-APTOS-2019-Blindness-Detection\nhttps://github.com/Mishunyayev-Nikita/Kaggle-Aptos-Blindness-Detection-2019-solution",
    "622319": "Congratulations.\nI loved your code, simple organized and effective. ",
    "621899": ""
  }
}