{
  "id": 242275,
  "title": "[86.385] 10th Place Solution",
  "url": "/competitions/plant-pathology-2021-fgvc8/writeups/86-385-10th-place-solution",
  "author_name": "",
  "post_date": "2021-05-28T09:59:22.090Z",
  "votes": 27,
  "comment_count": 10,
  "views": 0,
  "content": "<p>First of all, thanks to the organizers and Kaggle team for this competition, and congratulations to all the top scorers. This was a great learning experience throughout, read some great papers and tried to implement some of them, and reached a satisfactory position. <br>\nI started from the beginning then took a break and rejoined towards the end. My solution is quite simple but here are some key points for those who can benefit from it!</p>\n<h1><strong>Approach :</strong></h1>\n<ul>\n<li><p><strong>Data:</strong></p>\n<ul>\n<li>I used 640x640 resized tfrecords of the competition data. <a href=\"https://www.kaggle.com/ashish2001/640x640-plant-pathology-tfrecords\" target=\"_blank\">link</a></li>\n<li>Duplicates and misleading images were <strong>, not</strong> removed 🙃</li></ul></li>\n<li><p><strong>Model:</strong></p>\n<ul>\n<li>I experimented with Effnets (B0-B7), EffnetV2s, SeResNext-50, MobileNets, ViTs etc. with different image sizes (640, 512, 224) etc. Out of these Effnet B5 (imagenet) was my best single model (<strong>Public: 84.2, Private: 86.38</strong>)  B6s and B7s were overfitting a lot!</li>\n<li>Noisy student weight performed worse throughout my experiments.</li>\n<li><strong>Best Model:</strong><ul>\n<li>EffnetB5(imagenet), all BatchNormalization layers were kept frozen, GlobalAveragePooling, followed by Dropout and Output layer of 6 sigmoidal nodes.</li></ul></li></ul></li>\n<li><p><strong>Loss Function</strong></p>\n<ul>\n<li>This played an important role in my pipeline, normal cross-entropy didn't work well possibly due to the fact that it inhibits learning in the early stages by penalizing more on the wrongly classified samples.</li>\n<li>Focal loss seemed to work well with consistent results throughout.</li>\n<li>Asymmetric loss taken from <a href=\"https://arxiv.org/pdf/2009.14119.pdf\" target=\"_blank\">here</a> worked well for increasing CV but couldn't improve LB (It could have made overfitting easy or maybe I translated it poorly from PyTorch to TensorFlow 😂).</li>\n<li>The most promising loss was to use a differentiable version of MacroF1 as described in this great <a href=\"https://towardsdatascience.com/the-unknown-benefits-of-using-a-soft-f1-loss-in-classification-systems-753902c0105d\" target=\"_blank\">article</a></li></ul></li>\n<li><p><strong>Augmentations</strong></p>\n<ul>\n<li>Random Shearing/ Rotations/ Flipping/ Saturation/ Contrast/ Brightness/ Gamma and Central Cropping were used thanks to <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> 's awesome <a href=\"https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\" target=\"_blank\">notebook</a></li></ul></li>\n<li><p><strong>Other Details</strong></p>\n<ul>\n<li>Used Adam optimizer.</li>\n<li>Cosine Decay-based Learning Rate.</li>\n<li>5-Fold cross-validation scheme.</li>\n<li>Batch Size = 32 per replica of TPU.</li></ul></li>\n<li><p><strong>Inference Scheme</strong></p>\n<ul>\n<li>Averaged outputs of all the 5 folds of a model at a threshold of 0.5.</li></ul></li>\n</ul>\n<p>In the end, I chose one ensemble-based submission [Effnet B5, MobileNetv3 large initialized with Crop net weights, SeResNext50 and others.](Public: 82.9, Private 85.313), didn't have much time to create more diverse ensembles and my best EffNet B5. </p>\n<ul>\n<li>I could've tweaked my inference pipeline to figure out the best threshold (I got an increase of  0.2 on private lb just by tweaking the threshold but I didn't select that as my final submission).</li>\n</ul>\n<p>I would put the link to my code after tidying it up a bit. </p>\n<p>Overall I enjoyed participating in the competition and would love to know how other's tackled this problem. If you have any suggestions on how I could've improved please comment on them!</p>\n<p>See you guys again in some other competition!😄</p>",
  "messages": [
    {
      "id": "1326198",
      "postDate": "05/28/2021 09:57:16",
      "content": "<p>First of all, thanks to the organizers and Kaggle team for this competition, and congratulations to all the top scorers. This was a great learning experience throughout, read some great papers and tried to implement some of them, and reached a satisfactory position. <br>\nI started from the beginning then took a break and rejoined towards the end. My solution is quite simple but here are some key points for those who can benefit from it!</p>\n<h1><strong>Approach :</strong></h1>\n<ul>\n<li><p><strong>Data:</strong></p>\n<ul>\n<li>I used 640x640 resized tfrecords of the competition data. <a href=\"https://www.kaggle.com/ashish2001/640x640-plant-pathology-tfrecords\" target=\"_blank\">link</a></li>\n<li>Duplicates and misleading images were <strong>, not</strong> removed 🙃</li></ul></li>\n<li><p><strong>Model:</strong></p>\n<ul>\n<li>I experimented with Effnets (B0-B7), EffnetV2s, SeResNext-50, MobileNets, ViTs etc. with different image sizes (640, 512, 224) etc. Out of these Effnet B5 (imagenet) was my best single model (<strong>Public: 84.2, Private: 86.38</strong>)  B6s and B7s were overfitting a lot!</li>\n<li>Noisy student weight performed worse throughout my experiments.</li>\n<li><strong>Best Model:</strong><ul>\n<li>EffnetB5(imagenet), all BatchNormalization layers were kept frozen, GlobalAveragePooling, followed by Dropout and Output layer of 6 sigmoidal nodes.</li></ul></li></ul></li>\n<li><p><strong>Loss Function</strong></p>\n<ul>\n<li>This played an important role in my pipeline, normal cross-entropy didn't work well possibly due to the fact that it inhibits learning in the early stages by penalizing more on the wrongly classified samples.</li>\n<li>Focal loss seemed to work well with consistent results throughout.</li>\n<li>Asymmetric loss taken from <a href=\"https://arxiv.org/pdf/2009.14119.pdf\" target=\"_blank\">here</a> worked well for increasing CV but couldn't improve LB (It could have made overfitting easy or maybe I translated it poorly from PyTorch to TensorFlow 😂).</li>\n<li>The most promising loss was to use a differentiable version of MacroF1 as described in this great <a href=\"https://towardsdatascience.com/the-unknown-benefits-of-using-a-soft-f1-loss-in-classification-systems-753902c0105d\" target=\"_blank\">article</a></li></ul></li>\n<li><p><strong>Augmentations</strong></p>\n<ul>\n<li>Random Shearing/ Rotations/ Flipping/ Saturation/ Contrast/ Brightness/ Gamma and Central Cropping were used thanks to <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> 's awesome <a href=\"https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\" target=\"_blank\">notebook</a></li></ul></li>\n<li><p><strong>Other Details</strong></p>\n<ul>\n<li>Used Adam optimizer.</li>\n<li>Cosine Decay-based Learning Rate.</li>\n<li>5-Fold cross-validation scheme.</li>\n<li>Batch Size = 32 per replica of TPU.</li></ul></li>\n<li><p><strong>Inference Scheme</strong></p>\n<ul>\n<li>Averaged outputs of all the 5 folds of a model at a threshold of 0.5.</li></ul></li>\n</ul>\n<p>In the end, I chose one ensemble-based submission [Effnet B5, MobileNetv3 large initialized with Crop net weights, SeResNext50 and others.](Public: 82.9, Private 85.313), didn't have much time to create more diverse ensembles and my best EffNet B5. </p>\n<ul>\n<li>I could've tweaked my inference pipeline to figure out the best threshold (I got an increase of  0.2 on private lb just by tweaking the threshold but I didn't select that as my final submission).</li>\n</ul>\n<p>I would put the link to my code after tidying it up a bit. </p>\n<p>Overall I enjoyed participating in the competition and would love to know how other's tackled this problem. If you have any suggestions on how I could've improved please comment on them!</p>\n<p>See you guys again in some other competition!😄</p>",
      "rawMarkdown": "First of all, thanks to the organizers and Kaggle team for this competition, and congratulations to all the top scorers. This was a great learning experience throughout, read some great papers and tried to implement some of them, and reached a satisfactory position. \nI started from the beginning then took a break and rejoined towards the end. My solution is quite simple but here are some key points for those who can benefit from it!\n\n# **Approach :**\n- **Data:**\n     - I used 640x640 resized tfrecords of the competition data. [link](https://www.kaggle.com/ashish2001/640x640-plant-pathology-tfrecords)\n     - Duplicates and misleading images were **, not** removed 🙃\n- **Model:**\n     - I experimented with Effnets (B0-B7), EffnetV2s, SeResNext-50, MobileNets, ViTs etc. with different image sizes (640, 512, 224) etc. Out of these Effnet B5 (imagenet) was my best single model (**Public: 84.2, Private: 86.38**)  B6s and B7s were overfitting a lot!\n     - Noisy student weight performed worse throughout my experiments.\n     - **Best Model:**\n          - EffnetB5(imagenet), all BatchNormalization layers were kept frozen, GlobalAveragePooling, followed by Dropout and Output layer of 6 sigmoidal nodes.\n- **Loss Function**\n     - This played an important role in my pipeline, normal cross-entropy didn't work well possibly due to the fact that it inhibits learning in the early stages by penalizing more on the wrongly classified samples.\n     - Focal loss seemed to work well with consistent results throughout.\n     - Asymmetric loss taken from [here](https://arxiv.org/pdf/2009.14119.pdf) worked well for increasing CV but couldn't improve LB (It could have made overfitting easy or maybe I translated it poorly from PyTorch to TensorFlow 😂).\n     - The most promising loss was to use a differentiable version of MacroF1 as described in this great [article](https://towardsdatascience.com/the-unknown-benefits-of-using-a-soft-f1-loss-in-classification-systems-753902c0105d)\n- **Augmentations**\n     - Random Shearing/ Rotations/ Flipping/ Saturation/ Contrast/ Brightness/ Gamma and Central Cropping were used thanks to @cdeotte 's awesome [notebook](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96)\n- **Other Details**\n     - Used Adam optimizer.\n     - Cosine Decay-based Learning Rate.\n     - 5-Fold cross-validation scheme.\n     - Batch Size = 32 per replica of TPU.\n\n- **Inference Scheme**\n     - Averaged outputs of all the 5 folds of a model at a threshold of 0.5.\n\n\nIn the end, I chose one ensemble-based submission [Effnet B5, MobileNetv3 large initialized with Crop net weights, SeResNext50 and others.](Public: 82.9, Private 85.313), didn't have much time to create more diverse ensembles and my best EffNet B5. \n\n- I could've tweaked my inference pipeline to figure out the best threshold (I got an increase of  0.2 on private lb just by tweaking the threshold but I didn't select that as my final submission).\n\nI would put the link to my code after tidying it up a bit. \n\nOverall I enjoyed participating in the competition and would love to know how other's tackled this problem. If you have any suggestions on how I could've improved please comment on them!\n\nSee you guys again in some other competition!😄",
      "votes": null
    },
    {
      "id": "1326308",
      "postDate": "05/28/2021 11:42:47",
      "content": "<p>Congratulations Ashish!</p>",
      "rawMarkdown": "Congratulations Ashish!",
      "votes": null
    },
    {
      "id": "1326322",
      "postDate": "05/28/2021 11:56:07",
      "content": "<p>Congratulations</p>",
      "rawMarkdown": "Congratulations",
      "votes": null
    },
    {
      "id": "1326408",
      "postDate": "05/28/2021 13:14:17",
      "content": "<p>Thanks, Chris! 😀</p>",
      "rawMarkdown": "Thanks, Chris! 😀",
      "votes": null
    },
    {
      "id": "1326409",
      "postDate": "05/28/2021 13:14:37",
      "content": "<p>Thanks, and congrats to you too!</p>",
      "rawMarkdown": "Thanks, and congrats to you too!",
      "votes": null
    },
    {
      "id": "1326421",
      "postDate": "05/28/2021 13:29:55",
      "content": "<p>but i think we should forward，because 8th is a deleted user, and official don’t move it  from  leardboard </p>",
      "rawMarkdown": "but i think we should forward，because 8th is a deleted user, and official don’t move it  from  leardboard",
      "votes": null
    },
    {
      "id": "1326445",
      "postDate": "05/28/2021 14:00:11",
      "content": "<p>Congratulations and thanks for your sharing </p>",
      "rawMarkdown": "Congratulations and thanks for your sharing",
      "votes": null
    },
    {
      "id": "1327124",
      "postDate": "05/29/2021 00:16:20",
      "content": "<p>Congratulations Ashish and thanks for sharing!</p>",
      "rawMarkdown": "Congratulations Ashish and thanks for sharing!",
      "votes": null
    },
    {
      "id": "1327172",
      "postDate": "05/29/2021 02:59:14",
      "content": "<p>Congrats and thanks for sharing!!</p>",
      "rawMarkdown": "Congrats and thanks for sharing!!",
      "votes": null
    },
    {
      "id": "1330317",
      "postDate": "05/31/2021 17:27:43",
      "content": "<p>Congrats ashish ,thanks for sharing</p>",
      "rawMarkdown": "Congrats ashish ,thanks for sharing",
      "votes": null
    },
    {
      "id": "1330462",
      "postDate": "05/31/2021 20:08:23",
      "content": "<p>Thank you Ashish and Congratulations </p>",
      "rawMarkdown": "Thank you Ashish and Congratulations",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1326308,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "05/28/2021 11:42:47",
      "content": "<p>Congratulations Ashish!</p>",
      "votes": null,
      "replies": [
        {
          "id": 1326408,
          "author_name": "ashish2001",
          "author_url": "",
          "post_date": "05/28/2021 13:14:17",
          "content": "<p>Thanks, Chris! 😀</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1326322,
      "author_name": "zxyu1995",
      "author_url": "",
      "post_date": "05/28/2021 11:56:07",
      "content": "<p>Congratulations</p>",
      "votes": null,
      "replies": [
        {
          "id": 1326409,
          "author_name": "ashish2001",
          "author_url": "",
          "post_date": "05/28/2021 13:14:37",
          "content": "<p>Thanks, and congrats to you too!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1326421,
          "author_name": "zxyu1995",
          "author_url": "",
          "post_date": "05/28/2021 13:29:55",
          "content": "<p>but i think we should forward，because 8th is a deleted user, and official don’t move it  from  leardboard </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1326445,
      "author_name": "ytepzhi",
      "author_url": "",
      "post_date": "05/28/2021 14:00:11",
      "content": "<p>Congratulations and thanks for your sharing </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1327124,
      "author_name": "datasciencegeek",
      "author_url": "",
      "post_date": "05/29/2021 00:16:20",
      "content": "<p>Congratulations Ashish and thanks for sharing!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1327172,
      "author_name": "stanpotums",
      "author_url": "",
      "post_date": "05/29/2021 02:59:14",
      "content": "<p>Congrats and thanks for sharing!!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1330317,
      "author_name": "yashdixit24",
      "author_url": "",
      "post_date": "05/31/2021 17:27:43",
      "content": "<p>Congrats ashish ,thanks for sharing</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1330462,
      "author_name": "rahanapa",
      "author_url": "",
      "post_date": "05/31/2021 20:08:23",
      "content": "<p>Thank you Ashish and Congratulations </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1326198": "First of all, thanks to the organizers and Kaggle team for this competition, and congratulations to all the top scorers. This was a great learning experience throughout, read some great papers and tried to implement some of them, and reached a satisfactory position. \nI started from the beginning then took a break and rejoined towards the end. My solution is quite simple but here are some key points for those who can benefit from it!\n\n# **Approach :**\n- **Data:**\n     - I used 640x640 resized tfrecords of the competition data. [link](https://www.kaggle.com/ashish2001/640x640-plant-pathology-tfrecords)\n     - Duplicates and misleading images were **, not** removed 🙃\n- **Model:**\n     - I experimented with Effnets (B0-B7), EffnetV2s, SeResNext-50, MobileNets, ViTs etc. with different image sizes (640, 512, 224) etc. Out of these Effnet B5 (imagenet) was my best single model (**Public: 84.2, Private: 86.38**)  B6s and B7s were overfitting a lot!\n     - Noisy student weight performed worse throughout my experiments.\n     - **Best Model:**\n          - EffnetB5(imagenet), all BatchNormalization layers were kept frozen, GlobalAveragePooling, followed by Dropout and Output layer of 6 sigmoidal nodes.\n- **Loss Function**\n     - This played an important role in my pipeline, normal cross-entropy didn't work well possibly due to the fact that it inhibits learning in the early stages by penalizing more on the wrongly classified samples.\n     - Focal loss seemed to work well with consistent results throughout.\n     - Asymmetric loss taken from [here](https://arxiv.org/pdf/2009.14119.pdf) worked well for increasing CV but couldn't improve LB (It could have made overfitting easy or maybe I translated it poorly from PyTorch to TensorFlow 😂).\n     - The most promising loss was to use a differentiable version of MacroF1 as described in this great [article](https://towardsdatascience.com/the-unknown-benefits-of-using-a-soft-f1-loss-in-classification-systems-753902c0105d)\n- **Augmentations**\n     - Random Shearing/ Rotations/ Flipping/ Saturation/ Contrast/ Brightness/ Gamma and Central Cropping were used thanks to @cdeotte 's awesome [notebook](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96)\n- **Other Details**\n     - Used Adam optimizer.\n     - Cosine Decay-based Learning Rate.\n     - 5-Fold cross-validation scheme.\n     - Batch Size = 32 per replica of TPU.\n\n- **Inference Scheme**\n     - Averaged outputs of all the 5 folds of a model at a threshold of 0.5.\n\n\nIn the end, I chose one ensemble-based submission [Effnet B5, MobileNetv3 large initialized with Crop net weights, SeResNext50 and others.](Public: 82.9, Private 85.313), didn't have much time to create more diverse ensembles and my best EffNet B5. \n\n- I could've tweaked my inference pipeline to figure out the best threshold (I got an increase of  0.2 on private lb just by tweaking the threshold but I didn't select that as my final submission).\n\nI would put the link to my code after tidying it up a bit. \n\nOverall I enjoyed participating in the competition and would love to know how other's tackled this problem. If you have any suggestions on how I could've improved please comment on them!\n\nSee you guys again in some other competition!😄",
    "1326308": "Congratulations Ashish!",
    "1326322": "Congratulations",
    "1326408": "Thanks, Chris! 😀",
    "1326409": "Thanks, and congrats to you too!",
    "1326421": "but i think we should forward，because 8th is a deleted user, and official don’t move it  from  leardboard",
    "1326445": "Congratulations and thanks for your sharing",
    "1327124": "Congratulations Ashish and thanks for sharing!",
    "1327172": "Congrats and thanks for sharing!!",
    "1330317": "Congrats ashish ,thanks for sharing",
    "1330462": "Thank you Ashish and Congratulations"
  },
  "source": "meta"
}