{
  "id": 220889,
  "title": "18th Place Solution",
  "url": "/competitions/cassava-leaf-disease-classification/writeups/cassavacake-18th-place-solution",
  "author_name": "",
  "post_date": "2021-02-20T00:54:35.297992900Z",
  "votes": 18,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I would like to give thanks to all participants who shared their wealth of knowledge and wisdom for this competition thru notebook and discussion sharing.</p>\n<p>My submitted solution consist of simple average of 3 models namely: 5 folds gluon_seresnext101_32x4d, tf_efficientnet_b4_ns and seresnext50_32x4d. </p>\n<p>The following are the final scores of my model:</p>\n<p>gluon_seresnext101_32x4d<br>\n    CV: 0.9012; Public LB: 0.9011; Private LB: 0.8993</p>\n<p>tf_efficientnet_b4_ns<br>\n    CV: 0.9013; Public LB: 0.9011; Private LB: 0.8970</p>\n<p>seresnext50_32x4d<br>\n    CV: 0.9007; Public LB: 0.8978, Private LB: 0.8987</p>\n<p>Simple Average<br>\n    Public LB: 0.905; Private LB: 0.9013</p>\n<p>Training Methodology<br>\nThe parameters used during stages of model training:</p>\n<p>1st stage<br>\nimage size: 320x320 or 384x384<br>\n10 epochs<br>\nlearning rate: 1e-4<br>\ntrain set: 2019 + 2020 data<br>\nvalidation set: 2020<br>\nscheduler: GradualWarmupScheduler<br>\ncriterion: TaylorCrossEntropyLoss with label smoothing=0.2<br>\noptimizer: SAM + Adam<br>\naugmentation:  RandomResizedCrop, Transpose, HorizontalFlip, VerticalFlip, ShiftScaleRotate, Cutout</p>\n<p>2nd stage<br>\nimage size: 512x512<br>\n25-40 epochs<br>\nlearning rate: 1e-4<br>\ntrain set: 2020 data<br>\nvalidation set: 2020 data<br>\nscheduler: GradualWarmupScheduler<br>\ncriterion: Combo Loss (see notebook in detail)<br>\noptimizer: SAM + AdamP<br>\naugmentation:  RandomResizedCrop, Transpose, HorizontalFlip, VerticalFlip, ShiftScaleRotate, Cutout + Cutmix</p>\n<p>Finetune stage<br>\n5 epochs<br>\nlearning rate: 1e-5<br>\ntrain set: 2020 data<br>\nvalidation set: 2020 data<br>\nscheduler: no scheduler<br>\ncriterion: Combo Loss (see notebook in detail)<br>\noptimizer: SAM + AdamP<br>\naugmentation:  hard albumentation-based augmentations</p>\n<p>You may check this notebook for reference:<br>\n<a href=\"https://www.kaggle.com/projdev/base-notebook-cassava-leaf-disease-classification\" target=\"_blank\">https://www.kaggle.com/projdev/base-notebook-cassava-leaf-disease-classification</a></p>\n<p>I believe that the combination of SAM + AdamP optimizer provides the model robustness in terms of prediction generalization and the key to survive in competition shake-up.</p>\n<p>A possible improvement for my model score is using SAM + SGDP and add FocalCosineLoss in Combo loss and find the right learning rate as it converges quickly. Unfortunately, I have no available GPU resources to re-train my model as SAM optimizer double the training time and I spent too much time on finding the right parameters for my model training.</p>\n<p>Once again, thank you very much.</p>",
  "messages": [
    {
      "id": "1211112",
      "postDate": "02/20/2021 00:54:35",
      "content": "<p>I would like to give thanks to all participants who shared their wealth of knowledge and wisdom for this competition thru notebook and discussion sharing.</p>\n<p>My submitted solution consist of simple average of 3 models namely: 5 folds gluon_seresnext101_32x4d, tf_efficientnet_b4_ns and seresnext50_32x4d. </p>\n<p>The following are the final scores of my model:</p>\n<p>gluon_seresnext101_32x4d<br>\n    CV: 0.9012; Public LB: 0.9011; Private LB: 0.8993</p>\n<p>tf_efficientnet_b4_ns<br>\n    CV: 0.9013; Public LB: 0.9011; Private LB: 0.8970</p>\n<p>seresnext50_32x4d<br>\n    CV: 0.9007; Public LB: 0.8978, Private LB: 0.8987</p>\n<p>Simple Average<br>\n    Public LB: 0.905; Private LB: 0.9013</p>\n<p>Training Methodology<br>\nThe parameters used during stages of model training:</p>\n<p>1st stage<br>\nimage size: 320x320 or 384x384<br>\n10 epochs<br>\nlearning rate: 1e-4<br>\ntrain set: 2019 + 2020 data<br>\nvalidation set: 2020<br>\nscheduler: GradualWarmupScheduler<br>\ncriterion: TaylorCrossEntropyLoss with label smoothing=0.2<br>\noptimizer: SAM + Adam<br>\naugmentation:  RandomResizedCrop, Transpose, HorizontalFlip, VerticalFlip, ShiftScaleRotate, Cutout</p>\n<p>2nd stage<br>\nimage size: 512x512<br>\n25-40 epochs<br>\nlearning rate: 1e-4<br>\ntrain set: 2020 data<br>\nvalidation set: 2020 data<br>\nscheduler: GradualWarmupScheduler<br>\ncriterion: Combo Loss (see notebook in detail)<br>\noptimizer: SAM + AdamP<br>\naugmentation:  RandomResizedCrop, Transpose, HorizontalFlip, VerticalFlip, ShiftScaleRotate, Cutout + Cutmix</p>\n<p>Finetune stage<br>\n5 epochs<br>\nlearning rate: 1e-5<br>\ntrain set: 2020 data<br>\nvalidation set: 2020 data<br>\nscheduler: no scheduler<br>\ncriterion: Combo Loss (see notebook in detail)<br>\noptimizer: SAM + AdamP<br>\naugmentation:  hard albumentation-based augmentations</p>\n<p>You may check this notebook for reference:<br>\n<a href=\"https://www.kaggle.com/projdev/base-notebook-cassava-leaf-disease-classification\" target=\"_blank\">https://www.kaggle.com/projdev/base-notebook-cassava-leaf-disease-classification</a></p>\n<p>I believe that the combination of SAM + AdamP optimizer provides the model robustness in terms of prediction generalization and the key to survive in competition shake-up.</p>\n<p>A possible improvement for my model score is using SAM + SGDP and add FocalCosineLoss in Combo loss and find the right learning rate as it converges quickly. Unfortunately, I have no available GPU resources to re-train my model as SAM optimizer double the training time and I spent too much time on finding the right parameters for my model training.</p>\n<p>Once again, thank you very much.</p>",
      "rawMarkdown": "I would like to give thanks to all participants who shared their wealth of knowledge and wisdom for this competition thru notebook and discussion sharing.\n\nMy submitted solution consist of simple average of 3 models namely: 5 folds gluon_seresnext101_32x4d, tf_efficientnet_b4_ns and seresnext50_32x4d. \n\nThe following are the final scores of my model:\n\ngluon_seresnext101_32x4d\n\tCV: 0.9012; Public LB: 0.9011; Private LB: 0.8993\n\t\ntf_efficientnet_b4_ns\n\tCV: 0.9013; Public LB: 0.9011; Private LB: 0.8970\n\nseresnext50_32x4d\n\tCV: 0.9007; Public LB: 0.8978, Private LB: 0.8987\n\nSimple Average\n\tPublic LB: 0.905; Private LB: 0.9013\n\nTraining Methodology\nThe parameters used during stages of model training:\n\n1st stage\nimage size: 320x320 or 384x384\n10 epochs\nlearning rate: 1e-4\ntrain set: 2019 + 2020 data\nvalidation set: 2020\nscheduler: GradualWarmupScheduler\ncriterion: TaylorCrossEntropyLoss with label smoothing=0.2\noptimizer: SAM + Adam\naugmentation:  RandomResizedCrop, Transpose, HorizontalFlip, VerticalFlip, ShiftScaleRotate, Cutout\n\n2nd stage\nimage size: 512x512\n25-40 epochs\nlearning rate: 1e-4\ntrain set: 2020 data\nvalidation set: 2020 data\nscheduler: GradualWarmupScheduler\ncriterion: Combo Loss (see notebook in detail)\noptimizer: SAM + AdamP\naugmentation:  RandomResizedCrop, Transpose, HorizontalFlip, VerticalFlip, ShiftScaleRotate, Cutout + Cutmix\n\nFinetune stage\n5 epochs\nlearning rate: 1e-5\ntrain set: 2020 data\nvalidation set: 2020 data\nscheduler: no scheduler\ncriterion: Combo Loss (see notebook in detail)\noptimizer: SAM + AdamP\naugmentation:  hard albumentation-based augmentations\n\nYou may check this notebook for reference:\nhttps://www.kaggle.com/projdev/base-notebook-cassava-leaf-disease-classification\n\nI believe that the combination of SAM + AdamP optimizer provides the model robustness in terms of prediction generalization and the key to survive in competition shake-up.\n\nA possible improvement for my model score is using SAM + SGDP and add FocalCosineLoss in Combo loss and find the right learning rate as it converges quickly. Unfortunately, I have no available GPU resources to re-train my model as SAM optimizer double the training time and I spent too much time on finding the right parameters for my model training.\n\nOnce again, thank you very much.",
      "votes": null
    },
    {
      "id": "1211469",
      "postDate": "02/20/2021 09:06:50",
      "content": "<p>great work!</p>",
      "rawMarkdown": "great work!",
      "votes": null
    },
    {
      "id": "1220917",
      "postDate": "02/28/2021 13:40:11",
      "content": "<p>SAM + AdamP   is  really a  nice  move .</p>",
      "rawMarkdown": "SAM + AdamP   is  really a  nice  move .",
      "votes": null
    },
    {
      "id": "1250640",
      "postDate": "03/24/2021 07:17:28",
      "content": "<p>Thank you <a href=\"https://www.kaggle.com/projdev\" target=\"_blank\">@projdev</a> for sharing. This is encouraging.</p>",
      "rawMarkdown": "Thank you @projdev for sharing. This is encouraging.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1211469,
      "author_name": "wonjunpark",
      "author_url": "",
      "post_date": "02/20/2021 09:06:50",
      "content": "<p>great work!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1220917,
      "author_name": "xujingzhao",
      "author_url": "",
      "post_date": "02/28/2021 13:40:11",
      "content": "<p>SAM + AdamP   is  really a  nice  move .</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1250640,
      "author_name": "olusesiadebisi",
      "author_url": "",
      "post_date": "03/24/2021 07:17:28",
      "content": "<p>Thank you <a href=\"https://www.kaggle.com/projdev\" target=\"_blank\">@projdev</a> for sharing. This is encouraging.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1211112": "I would like to give thanks to all participants who shared their wealth of knowledge and wisdom for this competition thru notebook and discussion sharing.\n\nMy submitted solution consist of simple average of 3 models namely: 5 folds gluon_seresnext101_32x4d, tf_efficientnet_b4_ns and seresnext50_32x4d. \n\nThe following are the final scores of my model:\n\ngluon_seresnext101_32x4d\n\tCV: 0.9012; Public LB: 0.9011; Private LB: 0.8993\n\t\ntf_efficientnet_b4_ns\n\tCV: 0.9013; Public LB: 0.9011; Private LB: 0.8970\n\nseresnext50_32x4d\n\tCV: 0.9007; Public LB: 0.8978, Private LB: 0.8987\n\nSimple Average\n\tPublic LB: 0.905; Private LB: 0.9013\n\nTraining Methodology\nThe parameters used during stages of model training:\n\n1st stage\nimage size: 320x320 or 384x384\n10 epochs\nlearning rate: 1e-4\ntrain set: 2019 + 2020 data\nvalidation set: 2020\nscheduler: GradualWarmupScheduler\ncriterion: TaylorCrossEntropyLoss with label smoothing=0.2\noptimizer: SAM + Adam\naugmentation:  RandomResizedCrop, Transpose, HorizontalFlip, VerticalFlip, ShiftScaleRotate, Cutout\n\n2nd stage\nimage size: 512x512\n25-40 epochs\nlearning rate: 1e-4\ntrain set: 2020 data\nvalidation set: 2020 data\nscheduler: GradualWarmupScheduler\ncriterion: Combo Loss (see notebook in detail)\noptimizer: SAM + AdamP\naugmentation:  RandomResizedCrop, Transpose, HorizontalFlip, VerticalFlip, ShiftScaleRotate, Cutout + Cutmix\n\nFinetune stage\n5 epochs\nlearning rate: 1e-5\ntrain set: 2020 data\nvalidation set: 2020 data\nscheduler: no scheduler\ncriterion: Combo Loss (see notebook in detail)\noptimizer: SAM + AdamP\naugmentation:  hard albumentation-based augmentations\n\nYou may check this notebook for reference:\nhttps://www.kaggle.com/projdev/base-notebook-cassava-leaf-disease-classification\n\nI believe that the combination of SAM + AdamP optimizer provides the model robustness in terms of prediction generalization and the key to survive in competition shake-up.\n\nA possible improvement for my model score is using SAM + SGDP and add FocalCosineLoss in Combo loss and find the right learning rate as it converges quickly. Unfortunately, I have no available GPU resources to re-train my model as SAM optimizer double the training time and I spent too much time on finding the right parameters for my model training.\n\nOnce again, thank you very much.",
    "1211469": "great work!",
    "1220917": "SAM + AdamP   is  really a  nice  move .",
    "1250640": "Thank you @projdev for sharing. This is encouraging."
  },
  "source": "meta"
}