{
  "id": 205080,
  "title": "Pytorch Lightning + Hydra Starter Kit - Public LB 0.894",
  "url": "/competitions/cassava-leaf-disease-classification/discussion/205080",
  "author_name": "",
  "post_date": "2020-12-18T10:46:58.894714800Z",
  "votes": 9,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi, </p>\n<p>An important lesson I learned from observing the previous competition winners is that a solid ML/ DL pipeline is critical for getting to the top. The pipeline should let you </p>\n<ul>\n<li>Test different ideas</li>\n<li>Log metrics </li>\n<li>Easily allow you to compare the results between different runs/folds. </li>\n</ul>\n<p>With these goals in mind, I have built the pipeline that lets us experiment with different ideas without changing much of a code like </p>\n<ul>\n<li>Choosing different models from Resnet and Efficientnet Family.</li>\n<li>Choosing different hyperparameters like Learning Rate, weight decay, image size, epochs, batch size, etc. </li>\n<li>Integrating with <a href=\"https://www.wandb.com/\" target=\"_blank\">Wandb</a> for visualizing the metrics.</li>\n<li>Augmentations using the <a href=\"https://albumentations.ai/\" target=\"_blank\">Albumentation</a> library.</li>\n<li>All the good things that come with PyTorch Lightning like - Mixed Precision, Run on Multi GPUs, Use SyncBN, Gradient Accumulation, etc.</li>\n</ul>\n<p>A single model achieves a result of 0.883 on LB.<br>\nA 5 Fold model achieves a result of 0.894 on LB.</p>\n<p>Find the code in GitHub <a href=\"https://github.com/svishnu88/Cassava\" target=\"_blank\">here</a>.<br>\n<a href=\"https://www.kaggle.com/vishnus/cassava-pytorch-lightning-starter-notebook-0-895\" target=\"_blank\">Kaggle Kernel</a><br>\nKaggle Inference Kernel [Place Holder]</p>\n<h2>Updates - 25-12-2020</h2>\n<ul>\n<li>Done all the configuration management with Hydra an amazing library</li>\n<li>Added Label smoothing </li>\n<li>Added Gradual WarmUp Cosine Learning</li>\n<li>Added code as Kernel, so that it is easy to run in Kaggle.</li>\n</ul>\n<p>The lightning.py contains code without hydra and light_hydra contains code with Hydra. You can just refer one, left both for anyone to compare. </p>",
  "messages": [
    {
      "id": "1117693",
      "postDate": "12/18/2020 10:46:58",
      "content": "<p>Hi, </p>\n<p>An important lesson I learned from observing the previous competition winners is that a solid ML/ DL pipeline is critical for getting to the top. The pipeline should let you </p>\n<ul>\n<li>Test different ideas</li>\n<li>Log metrics </li>\n<li>Easily allow you to compare the results between different runs/folds. </li>\n</ul>\n<p>With these goals in mind, I have built the pipeline that lets us experiment with different ideas without changing much of a code like </p>\n<ul>\n<li>Choosing different models from Resnet and Efficientnet Family.</li>\n<li>Choosing different hyperparameters like Learning Rate, weight decay, image size, epochs, batch size, etc. </li>\n<li>Integrating with <a href=\"https://www.wandb.com/\" target=\"_blank\">Wandb</a> for visualizing the metrics.</li>\n<li>Augmentations using the <a href=\"https://albumentations.ai/\" target=\"_blank\">Albumentation</a> library.</li>\n<li>All the good things that come with PyTorch Lightning like - Mixed Precision, Run on Multi GPUs, Use SyncBN, Gradient Accumulation, etc.</li>\n</ul>\n<p>A single model achieves a result of 0.883 on LB.<br>\nA 5 Fold model achieves a result of 0.894 on LB.</p>\n<p>Find the code in GitHub <a href=\"https://github.com/svishnu88/Cassava\" target=\"_blank\">here</a>.<br>\n<a href=\"https://www.kaggle.com/vishnus/cassava-pytorch-lightning-starter-notebook-0-895\" target=\"_blank\">Kaggle Kernel</a><br>\nKaggle Inference Kernel [Place Holder]</p>\n<h2>Updates - 25-12-2020</h2>\n<ul>\n<li>Done all the configuration management with Hydra an amazing library</li>\n<li>Added Label smoothing </li>\n<li>Added Gradual WarmUp Cosine Learning</li>\n<li>Added code as Kernel, so that it is easy to run in Kaggle.</li>\n</ul>\n<p>The lightning.py contains code without hydra and light_hydra contains code with Hydra. You can just refer one, left both for anyone to compare. </p>",
      "rawMarkdown": "Hi, \n\nAn important lesson I learned from observing the previous competition winners is that a solid ML/ DL pipeline is critical for getting to the top. The pipeline should let you \n\n- Test different ideas\n- Log metrics \n- Easily allow you to compare the results between different runs/folds. \n\nWith these goals in mind, I have built the pipeline that lets us experiment with different ideas without changing much of a code like \n\n- Choosing different models from Resnet and Efficientnet Family.\n- Choosing different hyperparameters like Learning Rate, weight decay, image size, epochs, batch size, etc. \n- Integrating with [Wandb](https://www.wandb.com/) for visualizing the metrics.\n- Augmentations using the [Albumentation](https://albumentations.ai/) library.\n- All the good things that come with PyTorch Lightning like - Mixed Precision, Run on Multi GPUs, Use SyncBN, Gradient Accumulation, etc.\n\nA single model achieves a result of 0.883 on LB.\nA 5 Fold model achieves a result of 0.894 on LB.\n\nFind the code in GitHub [here](https://github.com/svishnu88/Cassava).\n[Kaggle Kernel](https://www.kaggle.com/vishnus/cassava-pytorch-lightning-starter-notebook-0-895)\nKaggle Inference Kernel [Place Holder]\n\n## Updates - 25-12-2020\n\n- Done all the configuration management with Hydra an amazing library\n- Added Label smoothing \n- Added Gradual WarmUp Cosine Learning\n- Added code as Kernel, so that it is easy to run in Kaggle.\n\n\nThe lightning.py contains code without hydra and light_hydra contains code with Hydra. You can just refer one, left both for anyone to compare.",
      "votes": null
    },
    {
      "id": "1117977",
      "postDate": "12/18/2020 16:21:10",
      "content": "<p>thanks for posting ✌✌</p>",
      "rawMarkdown": "thanks for posting ✌✌",
      "votes": null
    },
    {
      "id": "1121093",
      "postDate": "12/21/2020 11:17:14",
      "content": "<p>Thanks for posting. Is the use of focal loss or dice loss aiding?</p>",
      "rawMarkdown": "Thanks for posting. Is the use of focal loss or dice loss aiding?",
      "votes": null
    },
    {
      "id": "1121232",
      "postDate": "12/21/2020 13:13:43",
      "content": "<p>I tried focal loss, did not see any major improvement. LabelSmoothing gave a .002 improvement. </p>",
      "rawMarkdown": "I tried focal loss, did not see any major improvement. LabelSmoothing gave a .002 improvement.",
      "votes": null
    },
    {
      "id": "1122103",
      "postDate": "12/22/2020 07:37:00",
      "content": "<p>Have you tried TPU with pytorch-lightning?<br>\nI tried but it didn't work.</p>",
      "rawMarkdown": "Have you tried TPU with pytorch-lightning?\nI tried but it didn't work.",
      "votes": null
    },
    {
      "id": "1122388",
      "postDate": "12/22/2020 12:02:18",
      "content": "<p>Nope. I just used GPU.</p>",
      "rawMarkdown": "Nope. I just used GPU.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1117977,
      "author_name": "shyamgupta196",
      "author_url": "",
      "post_date": "12/18/2020 16:21:10",
      "content": "<p>thanks for posting ✌✌</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1121093,
      "author_name": "tamilselvanmoorthy",
      "author_url": "",
      "post_date": "12/21/2020 11:17:14",
      "content": "<p>Thanks for posting. Is the use of focal loss or dice loss aiding?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1121232,
          "author_name": "vishnus",
          "author_url": "",
          "post_date": "12/21/2020 13:13:43",
          "content": "<p>I tried focal loss, did not see any major improvement. LabelSmoothing gave a .002 improvement. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1122103,
      "author_name": "jabertuhin",
      "author_url": "",
      "post_date": "12/22/2020 07:37:00",
      "content": "<p>Have you tried TPU with pytorch-lightning?<br>\nI tried but it didn't work.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1122388,
          "author_name": "vishnus",
          "author_url": "",
          "post_date": "12/22/2020 12:02:18",
          "content": "<p>Nope. I just used GPU.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1117693": "Hi, \n\nAn important lesson I learned from observing the previous competition winners is that a solid ML/ DL pipeline is critical for getting to the top. The pipeline should let you \n\n- Test different ideas\n- Log metrics \n- Easily allow you to compare the results between different runs/folds. \n\nWith these goals in mind, I have built the pipeline that lets us experiment with different ideas without changing much of a code like \n\n- Choosing different models from Resnet and Efficientnet Family.\n- Choosing different hyperparameters like Learning Rate, weight decay, image size, epochs, batch size, etc. \n- Integrating with [Wandb](https://www.wandb.com/) for visualizing the metrics.\n- Augmentations using the [Albumentation](https://albumentations.ai/) library.\n- All the good things that come with PyTorch Lightning like - Mixed Precision, Run on Multi GPUs, Use SyncBN, Gradient Accumulation, etc.\n\nA single model achieves a result of 0.883 on LB.\nA 5 Fold model achieves a result of 0.894 on LB.\n\nFind the code in GitHub [here](https://github.com/svishnu88/Cassava).\n[Kaggle Kernel](https://www.kaggle.com/vishnus/cassava-pytorch-lightning-starter-notebook-0-895)\nKaggle Inference Kernel [Place Holder]\n\n## Updates - 25-12-2020\n\n- Done all the configuration management with Hydra an amazing library\n- Added Label smoothing \n- Added Gradual WarmUp Cosine Learning\n- Added code as Kernel, so that it is easy to run in Kaggle.\n\n\nThe lightning.py contains code without hydra and light_hydra contains code with Hydra. You can just refer one, left both for anyone to compare.",
    "1117977": "thanks for posting ✌✌",
    "1121093": "Thanks for posting. Is the use of focal loss or dice loss aiding?",
    "1121232": "I tried focal loss, did not see any major improvement. LabelSmoothing gave a .002 improvement.",
    "1122103": "Have you tried TPU with pytorch-lightning?\nI tried but it didn't work.",
    "1122388": "Nope. I just used GPU."
  },
  "source": "meta"
}