{
  "id": 104729,
  "title": "how to properly train an EfficientNet?",
  "url": "/competitions/aptos2019-blindness-detection/discussion/104729",
  "author_name": "",
  "post_date": "2019-08-18T19:39:51.292666900Z",
  "votes": 4,
  "comment_count": 7,
  "views": 0,
  "content": "<p>I started training Resnet50 and EfficientNet B0 &amp; B3.  once I removed black borders info to <a href=\"https://www.kaggle.com/taindow/be-careful-what-you-train-on\">\"be careful what you train on\"</a>, I wasn't able to have any nice EfficientNet model</p>\n\n<p>I've tried:</p>\n\n<pre><code>- the original image size for B0 &amp;amp; B3\n- train the head only\n- train the head only + unfreeze last convolutional block\n- train the head only + unfreeze last two convolutional blocks\n- train the head only + unfreeze the whole network\n- play with lr (I used fastai find_lr)\n</code></pre>\n\n<p>I've read in this forums about the difficulty of this kind of networks to be trained.</p>\n\n<p>I'd like to train a B6. any tips or advice are wellcomed!</p>",
  "messages": [
    {
      "id": "602222",
      "postDate": "08/18/2019 19:39:51",
      "content": "<p>I started training Resnet50 and EfficientNet B0 &amp; B3.  once I removed black borders info to <a href=\"https://www.kaggle.com/taindow/be-careful-what-you-train-on\">\"be careful what you train on\"</a>, I wasn't able to have any nice EfficientNet model</p>\n\n<p>I've tried:</p>\n\n<pre><code>- the original image size for B0 &amp;amp; B3\n- train the head only\n- train the head only + unfreeze last convolutional block\n- train the head only + unfreeze last two convolutional blocks\n- train the head only + unfreeze the whole network\n- play with lr (I used fastai find_lr)\n</code></pre>\n\n<p>I've read in this forums about the difficulty of this kind of networks to be trained.</p>\n\n<p>I'd like to train a B6. any tips or advice are wellcomed!</p>",
      "rawMarkdown": "I started training Resnet50 and EfficientNet B0 &amp; B3.  once I removed black borders info to [\"be careful what you train on\"](https://www.kaggle.com/taindow/be-careful-what-you-train-on), I wasn't able to have any nice EfficientNet model\n\nI've tried:\n\n\t- the original image size for B0 &amp; B3\n\t- train the head only\n\t- train the head only + unfreeze last convolutional block\n\t- train the head only + unfreeze last two convolutional blocks\n\t- train the head only + unfreeze the whole network\n\t- play with lr (I used fastai find_lr)\n\nI've read in this forums about the difficulty of this kind of networks to be trained.\n\nI'd like to train a B6. any tips or advice are wellcomed!",
      "votes": null
    },
    {
      "id": "602224",
      "postDate": "08/18/2019 19:46:58",
      "content": "<p>hello dear, how about this? : <a href=\"https://www.kaggle.com/meaninglesslives/unet-plus-plus-with-efficientnet-encoder\">https://www.kaggle.com/meaninglesslives/unet-plus-plus-with-efficientnet-encoder</a></p>",
      "rawMarkdown": "hello dear, how about this? : https://www.kaggle.com/meaninglesslives/unet-plus-plus-with-efficientnet-encoder",
      "votes": null
    },
    {
      "id": "602396",
      "postDate": "08/19/2019 03:27:37",
      "content": "<p>I just train whole network simply. \nI got similar score with/without border in cv. ( I dont know how it will different at lb. I didn’t submit yet)</p>",
      "rawMarkdown": "I just train whole network simply. \nI got similar score with/without border in cv. ( I dont know how it will different at lb. I didn’t submit yet)",
      "votes": null
    },
    {
      "id": "604130",
      "postDate": "08/21/2019 04:22:37",
      "content": "<p><a href=\"/virilo\">@virilo</a> Do u figure it out?</p>",
      "rawMarkdown": "virilo Do u figure it out?",
      "votes": null
    },
    {
      "id": "604777",
      "postDate": "08/21/2019 19:15:53",
      "content": "<p>No need for any of those tricks to get &gt; 0.800, just make ssure you train for long enough and reduce ur lr on the way</p>",
      "rawMarkdown": "No need for any of those tricks to get &gt; 0.800, just make ssure you train for long enough and reduce ur lr on the way",
      "votes": null
    },
    {
      "id": "605096",
      "postDate": "08/22/2019 04:12:39",
      "content": "<p>thanks <a href=\"/mobassir\">@mobassir</a>, I have read this link.  Interesting reading... upvoted!</p>\n\n<p>This notebook is using Efficientnet as base architecture in an Unet++ for a segmentation problem</p>\n\n<p>A little bit different to our task, but I jotted down some ideas from your link:</p>\n\n<pre><code>- SWA\n- cosine_anneal_schedule: I've to check the differences between this implementation and fastai fit_one_cycle\n</code></pre>\n\n<p>Also, I'll give a try to train the whole network unfreeze from the beginning a as <a href=\"/vanche\">@vanche</a>  purposed (thanks <a href=\"/vanche\">@vanche</a> )</p>\n\n<p>Apart from that, I'd like to play with EfficientNet hyperparameters.  Perhaps it should have been the first step.</p>\n\n<p>Thanks a lot.</p>\n\n<p>Any other ideas will be very welcomed</p>",
      "rawMarkdown": "thanks @mobassir, I have read this link.  Interesting reading... upvoted!\n\nThis notebook is using Efficientnet as base architecture in an Unet++ for a segmentation problem\n\nA little bit different to our task, but I jotted down some ideas from your link:\n\n\t- SWA\n\t- cosine_anneal_schedule: I've to check the differences between this implementation and fastai fit_one_cycle\n\nAlso, I'll give a try to train the whole network unfreeze from the beginning a as @vanche  purposed (thanks @vanche )\n\nApart from that, I'd like to play with EfficientNet hyperparameters.  Perhaps it should have been the first step.\n\nThanks a lot.\n\nAny other ideas will be very welcomed",
      "votes": null
    },
    {
      "id": "605099",
      "postDate": "08/22/2019 04:16:40",
      "content": "<p>thanks <a href=\"/sidhanthholalkere\">@sidhanthholalkere</a>. I'll give a try to \"reduce on plateau\"</p>\n\n<p>How long would I need to train? More than 9 hours?</p>",
      "rawMarkdown": "thanks @sidhanthholalkere. I'll give a try to \"reduce on plateau\"\n\nHow long would I need to train? More than 9 hours?",
      "votes": null
    },
    {
      "id": "605220",
      "postDate": "08/22/2019 07:15:22",
      "content": "<p>thata great,how about radams for your model? <a href=\"/virilo\">@virilo</a> </p>",
      "rawMarkdown": "thata great,how about radams for your model? @virilo",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 602224,
      "author_name": "mobassir",
      "author_url": "",
      "post_date": "08/18/2019 19:46:58",
      "content": "<p>hello dear, how about this? : <a href=\"https://www.kaggle.com/meaninglesslives/unet-plus-plus-with-efficientnet-encoder\">https://www.kaggle.com/meaninglesslives/unet-plus-plus-with-efficientnet-encoder</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 605096,
          "author_name": "virilo",
          "author_url": "",
          "post_date": "08/22/2019 04:12:39",
          "content": "<p>thanks <a href=\"/mobassir\">@mobassir</a>, I have read this link.  Interesting reading... upvoted!</p>\n\n<p>This notebook is using Efficientnet as base architecture in an Unet++ for a segmentation problem</p>\n\n<p>A little bit different to our task, but I jotted down some ideas from your link:</p>\n\n<pre><code>- SWA\n- cosine_anneal_schedule: I've to check the differences between this implementation and fastai fit_one_cycle\n</code></pre>\n\n<p>Also, I'll give a try to train the whole network unfreeze from the beginning a as <a href=\"/vanche\">@vanche</a>  purposed (thanks <a href=\"/vanche\">@vanche</a> )</p>\n\n<p>Apart from that, I'd like to play with EfficientNet hyperparameters.  Perhaps it should have been the first step.</p>\n\n<p>Thanks a lot.</p>\n\n<p>Any other ideas will be very welcomed</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 605220,
          "author_name": "mobassir",
          "author_url": "",
          "post_date": "08/22/2019 07:15:22",
          "content": "<p>thata great,how about radams for your model? <a href=\"/virilo\">@virilo</a> </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 602396,
      "author_name": "vanche",
      "author_url": "",
      "post_date": "08/19/2019 03:27:37",
      "content": "<p>I just train whole network simply. \nI got similar score with/without border in cv. ( I dont know how it will different at lb. I didn’t submit yet)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 604130,
      "author_name": "zhan2019",
      "author_url": "",
      "post_date": "08/21/2019 04:22:37",
      "content": "<p><a href=\"/virilo\">@virilo</a> Do u figure it out?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 604777,
      "author_name": "sidhanthholalkere",
      "author_url": "",
      "post_date": "08/21/2019 19:15:53",
      "content": "<p>No need for any of those tricks to get &gt; 0.800, just make ssure you train for long enough and reduce ur lr on the way</p>",
      "votes": null,
      "replies": [
        {
          "id": 605099,
          "author_name": "virilo",
          "author_url": "",
          "post_date": "08/22/2019 04:16:40",
          "content": "<p>thanks <a href=\"/sidhanthholalkere\">@sidhanthholalkere</a>. I'll give a try to \"reduce on plateau\"</p>\n\n<p>How long would I need to train? More than 9 hours?</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "602222": "I started training Resnet50 and EfficientNet B0 &amp; B3.  once I removed black borders info to [\"be careful what you train on\"](https://www.kaggle.com/taindow/be-careful-what-you-train-on), I wasn't able to have any nice EfficientNet model\n\nI've tried:\n\n\t- the original image size for B0 &amp; B3\n\t- train the head only\n\t- train the head only + unfreeze last convolutional block\n\t- train the head only + unfreeze last two convolutional blocks\n\t- train the head only + unfreeze the whole network\n\t- play with lr (I used fastai find_lr)\n\nI've read in this forums about the difficulty of this kind of networks to be trained.\n\nI'd like to train a B6. any tips or advice are wellcomed!",
    "602224": "hello dear, how about this? : https://www.kaggle.com/meaninglesslives/unet-plus-plus-with-efficientnet-encoder",
    "602396": "I just train whole network simply. \nI got similar score with/without border in cv. ( I dont know how it will different at lb. I didn’t submit yet)",
    "604130": "virilo Do u figure it out?",
    "604777": "No need for any of those tricks to get &gt; 0.800, just make ssure you train for long enough and reduce ur lr on the way",
    "605096": "thanks @mobassir, I have read this link.  Interesting reading... upvoted!\n\nThis notebook is using Efficientnet as base architecture in an Unet++ for a segmentation problem\n\nA little bit different to our task, but I jotted down some ideas from your link:\n\n\t- SWA\n\t- cosine_anneal_schedule: I've to check the differences between this implementation and fastai fit_one_cycle\n\nAlso, I'll give a try to train the whole network unfreeze from the beginning a as @vanche  purposed (thanks @vanche )\n\nApart from that, I'd like to play with EfficientNet hyperparameters.  Perhaps it should have been the first step.\n\nThanks a lot.\n\nAny other ideas will be very welcomed",
    "605099": "thanks @sidhanthholalkere. I'll give a try to \"reduce on plateau\"\n\nHow long would I need to train? More than 9 hours?",
    "605220": "thata great,how about radams for your model? @virilo"
  },
  "source": "meta"
}