{
  "id": 93947,
  "title": "LR/momentum scheduling > everything else",
  "url": "/competitions/freesound-audio-tagging-2019/discussion/93947",
  "author_name": "",
  "post_date": "2019-05-31T09:58:16.627932900Z",
  "votes": 2,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Another frustration thread just like <a href=\"https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/93664\">https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/93664</a></p>\n\n<p>So far I've tried the most sophisticated augmentation methods, model structures and training methods but as you can see -&gt; I'm still below medal area and drifting down.\nMost importantly is that all my achievements are made with pure PyTorch and sometimes it takes 4-6 hours to train a single model.</p>\n\n<p>But why did I mention LR scheduling here? Because just by incorporating Fast.AI, which obviously does a tonne  under the hood, I'm able to achieve the same results within less than an hour of training time (and without any crazy augumentation technics)... So frustrating 😈 </p>",
  "messages": [
    {
      "id": "540292",
      "postDate": "05/31/2019 09:58:16",
      "content": "<p>Another frustration thread just like <a href=\"https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/93664\">https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/93664</a></p>\n\n<p>So far I've tried the most sophisticated augmentation methods, model structures and training methods but as you can see -&gt; I'm still below medal area and drifting down.\nMost importantly is that all my achievements are made with pure PyTorch and sometimes it takes 4-6 hours to train a single model.</p>\n\n<p>But why did I mention LR scheduling here? Because just by incorporating Fast.AI, which obviously does a tonne  under the hood, I'm able to achieve the same results within less than an hour of training time (and without any crazy augumentation technics)... So frustrating 😈 </p>",
      "rawMarkdown": "Another frustration thread just like https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/93664\n\nSo far I've tried the most sophisticated augmentation methods, model structures and training methods but as you can see -&gt; I'm still below medal area and drifting down.\nMost importantly is that all my achievements are made with pure PyTorch and sometimes it takes 4-6 hours to train a single model.\n\nBut why did I mention LR scheduling here? Because just by incorporating Fast.AI, which obviously does a tonne  under the hood, I'm able to achieve the same results within less than an hour of training time (and without any crazy augumentation technics)... So frustrating 😈",
      "votes": null
    },
    {
      "id": "540737",
      "postDate": "06/01/2019 01:06:08",
      "content": "<p>Yes LR/momentum scheduling is very important, but even with fastai I am finding it difficult to improve my score. So I am not sure if that is the key aspect. Anybody else care to comment?</p>",
      "rawMarkdown": "Yes LR/momentum scheduling is very important, but even with fastai I am finding it difficult to improve my score. So I am not sure if that is the key aspect. Anybody else care to comment?",
      "votes": null
    },
    {
      "id": "540983",
      "postDate": "06/01/2019 13:12:49",
      "content": "<p>I tried different learning rate schedules, for me \"reduce on plateau\" and \"one cycle\" policies give almost the same results in terms of score/convergence speed.</p>",
      "rawMarkdown": "I tried different learning rate schedules, for me \"reduce on plateau\" and \"one cycle\" policies give almost the same results in terms of score/convergence speed.",
      "votes": null
    },
    {
      "id": "541143",
      "postDate": "06/01/2019 20:59:07",
      "content": "<p>Just to continue PyTorch vs Fast.ai rant :)\nTheir cyclic lr schedulers are based on different papers (and have somewhat different implementations)\n- CyclicLR in pytorch -&gt; <a href=\"https://arxiv.org/pdf/1506.01186.pdf\">https://arxiv.org/pdf/1506.01186.pdf</a>\n<a href=\"https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#CyclicLR\">https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#CyclicLR</a></p>\n\n<ul>\n<li>OneCycleScheduler in fast.ai -&gt; <a href=\"https://arxiv.org/pdf/1803.09820.pdf\">https://arxiv.org/pdf/1803.09820.pdf</a>\n<a href=\"https://github.com/fastai/fastai/blob/79ecfd5c60087766c6f1952e674f98470d488069/fastai/callbacks/one_cycle.py#L8\">https://github.com/fastai/fastai/blob/79ecfd5c60087766c6f1952e674f98470d488069/fastai/callbacks/one_cycle.py#L8</a></li>\n</ul>\n\n<p>And at least for such non-sophisticated user as myself, the fast.ai approach looks more advance :)</p>",
      "rawMarkdown": "Just to continue PyTorch vs Fast.ai rant :)\nTheir cyclic lr schedulers are based on different papers (and have somewhat different implementations)\n- CyclicLR in pytorch -&gt; https://arxiv.org/pdf/1506.01186.pdf\nhttps://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#CyclicLR\n\n- OneCycleScheduler in fast.ai -&gt; https://arxiv.org/pdf/1803.09820.pdf\nhttps://github.com/fastai/fastai/blob/79ecfd5c60087766c6f1952e674f98470d488069/fastai/callbacks/one_cycle.py#L8\n\nAnd at least for such non-sophisticated user as myself, the fast.ai approach looks more advance :)",
      "votes": null
    },
    {
      "id": "541220",
      "postDate": "06/02/2019 03:30:29",
      "content": "<p>I think you switched the articles. These are actually almost equivalent, with fastai using the one-cycle variant.</p>",
      "rawMarkdown": "I think you switched the articles. These are actually almost equivalent, with fastai using the one-cycle variant.",
      "votes": null
    },
    {
      "id": "541221",
      "postDate": "06/02/2019 03:32:37",
      "content": "<p><a href=\"/ddanevskyi\">@ddanevskyi</a> I am using one-cycle training exclusively. In addition ReduceOnLRPlateau improved my score. But I bet you are doing something completely different than me anyway.</p>",
      "rawMarkdown": "ddanevskyi I am using one-cycle training exclusively. In addition ReduceOnLRPlateau improved my score. But I bet you are doing something completely different than me anyway.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 540737,
      "author_name": "tanlikesmath",
      "author_url": "",
      "post_date": "06/01/2019 01:06:08",
      "content": "<p>Yes LR/momentum scheduling is very important, but even with fastai I am finding it difficult to improve my score. So I am not sure if that is the key aspect. Anybody else care to comment?</p>",
      "votes": null,
      "replies": [
        {
          "id": 540983,
          "author_name": "ddanevskyi",
          "author_url": "",
          "post_date": "06/01/2019 13:12:49",
          "content": "<p>I tried different learning rate schedules, for me \"reduce on plateau\" and \"one cycle\" policies give almost the same results in terms of score/convergence speed.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 541143,
          "author_name": "vandalko",
          "author_url": "",
          "post_date": "06/01/2019 20:59:07",
          "content": "<p>Just to continue PyTorch vs Fast.ai rant :)\nTheir cyclic lr schedulers are based on different papers (and have somewhat different implementations)\n- CyclicLR in pytorch -&gt; <a href=\"https://arxiv.org/pdf/1506.01186.pdf\">https://arxiv.org/pdf/1506.01186.pdf</a>\n<a href=\"https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#CyclicLR\">https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#CyclicLR</a></p>\n\n<ul>\n<li>OneCycleScheduler in fast.ai -&gt; <a href=\"https://arxiv.org/pdf/1803.09820.pdf\">https://arxiv.org/pdf/1803.09820.pdf</a>\n<a href=\"https://github.com/fastai/fastai/blob/79ecfd5c60087766c6f1952e674f98470d488069/fastai/callbacks/one_cycle.py#L8\">https://github.com/fastai/fastai/blob/79ecfd5c60087766c6f1952e674f98470d488069/fastai/callbacks/one_cycle.py#L8</a></li>\n</ul>\n\n<p>And at least for such non-sophisticated user as myself, the fast.ai approach looks more advance :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 541220,
          "author_name": "tanlikesmath",
          "author_url": "",
          "post_date": "06/02/2019 03:30:29",
          "content": "<p>I think you switched the articles. These are actually almost equivalent, with fastai using the one-cycle variant.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 541221,
          "author_name": "tanlikesmath",
          "author_url": "",
          "post_date": "06/02/2019 03:32:37",
          "content": "<p><a href=\"/ddanevskyi\">@ddanevskyi</a> I am using one-cycle training exclusively. In addition ReduceOnLRPlateau improved my score. But I bet you are doing something completely different than me anyway.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "540292": "Another frustration thread just like https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/93664\n\nSo far I've tried the most sophisticated augmentation methods, model structures and training methods but as you can see -&gt; I'm still below medal area and drifting down.\nMost importantly is that all my achievements are made with pure PyTorch and sometimes it takes 4-6 hours to train a single model.\n\nBut why did I mention LR scheduling here? Because just by incorporating Fast.AI, which obviously does a tonne  under the hood, I'm able to achieve the same results within less than an hour of training time (and without any crazy augumentation technics)... So frustrating 😈",
    "540737": "Yes LR/momentum scheduling is very important, but even with fastai I am finding it difficult to improve my score. So I am not sure if that is the key aspect. Anybody else care to comment?",
    "540983": "I tried different learning rate schedules, for me \"reduce on plateau\" and \"one cycle\" policies give almost the same results in terms of score/convergence speed.",
    "541143": "Just to continue PyTorch vs Fast.ai rant :)\nTheir cyclic lr schedulers are based on different papers (and have somewhat different implementations)\n- CyclicLR in pytorch -&gt; https://arxiv.org/pdf/1506.01186.pdf\nhttps://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#CyclicLR\n\n- OneCycleScheduler in fast.ai -&gt; https://arxiv.org/pdf/1803.09820.pdf\nhttps://github.com/fastai/fastai/blob/79ecfd5c60087766c6f1952e674f98470d488069/fastai/callbacks/one_cycle.py#L8\n\nAnd at least for such non-sophisticated user as myself, the fast.ai approach looks more advance :)",
    "541220": "I think you switched the articles. These are actually almost equivalent, with fastai using the one-cycle variant.",
    "541221": "ddanevskyi I am using one-cycle training exclusively. In addition ReduceOnLRPlateau improved my score. But I bet you are doing something completely different than me anyway."
  },
  "source": "meta"
}