{
  "id": 104908,
  "title": "Learning Rate Tricks",
  "url": "/competitions/aptos2019-blindness-detection/discussion/104908",
  "author_name": "",
  "post_date": "2019-08-19T23:42:44.136074100Z",
  "votes": 4,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Just realized that my current best LB score was achieved using a fixed learning rate. That's probably not optimal. Have any of you experienced decent improvements using warm-up LR or cosine-annealing LR, or warm restarts, or anything else of the sort?</p>",
  "messages": [
    {
      "id": "603145",
      "postDate": "08/19/2019 23:42:44",
      "content": "<p>Just realized that my current best LB score was achieved using a fixed learning rate. That's probably not optimal. Have any of you experienced decent improvements using warm-up LR or cosine-annealing LR, or warm restarts, or anything else of the sort?</p>",
      "rawMarkdown": "Just realized that my current best LB score was achieved using a fixed learning rate. That's probably not optimal. Have any of you experienced decent improvements using warm-up LR or cosine-annealing LR, or warm restarts, or anything else of the sort?",
      "votes": null
    },
    {
      "id": "603167",
      "postDate": "08/20/2019 01:02:30",
      "content": "<p>Did you try reduce learning rate after 5 epochs of plateau on validation loss?</p>",
      "rawMarkdown": "Did you try reduce learning rate after 5 epochs of plateau on validation loss?",
      "votes": null
    },
    {
      "id": "604264",
      "postDate": "08/21/2019 07:45:02",
      "content": "<p>Using cosine LR in my model led to an increase of overfitting. My best score also was achieved with fixed LR on SGD with momentum.</p>",
      "rawMarkdown": "Using cosine LR in my model led to an increase of overfitting. My best score also was achieved with fixed LR on SGD with momentum.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 603167,
      "author_name": "quandapro",
      "author_url": "",
      "post_date": "08/20/2019 01:02:30",
      "content": "<p>Did you try reduce learning rate after 5 epochs of plateau on validation loss?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 604264,
      "author_name": "leleko",
      "author_url": "",
      "post_date": "08/21/2019 07:45:02",
      "content": "<p>Using cosine LR in my model led to an increase of overfitting. My best score also was achieved with fixed LR on SGD with momentum.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "603145": "Just realized that my current best LB score was achieved using a fixed learning rate. That's probably not optimal. Have any of you experienced decent improvements using warm-up LR or cosine-annealing LR, or warm restarts, or anything else of the sort?",
    "603167": "Did you try reduce learning rate after 5 epochs of plateau on validation loss?",
    "604264": "Using cosine LR in my model led to an increase of overfitting. My best score also was achieved with fixed LR on SGD with momentum."
  },
  "source": "meta"
}