{
  "id": 179471,
  "title": "Is a LB score of -6.81 over fitting on the leaderboard?",
  "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/179471",
  "author_name": "",
  "post_date": "2020-09-02T10:56:37.740360500Z",
  "votes": 5,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Does anyone else feel like a score of -6.81 is over fitting on the public LB? Even a simple thing like changing the seed or dropout rate makes significant changes to the LB score even though CV is in the same range in all experiments.</p>",
  "messages": [
    {
      "id": "995320",
      "postDate": "09/02/2020 10:56:37",
      "content": "<p>Does anyone else feel like a score of -6.81 is over fitting on the public LB? Even a simple thing like changing the seed or dropout rate makes significant changes to the LB score even though CV is in the same range in all experiments.</p>",
      "rawMarkdown": "Does anyone else feel like a score of -6.81 is over fitting on the public LB? Even a simple thing like changing the seed or dropout rate makes significant changes to the LB score even though CV is in the same range in all experiments.",
      "votes": null
    },
    {
      "id": "996455",
      "postDate": "09/03/2020 10:02:05",
      "content": "<p>Oh, good insight for seed and dropout rate! In this case, I would trust CV more, but I think, final two submissions are always the best CV submission and the best LB submission, isn't it? </p>",
      "rawMarkdown": "Oh, good insight for seed and dropout rate! In this case, I would trust CV more, but I think, final two submissions are always the best CV submission and the best LB submission, isn't it?",
      "votes": null
    },
    {
      "id": "996542",
      "postDate": "09/03/2020 11:26:39",
      "content": "<p>Yes, when it comes to manual selection of final submission 1 would be best CV and the other best public LB. However, i find it difficult to find the best performing model based on CV since there is no stable LB and CV correlation.</p>",
      "rawMarkdown": "Yes, when it comes to manual selection of final submission 1 would be best CV and the other best public LB. However, i find it difficult to find the best performing model based on CV since there is no stable LB and CV correlation.",
      "votes": null
    },
    {
      "id": "996559",
      "postDate": "09/03/2020 11:47:45",
      "content": "<p>So, in the case you can use different seeds for improve your LB score)</p>",
      "rawMarkdown": "So, in the case you can use different seeds for improve your LB score)",
      "votes": null
    },
    {
      "id": "997132",
      "postDate": "09/03/2020 19:08:36",
      "content": "<p>Well, the public LB is based on only a few patients, so I would think it's really hard to tell whether it's overfitting or not. And the evidence you presented, that changing seed leads to significant changes in the LB would also reiterate that the public LB is a really rough indicator this time.</p>\n<p>In any case, best of luck in the competition!</p>",
      "rawMarkdown": "Well, the public LB is based on only a few patients, so I would think it's really hard to tell whether it's overfitting or not. And the evidence you presented, that changing seed leads to significant changes in the LB would also reiterate that the public LB is a really rough indicator this time.\n\nIn any case, best of luck in the competition!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 996455,
      "author_name": "koza4ukdmitrij",
      "author_url": "",
      "post_date": "09/03/2020 10:02:05",
      "content": "<p>Oh, good insight for seed and dropout rate! In this case, I would trust CV more, but I think, final two submissions are always the best CV submission and the best LB submission, isn't it? </p>",
      "votes": null,
      "replies": [
        {
          "id": 996542,
          "author_name": "yovinyahathugoda",
          "author_url": "",
          "post_date": "09/03/2020 11:26:39",
          "content": "<p>Yes, when it comes to manual selection of final submission 1 would be best CV and the other best public LB. However, i find it difficult to find the best performing model based on CV since there is no stable LB and CV correlation.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 996559,
          "author_name": "koza4ukdmitrij",
          "author_url": "",
          "post_date": "09/03/2020 11:47:45",
          "content": "<p>So, in the case you can use different seeds for improve your LB score)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 997132,
      "author_name": "douglaskgaraujo",
      "author_url": "",
      "post_date": "09/03/2020 19:08:36",
      "content": "<p>Well, the public LB is based on only a few patients, so I would think it's really hard to tell whether it's overfitting or not. And the evidence you presented, that changing seed leads to significant changes in the LB would also reiterate that the public LB is a really rough indicator this time.</p>\n<p>In any case, best of luck in the competition!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "995320": "Does anyone else feel like a score of -6.81 is over fitting on the public LB? Even a simple thing like changing the seed or dropout rate makes significant changes to the LB score even though CV is in the same range in all experiments.",
    "996455": "Oh, good insight for seed and dropout rate! In this case, I would trust CV more, but I think, final two submissions are always the best CV submission and the best LB submission, isn't it?",
    "996542": "Yes, when it comes to manual selection of final submission 1 would be best CV and the other best public LB. However, i find it difficult to find the best performing model based on CV since there is no stable LB and CV correlation.",
    "996559": "So, in the case you can use different seeds for improve your LB score)",
    "997132": "Well, the public LB is based on only a few patients, so I would think it's really hard to tell whether it's overfitting or not. And the evidence you presented, that changing seed leads to significant changes in the LB would also reiterate that the public LB is a really rough indicator this time.\n\nIn any case, best of luck in the competition!"
  },
  "source": "meta"
}