{
  "id": 17032,
  "title": "Internal Validation Mismatch",
  "url": "/competitions/noaa-right-whale-recognition/discussion/17032",
  "author_name": "",
  "post_date": "2015-10-15T23:43:07.827Z",
  "votes": -1,
  "comment_count": 2,
  "views": 703,
  "content": "<p>I've noticed that all my submissions to the leaderboard perform worse than estimated on my internal validation logloss. Has anyone else noticed a similar difference in leaderboard validation vs internal validation, or is it perhaps my own sampling/loss calculation that is bugged?</p>",
  "messages": [
    {
      "id": "96386",
      "postDate": "10/15/2015 23:43:07",
      "content": "<p>I've noticed that all my submissions to the leaderboard perform worse than estimated on my internal validation logloss. Has anyone else noticed a similar difference in leaderboard validation vs internal validation, or is it perhaps my own sampling/loss calculation that is bugged?</p>",
      "rawMarkdown": "I've noticed that all my submissions to the leaderboard perform worse than estimated on my internal validation logloss. Has anyone else noticed a similar difference in leaderboard validation vs internal validation, or is it perhaps my own sampling/loss calculation that is bugged?",
      "votes": null
    },
    {
      "id": "96567",
      "postDate": "10/18/2015 13:34:25",
      "content": "<p>Hi MrTwiggy,</p>\n\n<p>My leaderboard score is a little bit worse than my validation score.<br>\nI think the way to make validation set is important.<br>\nIn this competition, only several images are given for each class.</p>",
      "rawMarkdown": "Hi MrTwiggy,\r\n\r\nMy leaderboard score is a little bit worse than my validation score.<br>\r\nI think the way to make validation set is important.<br>\r\nIn this competition, only several images are given for each class.",
      "votes": null
    },
    {
      "id": "96932",
      "postDate": "10/21/2015 12:59:12",
      "content": "<p>My internal validation is significantly worse than my leaderboard score. toshi_k has it right - given the data, the validation set needs to be constructed carefully for the model to learn well.</p>",
      "rawMarkdown": "My internal validation is significantly worse than my leaderboard score. toshi_k has it right - given the data, the validation set needs to be constructed carefully for the model to learn well.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 96567,
      "author_name": "toshik",
      "author_url": "",
      "post_date": "10/18/2015 13:34:25",
      "content": "<p>Hi MrTwiggy,</p>\n\n<p>My leaderboard score is a little bit worse than my validation score.<br>\nI think the way to make validation set is important.<br>\nIn this competition, only several images are given for each class.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 96932,
      "author_name": "jwjohnson314",
      "author_url": "",
      "post_date": "10/21/2015 12:59:12",
      "content": "<p>My internal validation is significantly worse than my leaderboard score. toshi_k has it right - given the data, the validation set needs to be constructed carefully for the model to learn well.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "96386": "I've noticed that all my submissions to the leaderboard perform worse than estimated on my internal validation logloss. Has anyone else noticed a similar difference in leaderboard validation vs internal validation, or is it perhaps my own sampling/loss calculation that is bugged?",
    "96567": "Hi MrTwiggy,\r\n\r\nMy leaderboard score is a little bit worse than my validation score.<br>\r\nI think the way to make validation set is important.<br>\r\nIn this competition, only several images are given for each class.",
    "96932": "My internal validation is significantly worse than my leaderboard score. toshi_k has it right - given the data, the validation set needs to be constructed carefully for the model to learn well."
  },
  "source": "meta"
}