{
  "id": 313437,
  "title": "Test set different than train set?",
  "url": "/competitions/ultra-mnist/discussion/313437",
  "author_name": "",
  "post_date": "2022-03-17T05:25:41.369749500Z",
  "votes": 7,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I noticed that my local validation score is different than leaderboard score.<br>\nWould that suggest that test set is different than train set?</p>\n<p>local validation: 0.47<br>\nlb score: 0.21</p>",
  "messages": [
    {
      "id": "1725444",
      "postDate": "03/17/2022 05:25:41",
      "content": "<p>I noticed that my local validation score is different than leaderboard score.<br>\nWould that suggest that test set is different than train set?</p>\n<p>local validation: 0.47<br>\nlb score: 0.21</p>",
      "rawMarkdown": "I noticed that my local validation score is different than leaderboard score.\nWould that suggest that test set is different than train set?\n\nlocal validation: 0.47\nlb score: 0.21",
      "votes": null
    },
    {
      "id": "1725515",
      "postDate": "03/17/2022 07:06:13",
      "content": "<p>Saw similar results so far, but I could say cv and lb are correlating.</p>\n<p>cv: 0.74 --&gt; lb: 0.54<br>\ncv: 0.45 ---&gt; lb: 0.32<br>\ncv: 0.35 ---&gt; lb: 0.23</p>",
      "rawMarkdown": "Saw similar results so far, but I could say cv and lb are correlating.\n\ncv: 0.74 --> lb: 0.54\ncv: 0.45 ---> lb: 0.32\ncv: 0.35 ---> lb: 0.23",
      "votes": null
    },
    {
      "id": "1725683",
      "postDate": "03/17/2022 10:16:10",
      "content": "<p>Don't know if it's about train vs test distribution difference, but I observed the same: resnet50 + images rescaled to 256x256, cv: 0.25, lb: 0.09</p>",
      "rawMarkdown": "Don't know if it's about train vs test distribution difference, but I observed the same: resnet50 + images rescaled to 256x256, cv: 0.25, lb: 0.09",
      "votes": null
    },
    {
      "id": "1731125",
      "postDate": "03/22/2022 02:21:09",
      "content": "<p>Intersting question. Are you making sure that your validation dataset is having uniform distribution? Because given that evaluation uses simple accuracy score, that test set data distribution must be uniform.</p>\n<p><a href=\"https://www.kaggle.com/miwojc\" target=\"_blank\">@miwojc</a> <a href=\"https://www.kaggle.com/konradb\" target=\"_blank\">@konradb</a> <a href=\"https://www.kaggle.com/nischaydnk\" target=\"_blank\">@nischaydnk</a> </p>",
      "rawMarkdown": "Intersting question. Are you making sure that your validation dataset is having uniform distribution? Because given that evaluation uses simple accuracy score, that test set data distribution must be uniform.\n\n@miwojc @konradb @nischaydnk",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1725515,
      "author_name": "nischaydnk",
      "author_url": "",
      "post_date": "03/17/2022 07:06:13",
      "content": "<p>Saw similar results so far, but I could say cv and lb are correlating.</p>\n<p>cv: 0.74 --&gt; lb: 0.54<br>\ncv: 0.45 ---&gt; lb: 0.32<br>\ncv: 0.35 ---&gt; lb: 0.23</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1725683,
      "author_name": "konradb",
      "author_url": "",
      "post_date": "03/17/2022 10:16:10",
      "content": "<p>Don't know if it's about train vs test distribution difference, but I observed the same: resnet50 + images rescaled to 256x256, cv: 0.25, lb: 0.09</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1731125,
      "author_name": "l0new0lf",
      "author_url": "",
      "post_date": "03/22/2022 02:21:09",
      "content": "<p>Intersting question. Are you making sure that your validation dataset is having uniform distribution? Because given that evaluation uses simple accuracy score, that test set data distribution must be uniform.</p>\n<p><a href=\"https://www.kaggle.com/miwojc\" target=\"_blank\">@miwojc</a> <a href=\"https://www.kaggle.com/konradb\" target=\"_blank\">@konradb</a> <a href=\"https://www.kaggle.com/nischaydnk\" target=\"_blank\">@nischaydnk</a> </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1725444": "I noticed that my local validation score is different than leaderboard score.\nWould that suggest that test set is different than train set?\n\nlocal validation: 0.47\nlb score: 0.21",
    "1725515": "Saw similar results so far, but I could say cv and lb are correlating.\n\ncv: 0.74 --> lb: 0.54\ncv: 0.45 ---> lb: 0.32\ncv: 0.35 ---> lb: 0.23",
    "1725683": "Don't know if it's about train vs test distribution difference, but I observed the same: resnet50 + images rescaled to 256x256, cv: 0.25, lb: 0.09",
    "1731125": "Intersting question. Are you making sure that your validation dataset is having uniform distribution? Because given that evaluation uses simple accuracy score, that test set data distribution must be uniform.\n\n@miwojc @konradb @nischaydnk"
  },
  "source": "meta"
}