{
  "id": 21164,
  "title": "Public/Private split",
  "url": "/competitions/state-farm-distracted-driver-detection/discussion/21164",
  "author_name": "",
  "post_date": "2016-05-24T03:06:30.107Z",
  "votes": null,
  "comment_count": 6,
  "views": 1115,
  "content": "<p>Is data split between Public and Private Leaderboars based on drivers or it is random split?</p>",
  "messages": [
    {
      "id": "121112",
      "postDate": "05/24/2016 03:06:30",
      "content": "<p>Is data split between Public and Private Leaderboars based on drivers or it is random split?</p>",
      "rawMarkdown": "Is data split between Public and Private Leaderboars based on drivers or it is random split?",
      "votes": null
    },
    {
      "id": "121118",
      "postDate": "05/24/2016 04:32:15",
      "content": "<blockquote>\n  <p>The train and test data are split on the drivers, such that one driver\n  can only appear on either train or test set.</p>\n</blockquote>\n\n<p><a href=\"https://www.kaggle.com/c/state-farm-distracted-driver-detection/data\">https://www.kaggle.com/c/state-farm-distracted-driver-detection/data</a></p>",
      "rawMarkdown": "> The train and test data are split on the drivers, such that one driver\r\n> can only appear on either train or test set.\r\n\r\nhttps://www.kaggle.com/c/state-farm-distracted-driver-detection/data",
      "votes": null
    },
    {
      "id": "121162",
      "postDate": "05/24/2016 17:00:07",
      "content": "<p>[quote=xyz;121118]</p>\n\n<blockquote>\n  <p>The train and test data are split on the drivers, such that one driver\n  can only appear on either train or test set.</p>\n</blockquote>\n\n<p><a href=\"https://www.kaggle.com/c/state-farm-distracted-driver-detection/data\">https://www.kaggle.com/c/state-farm-distracted-driver-detection/data</a></p>\n\n<p>[/quote]</p>\n\n<p>This part I understand. But my question is how test set is splitted between Private and Public leaderboards.</p>",
      "rawMarkdown": "[quote=xyz;121118]\r\n\r\n> The train and test data are split on the drivers, such that one driver\r\n> can only appear on either train or test set.\r\n\r\nhttps://www.kaggle.com/c/state-farm-distracted-driver-detection/data\r\n\r\n[/quote]\r\n\r\nThis part I understand. But my question is how test set is splitted between Private and Public leaderboards.",
      "votes": null
    },
    {
      "id": "121163",
      "postDate": "05/24/2016 17:08:23",
      "content": "<p>[quote=Vladimir Iglovikov;121162]</p>\n\n<p>This part I understand. But my question is show test set is splitted between Private and Public leaderboards.</p>\n\n<p>[/quote]</p>\n\n<p>You could probably engineer some LB probes to get an idea.</p>",
      "rawMarkdown": "[quote=Vladimir Iglovikov;121162]\r\n\r\nThis part I understand. But my question is show test set is splitted between Private and Public leaderboards.\r\n\r\n[/quote]\r\n\r\nYou could probably engineer some LB probes to get an idea.",
      "votes": null
    },
    {
      "id": "121247",
      "postDate": "05/25/2016 06:14:29",
      "content": "<p>I am asking because my cross validation score is much worse than LB score, so I am worried that if Public and private are splitted based on drivers and not on random, thus Public LB may be bad indicator of the model performance.</p>\n\n<p>Say, my last submission:</p>\n\n<ul>\n<li>Local cv = 0.578 </li>\n<li>Public LB = 0.269</li>\n</ul>\n\n<p>And this mismatch gives me some strange feeling that I have no clue what Private LB may give...</p>",
      "rawMarkdown": "I am asking because my cross validation score is much worse than LB score, so I am worried that if Public and private are splitted based on drivers and not on random, thus Public LB may be bad indicator of the model performance.\r\n\r\nSay, my last submission:\r\n\r\n - Local cv = 0.578 \r\n - Public LB = 0.269\r\n\r\nAnd this mismatch gives me some strange feeling that I have no clue what Private LB may give...",
      "votes": null
    },
    {
      "id": "121293",
      "postDate": "05/25/2016 12:15:44",
      "content": "<p>I also wonder the same. If it is a random split, then the current public leader board will be a very accurate indicate of the private LB. Any over fit to the public LB also fits the private LB.</p>\n\n<p>[quote=Vladimir Iglovikov;121112]</p>\n\n<p>Is data split between Public and Private Leaderboars based on drivers or it is random split?</p>\n\n<p>[/quote]</p>",
      "rawMarkdown": "I also wonder the same. If it is a random split, then the current public leader board will be a very accurate indicate of the private LB. Any over fit to the public LB also fits the private LB.\r\n\r\n[quote=Vladimir Iglovikov;121112]\r\n\r\nIs data split between Public and Private Leaderboars based on drivers or it is random split?\r\n\r\n[/quote]",
      "votes": null
    },
    {
      "id": "121328",
      "postDate": "05/25/2016 17:50:47",
      "content": "<p>The public/private LB are randomly split. I can't comment on the overfitting situation :) </p>",
      "rawMarkdown": "The public/private LB are randomly split. I can't comment on the overfitting situation :)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 121118,
      "author_name": "zhugds",
      "author_url": "",
      "post_date": "05/24/2016 04:32:15",
      "content": "<blockquote>\n  <p>The train and test data are split on the drivers, such that one driver\n  can only appear on either train or test set.</p>\n</blockquote>\n\n<p><a href=\"https://www.kaggle.com/c/state-farm-distracted-driver-detection/data\">https://www.kaggle.com/c/state-farm-distracted-driver-detection/data</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121162,
      "author_name": "iglovikov",
      "author_url": "",
      "post_date": "05/24/2016 17:00:07",
      "content": "<p>[quote=xyz;121118]</p>\n\n<blockquote>\n  <p>The train and test data are split on the drivers, such that one driver\n  can only appear on either train or test set.</p>\n</blockquote>\n\n<p><a href=\"https://www.kaggle.com/c/state-farm-distracted-driver-detection/data\">https://www.kaggle.com/c/state-farm-distracted-driver-detection/data</a></p>\n\n<p>[/quote]</p>\n\n<p>This part I understand. But my question is how test set is splitted between Private and Public leaderboards.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121163,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "05/24/2016 17:08:23",
      "content": "<p>[quote=Vladimir Iglovikov;121162]</p>\n\n<p>This part I understand. But my question is show test set is splitted between Private and Public leaderboards.</p>\n\n<p>[/quote]</p>\n\n<p>You could probably engineer some LB probes to get an idea.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121247,
      "author_name": "iglovikov",
      "author_url": "",
      "post_date": "05/25/2016 06:14:29",
      "content": "<p>I am asking because my cross validation score is much worse than LB score, so I am worried that if Public and private are splitted based on drivers and not on random, thus Public LB may be bad indicator of the model performance.</p>\n\n<p>Say, my last submission:</p>\n\n<ul>\n<li>Local cv = 0.578 </li>\n<li>Public LB = 0.269</li>\n</ul>\n\n<p>And this mismatch gives me some strange feeling that I have no clue what Private LB may give...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121293,
      "author_name": "woshialex",
      "author_url": "",
      "post_date": "05/25/2016 12:15:44",
      "content": "<p>I also wonder the same. If it is a random split, then the current public leader board will be a very accurate indicate of the private LB. Any over fit to the public LB also fits the private LB.</p>\n\n<p>[quote=Vladimir Iglovikov;121112]</p>\n\n<p>Is data split between Public and Private Leaderboars based on drivers or it is random split?</p>\n\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121328,
      "author_name": "wendykan",
      "author_url": "",
      "post_date": "05/25/2016 17:50:47",
      "content": "<p>The public/private LB are randomly split. I can't comment on the overfitting situation :) </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "121112": "Is data split between Public and Private Leaderboars based on drivers or it is random split?",
    "121118": "> The train and test data are split on the drivers, such that one driver\r\n> can only appear on either train or test set.\r\n\r\nhttps://www.kaggle.com/c/state-farm-distracted-driver-detection/data",
    "121162": "[quote=xyz;121118]\r\n\r\n> The train and test data are split on the drivers, such that one driver\r\n> can only appear on either train or test set.\r\n\r\nhttps://www.kaggle.com/c/state-farm-distracted-driver-detection/data\r\n\r\n[/quote]\r\n\r\nThis part I understand. But my question is how test set is splitted between Private and Public leaderboards.",
    "121163": "[quote=Vladimir Iglovikov;121162]\r\n\r\nThis part I understand. But my question is show test set is splitted between Private and Public leaderboards.\r\n\r\n[/quote]\r\n\r\nYou could probably engineer some LB probes to get an idea.",
    "121247": "I am asking because my cross validation score is much worse than LB score, so I am worried that if Public and private are splitted based on drivers and not on random, thus Public LB may be bad indicator of the model performance.\r\n\r\nSay, my last submission:\r\n\r\n - Local cv = 0.578 \r\n - Public LB = 0.269\r\n\r\nAnd this mismatch gives me some strange feeling that I have no clue what Private LB may give...",
    "121293": "I also wonder the same. If it is a random split, then the current public leader board will be a very accurate indicate of the private LB. Any over fit to the public LB also fits the private LB.\r\n\r\n[quote=Vladimir Iglovikov;121112]\r\n\r\nIs data split between Public and Private Leaderboars based on drivers or it is random split?\r\n\r\n[/quote]",
    "121328": "The public/private LB are randomly split. I can't comment on the overfitting situation :)"
  },
  "source": "meta"
}