{
  "id": 20025,
  "title": "CV vs LB",
  "url": "/competitions/state-farm-distracted-driver-detection/discussion/20025",
  "author_name": "",
  "post_date": "2016-04-09T03:58:12.330Z",
  "votes": null,
  "comment_count": 11,
  "views": 2495,
  "content": "<p>I am doing CV by keeping 2 driver images in validation and rest in train</p>\n\n<p>CV - 0.25\nLB - 1.88</p>\n\n<p>Can anyone please share what approach of CV is working the best.</p>",
  "messages": [
    {
      "id": "114302",
      "postDate": "04/09/2016 03:58:12",
      "content": "<p>I am doing CV by keeping 2 driver images in validation and rest in train</p>\n\n<p>CV - 0.25\nLB - 1.88</p>\n\n<p>Can anyone please share what approach of CV is working the best.</p>",
      "rawMarkdown": "I am doing CV by keeping 2 driver images in validation and rest in train\r\n\r\nCV - 0.25\r\nLB - 1.88\r\n\r\nCan anyone please share what approach of CV is working the best.",
      "votes": null
    },
    {
      "id": "114354",
      "postDate": "04/09/2016 17:43:36",
      "content": "<p>I've got CV - 0.08, LB - 0.65 using one (p002) driver images for CV.</p>",
      "rawMarkdown": "I've got CV - 0.08, LB - 0.65 using one (p002) driver images for CV.",
      "votes": null
    },
    {
      "id": "114356",
      "postDate": "04/09/2016 18:00:28",
      "content": "<p>[quote=Mike;114354]</p>\n\n<p>I've got CV - 0.08, LB - 0.65 using one (p002) driver images for CV.</p>\n\n<p>[/quote]\nSo you hold p002 completely out of training process, i.e. no k-folds?</p>",
      "rawMarkdown": "[quote=Mike;114354]\r\n\r\nI've got CV - 0.08, LB - 0.65 using one (p002) driver images for CV.\r\n\r\n[/quote]\r\nSo you hold p002 completely out of training process, i.e. no k-folds?",
      "votes": null
    },
    {
      "id": "114553",
      "postDate": "04/11/2016 21:33:29",
      "content": "<p>With 5-fold CV, my local val. loss was ~0.7, which led to ~0.63 on LB. So I guess I'm getting there.</p>",
      "rawMarkdown": "With 5-fold CV, my local val. loss was ~0.7, which led to ~0.63 on LB. So I guess I'm getting there.",
      "votes": null
    },
    {
      "id": "114570",
      "postDate": "04/12/2016 01:43:46",
      "content": "<p>[quote=Marko Jocic;114553]</p>\n\n<p>With 5-fold CV, my local val. loss was ~0.7, which led to ~0.63 on LB. So I guess I'm getting there.</p>\n\n<p>[/quote]</p>\n\n<p>Thanks Marko.</p>\n\n<p>Are you doing 5-fold CV based on classes or unique drivers ?</p>",
      "rawMarkdown": "[quote=Marko Jocic;114553]\r\n\r\nWith 5-fold CV, my local val. loss was ~0.7, which led to ~0.63 on LB. So I guess I'm getting there.\r\n\r\n[/quote]\r\n\r\nThanks Marko.\r\n\r\nAre you doing 5-fold CV based on classes or unique drivers ?",
      "votes": null
    },
    {
      "id": "114694",
      "postDate": "04/12/2016 22:42:01",
      "content": "<p>I'm also having problems regarding cross-validation. I can get up to 0.03x 3-Fold CV score versus 2.2x LB score, which is quite strange. I'm randomly splitting the training set when CVing, which maybe is causing this. I'll try this approach of splitting based on unique drivers next.</p>",
      "rawMarkdown": "I'm also having problems regarding cross-validation. I can get up to 0.03x 3-Fold CV score versus 2.2x LB score, which is quite strange. I'm randomly splitting the training set when CVing, which maybe is causing this. I'll try this approach of splitting based on unique drivers next.",
      "votes": null
    },
    {
      "id": "114695",
      "postDate": "04/12/2016 22:56:17",
      "content": "<p>[quote=pennacchio;114694]\nI'm also having problems regarding cross-validation. I can get up to 0.03x 3-Fold CV score versus 2.2x LB score, which is quite strange. I'm randomly splitting the training set when CVing, which maybe is causing this. I'll try this approach of splitting based on unique drivers next.\n[/quote]</p>\n\n<p>A fully random split is useless, most CNNs seem to learn about individual drivers very well, so there is lots of overfitting to the CV set. It would seem to be quite easy to train a CNN to recognise the classes of a specific driver (with same clothes, hairstyle etc in same car). </p>\n\n<p>CV split by driver works OK-ish for me - e.g. CV 1.5x, LB 1.3x. </p>\n\n<p>Also of course the test set is split by driver, all the drivers in it are not in the training set - and it is generally good advice to follow how the test set is split when doing CV.</p>",
      "rawMarkdown": "[quote=pennacchio;114694]\r\nI'm also having problems regarding cross-validation. I can get up to 0.03x 3-Fold CV score versus 2.2x LB score, which is quite strange. I'm randomly splitting the training set when CVing, which maybe is causing this. I'll try this approach of splitting based on unique drivers next.\r\n[/quote]\r\n\r\nA fully random split is useless, most CNNs seem to learn about individual drivers very well, so there is lots of overfitting to the CV set. It would seem to be quite easy to train a CNN to recognise the classes of a specific driver (with same clothes, hairstyle etc in same car). \r\n\r\nCV split by driver works OK-ish for me - e.g. CV 1.5x, LB 1.3x. \r\n\r\nAlso of course the test set is split by driver, all the drivers in it are not in the training set - and it is generally good advice to follow how the test set is split when doing CV.",
      "votes": null
    },
    {
      "id": "124703",
      "postDate": "06/21/2016 12:59:44",
      "content": "<p>LB------leader board?</p>",
      "rawMarkdown": "LB------leader board?",
      "votes": null
    },
    {
      "id": "125178",
      "postDate": "06/27/2016 07:19:21",
      "content": "<p>[quote=panzer;124703]</p>\n\n<p>LB------leader board?</p>\n\n<p>[/quote]</p>\n\n<p>I guess so</p>",
      "rawMarkdown": "[quote=panzer;124703]\r\n\r\nLB------leader board?\r\n\r\n[/quote]\r\n\r\nI guess so",
      "votes": null
    },
    {
      "id": "125287",
      "postDate": "06/28/2016 12:43:54",
      "content": "<p>Hi everyone,</p>\n\n<p>My scores are as follows now.\n$$\n\\begin{array}{|c|c|}\n\\hline \\bf{data} &amp; \\bf{score} \\\\ \\hline\n\\hline \\rm{Training} &amp; 0.0534 \\\\ \\hline\n\\hline \\rm{Validation} &amp; 0.2773 \\\\ \\hline\n\\hline \\rm{Public \\space Leaderboard} &amp; 0.1477 \\\\ \\hline\n\\end{array}\n$$\nThe way to split data for validation is not fully random split.<br></p>",
      "rawMarkdown": "Hi everyone,\r\n\r\nMy scores are as follows now.\r\n$$\r\n\\begin{array}{|c|c|}\r\n\\hline \\bf{data} & \\bf{score} \\\\ \\hline\r\n\\hline \\rm{Training} & 0.0534 \\\\ \\hline\r\n\\hline \\rm{Validation} & 0.2773 \\\\ \\hline\r\n\\hline \\rm{Public \\space Leaderboard} & 0.1477 \\\\ \\hline\r\n\\end{array}\r\n$$\r\nThe way to split data for validation is not fully random split.<br>",
      "votes": null
    },
    {
      "id": "125301",
      "postDate": "06/28/2016 13:57:38",
      "content": "<p>@toshi_k, can you give a hint on how to split the data? I have been spending the past few days to make the validation score close to the LB. But my results are sometimes unpredictable. Currently, LB = validation + 0.16. I attached my results.  Can you suggest how to improve this? Thanks a lot!</p>\n\n<p>[quote=toshi_k;125287]</p>\n\n<p>Hi everyone,</p>\n\n<p>My scores are as follows now.\n$$\n\\begin{array}{|c|c|}\n\\hline \\bf{data} &amp; \\bf{score} \\\\ \\hline\n\\hline \\rm{Training} &amp; 0.0534 \\\\ \\hline\n\\hline \\rm{Validation} &amp; 0.2773 \\\\ \\hline\n\\hline \\rm{Public \\space Leaderboard} &amp; 0.1477 \\\\ \\hline\n\\end{array}\n$$\nThe way to split data for validation is not fully random split.<br></p>\n\n<p>[/quote]</p>",
      "rawMarkdown": "toshi_k, can you give a hint on how to split the data? I have been spending the past few days to make the validation score close to the LB. But my results are sometimes unpredictable. Currently, LB = validation + 0.16. I attached my results.  Can you suggest how to improve this? Thanks a lot!\r\n\r\n[quote=toshi_k;125287]\r\n\r\nHi everyone,\r\n\r\nMy scores are as follows now.\r\n$$\r\n\\begin{array}{|c|c|}\r\n\\hline \\bf{data} & \\bf{score} \\\\ \\hline\r\n\\hline \\rm{Training} & 0.0534 \\\\ \\hline\r\n\\hline \\rm{Validation} & 0.2773 \\\\ \\hline\r\n\\hline \\rm{Public \\space Leaderboard} & 0.1477 \\\\ \\hline\r\n\\end{array}\r\n$$\r\nThe way to split data for validation is not fully random split.<br>\r\n\r\n[/quote]",
      "votes": null
    },
    {
      "id": "125303",
      "postDate": "06/28/2016 14:03:38",
      "content": "<p>By the way , let's say training set set is {p1,c1} ...{p1,c10} .... {p26,c1}...{p26,c10}.\nHere p1,... p26 are 26 drivers. c1,...,10 are 10 actions.</p>\n\n<p>Most people split train/validation by drivers. Anyone split like this below? If so, what is the results?</p>\n\n<ul>\n<li>Validation : {some driver, some action}\n                   e.g. {p1,c1}, {p5,c3} </li>\n<li>Training:  all except {p,c} in  validation</li>\n</ul>",
      "rawMarkdown": "By the way , let's say training set set is {p1,c1} ...{p1,c10} .... {p26,c1}...{p26,c10}.\r\nHere p1,... p26 are 26 drivers. c1,...,10 are 10 actions.\r\n\r\nMost people split train/validation by drivers. Anyone split like this below? If so, what is the results?\r\n\r\n - Validation : {some driver, some action}\r\n                       e.g. {p1,c1}, {p5,c3} \r\n - Training:  all except {p,c} in  validation",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 114354,
      "author_name": "zerrxy",
      "author_url": "",
      "post_date": "04/09/2016 17:43:36",
      "content": "<p>I've got CV - 0.08, LB - 0.65 using one (p002) driver images for CV.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 114356,
      "author_name": "inoryy",
      "author_url": "",
      "post_date": "04/09/2016 18:00:28",
      "content": "<p>[quote=Mike;114354]</p>\n\n<p>I've got CV - 0.08, LB - 0.65 using one (p002) driver images for CV.</p>\n\n<p>[/quote]\nSo you hold p002 completely out of training process, i.e. no k-folds?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 114553,
      "author_name": "jocicmarko",
      "author_url": "",
      "post_date": "04/11/2016 21:33:29",
      "content": "<p>With 5-fold CV, my local val. loss was ~0.7, which led to ~0.63 on LB. So I guess I'm getting there.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 114570,
      "author_name": "thakurrajanand",
      "author_url": "",
      "post_date": "04/12/2016 01:43:46",
      "content": "<p>[quote=Marko Jocic;114553]</p>\n\n<p>With 5-fold CV, my local val. loss was ~0.7, which led to ~0.63 on LB. So I guess I'm getting there.</p>\n\n<p>[/quote]</p>\n\n<p>Thanks Marko.</p>\n\n<p>Are you doing 5-fold CV based on classes or unique drivers ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 114694,
      "author_name": "pennacchio",
      "author_url": "",
      "post_date": "04/12/2016 22:42:01",
      "content": "<p>I'm also having problems regarding cross-validation. I can get up to 0.03x 3-Fold CV score versus 2.2x LB score, which is quite strange. I'm randomly splitting the training set when CVing, which maybe is causing this. I'll try this approach of splitting based on unique drivers next.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 114695,
      "author_name": "slobo777",
      "author_url": "",
      "post_date": "04/12/2016 22:56:17",
      "content": "<p>[quote=pennacchio;114694]\nI'm also having problems regarding cross-validation. I can get up to 0.03x 3-Fold CV score versus 2.2x LB score, which is quite strange. I'm randomly splitting the training set when CVing, which maybe is causing this. I'll try this approach of splitting based on unique drivers next.\n[/quote]</p>\n\n<p>A fully random split is useless, most CNNs seem to learn about individual drivers very well, so there is lots of overfitting to the CV set. It would seem to be quite easy to train a CNN to recognise the classes of a specific driver (with same clothes, hairstyle etc in same car). </p>\n\n<p>CV split by driver works OK-ish for me - e.g. CV 1.5x, LB 1.3x. </p>\n\n<p>Also of course the test set is split by driver, all the drivers in it are not in the training set - and it is generally good advice to follow how the test set is split when doing CV.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124703,
      "author_name": "lasshpanzer",
      "author_url": "",
      "post_date": "06/21/2016 12:59:44",
      "content": "<p>LB------leader board?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125178,
      "author_name": "nanoix9",
      "author_url": "",
      "post_date": "06/27/2016 07:19:21",
      "content": "<p>[quote=panzer;124703]</p>\n\n<p>LB------leader board?</p>\n\n<p>[/quote]</p>\n\n<p>I guess so</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125287,
      "author_name": "toshik",
      "author_url": "",
      "post_date": "06/28/2016 12:43:54",
      "content": "<p>Hi everyone,</p>\n\n<p>My scores are as follows now.\n$$\n\\begin{array}{|c|c|}\n\\hline \\bf{data} &amp; \\bf{score} \\\\ \\hline\n\\hline \\rm{Training} &amp; 0.0534 \\\\ \\hline\n\\hline \\rm{Validation} &amp; 0.2773 \\\\ \\hline\n\\hline \\rm{Public \\space Leaderboard} &amp; 0.1477 \\\\ \\hline\n\\end{array}\n$$\nThe way to split data for validation is not fully random split.<br></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125301,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "06/28/2016 13:57:38",
      "content": "<p>@toshi_k, can you give a hint on how to split the data? I have been spending the past few days to make the validation score close to the LB. But my results are sometimes unpredictable. Currently, LB = validation + 0.16. I attached my results.  Can you suggest how to improve this? Thanks a lot!</p>\n\n<p>[quote=toshi_k;125287]</p>\n\n<p>Hi everyone,</p>\n\n<p>My scores are as follows now.\n$$\n\\begin{array}{|c|c|}\n\\hline \\bf{data} &amp; \\bf{score} \\\\ \\hline\n\\hline \\rm{Training} &amp; 0.0534 \\\\ \\hline\n\\hline \\rm{Validation} &amp; 0.2773 \\\\ \\hline\n\\hline \\rm{Public \\space Leaderboard} &amp; 0.1477 \\\\ \\hline\n\\end{array}\n$$\nThe way to split data for validation is not fully random split.<br></p>\n\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125303,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "06/28/2016 14:03:38",
      "content": "<p>By the way , let's say training set set is {p1,c1} ...{p1,c10} .... {p26,c1}...{p26,c10}.\nHere p1,... p26 are 26 drivers. c1,...,10 are 10 actions.</p>\n\n<p>Most people split train/validation by drivers. Anyone split like this below? If so, what is the results?</p>\n\n<ul>\n<li>Validation : {some driver, some action}\n                   e.g. {p1,c1}, {p5,c3} </li>\n<li>Training:  all except {p,c} in  validation</li>\n</ul>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "114302": "I am doing CV by keeping 2 driver images in validation and rest in train\r\n\r\nCV - 0.25\r\nLB - 1.88\r\n\r\nCan anyone please share what approach of CV is working the best.",
    "114354": "I've got CV - 0.08, LB - 0.65 using one (p002) driver images for CV.",
    "114356": "[quote=Mike;114354]\r\n\r\nI've got CV - 0.08, LB - 0.65 using one (p002) driver images for CV.\r\n\r\n[/quote]\r\nSo you hold p002 completely out of training process, i.e. no k-folds?",
    "114553": "With 5-fold CV, my local val. loss was ~0.7, which led to ~0.63 on LB. So I guess I'm getting there.",
    "114570": "[quote=Marko Jocic;114553]\r\n\r\nWith 5-fold CV, my local val. loss was ~0.7, which led to ~0.63 on LB. So I guess I'm getting there.\r\n\r\n[/quote]\r\n\r\nThanks Marko.\r\n\r\nAre you doing 5-fold CV based on classes or unique drivers ?",
    "114694": "I'm also having problems regarding cross-validation. I can get up to 0.03x 3-Fold CV score versus 2.2x LB score, which is quite strange. I'm randomly splitting the training set when CVing, which maybe is causing this. I'll try this approach of splitting based on unique drivers next.",
    "114695": "[quote=pennacchio;114694]\r\nI'm also having problems regarding cross-validation. I can get up to 0.03x 3-Fold CV score versus 2.2x LB score, which is quite strange. I'm randomly splitting the training set when CVing, which maybe is causing this. I'll try this approach of splitting based on unique drivers next.\r\n[/quote]\r\n\r\nA fully random split is useless, most CNNs seem to learn about individual drivers very well, so there is lots of overfitting to the CV set. It would seem to be quite easy to train a CNN to recognise the classes of a specific driver (with same clothes, hairstyle etc in same car). \r\n\r\nCV split by driver works OK-ish for me - e.g. CV 1.5x, LB 1.3x. \r\n\r\nAlso of course the test set is split by driver, all the drivers in it are not in the training set - and it is generally good advice to follow how the test set is split when doing CV.",
    "124703": "LB------leader board?",
    "125178": "[quote=panzer;124703]\r\n\r\nLB------leader board?\r\n\r\n[/quote]\r\n\r\nI guess so",
    "125287": "Hi everyone,\r\n\r\nMy scores are as follows now.\r\n$$\r\n\\begin{array}{|c|c|}\r\n\\hline \\bf{data} & \\bf{score} \\\\ \\hline\r\n\\hline \\rm{Training} & 0.0534 \\\\ \\hline\r\n\\hline \\rm{Validation} & 0.2773 \\\\ \\hline\r\n\\hline \\rm{Public \\space Leaderboard} & 0.1477 \\\\ \\hline\r\n\\end{array}\r\n$$\r\nThe way to split data for validation is not fully random split.<br>",
    "125301": "toshi_k, can you give a hint on how to split the data? I have been spending the past few days to make the validation score close to the LB. But my results are sometimes unpredictable. Currently, LB = validation + 0.16. I attached my results.  Can you suggest how to improve this? Thanks a lot!\r\n\r\n[quote=toshi_k;125287]\r\n\r\nHi everyone,\r\n\r\nMy scores are as follows now.\r\n$$\r\n\\begin{array}{|c|c|}\r\n\\hline \\bf{data} & \\bf{score} \\\\ \\hline\r\n\\hline \\rm{Training} & 0.0534 \\\\ \\hline\r\n\\hline \\rm{Validation} & 0.2773 \\\\ \\hline\r\n\\hline \\rm{Public \\space Leaderboard} & 0.1477 \\\\ \\hline\r\n\\end{array}\r\n$$\r\nThe way to split data for validation is not fully random split.<br>\r\n\r\n[/quote]",
    "125303": "By the way , let's say training set set is {p1,c1} ...{p1,c10} .... {p26,c1}...{p26,c10}.\r\nHere p1,... p26 are 26 drivers. c1,...,10 are 10 actions.\r\n\r\nMost people split train/validation by drivers. Anyone split like this below? If so, what is the results?\r\n\r\n - Validation : {some driver, some action}\r\n                       e.g. {p1,c1}, {p5,c3} \r\n - Training:  all except {p,c} in  validation"
  },
  "source": "meta"
}