{
  "id": 24255,
  "title": "CV vs LB",
  "url": "/competitions/outbrain-click-prediction/discussion/24255",
  "author_name": "",
  "post_date": "2016-10-10T18:13:23.517Z",
  "votes": 4,
  "comment_count": 13,
  "views": 2342,
  "content": "<p>Classic Thread : )</p>\n\n<p>I made three submissions:</p>\n\n<p>1) cv: 0.632. lb: 0.652</p>\n\n<p>2) cv: 0.640. lb: 0.659</p>\n\n<p>3) cv: 0.649. lb: 0.664</p>\n\n<p>It is kind of surprising to me that 3)-2) cv improvement is 2x of lb improvement.</p>",
  "messages": [
    {
      "id": "138741",
      "postDate": "10/10/2016 18:13:23",
      "content": "<p>Classic Thread : )</p>\n\n<p>I made three submissions:</p>\n\n<p>1) cv: 0.632. lb: 0.652</p>\n\n<p>2) cv: 0.640. lb: 0.659</p>\n\n<p>3) cv: 0.649. lb: 0.664</p>\n\n<p>It is kind of surprising to me that 3)-2) cv improvement is 2x of lb improvement.</p>",
      "rawMarkdown": "Classic Thread : )\r\n\r\nI made three submissions:\r\n\r\n1) cv: 0.632. lb: 0.652\r\n\r\n2) cv: 0.640. lb: 0.659\r\n\r\n3) cv: 0.649. lb: 0.664\r\n\r\nIt is kind of surprising to me that 3)-2) cv improvement is 2x of lb improvement.",
      "votes": null
    },
    {
      "id": "138818",
      "postDate": "10/11/2016 01:10:34",
      "content": "<p>The problem: normal CV is not a representative sample from test data</p>\n\n<p>Look the train/test distributions (@joconnor kernel):</p>\n\n<p><img src=\"https://www.kaggle.io/svf/387845/4dde313fc3ec72f1a13a6f4638501efa/__results___files/__results___6_1.png\" alt=\"Test Distribution\" title></p>\n\n<p>What about this approach?</p>\n\n<ol>\n<li>CV using random split (estimation for 'present' data)</li>\n<li>CV using time based split (estimation for 'future' data)</li>\n<li>Final CV = weighted mean from 1 and 2</li>\n</ol>",
      "rawMarkdown": "The problem: normal CV is not a representative sample from test data\r\n\r\nLook the train/test distributions (@joconnor kernel):\r\n\r\n![Test Distribution][1]\r\n\r\nWhat about this approach?\r\n\r\n 1. CV using random split (estimation for 'present' data)\r\n 2. CV using time based split (estimation for 'future' data)\r\n 3. Final CV = weighted mean from 1 and 2\r\n\r\n  [1]: https://www.kaggle.io/svf/387845/4dde313fc3ec72f1a13a6f4638501efa/__results___files/__results___6_1.png",
      "votes": null
    },
    {
      "id": "138844",
      "postDate": "10/11/2016 07:03:05",
      "content": "<p>@rcarson Interesting.</p>\n\n<p>Are you using a time based split or random split?</p>",
      "rawMarkdown": "rcarson Interesting.\r\n\r\nAre you using a time based split or random split?",
      "votes": null
    },
    {
      "id": "138884",
      "postDate": "10/11/2016 13:13:01",
      "content": "<p>@Eric, thank you! I don't know why I can't upvote you, maybe something wrong with my browser.</p>\n\n<p>@Sonny, I'm using naive split. First 60% rows in click_train as train and the rest 40% as validation.</p>",
      "rawMarkdown": "Eric, thank you! I don't know why I can't upvote you, maybe something wrong with my browser.\r\n\r\n@Sonny, I'm using naive split. First 60% rows in click_train as train and the rest 40% as validation.",
      "votes": null
    },
    {
      "id": "138886",
      "postDate": "10/11/2016 13:38:30",
      "content": "<p>@rcarson, same with me.  I think is a Kaggle problem. =(</p>\n\n<p>Looking the train/test distributions again, a good way to split can be:</p>\n\n<ol>\n<li><strong>Train:</strong> 80% of train data from day 0 to 10</li>\n<li><strong>Validation:</strong> 20% of train data from day 0 to 10 + day 11 + day 12</li>\n</ol>",
      "rawMarkdown": "rcarson, same with me.  I think is a Kaggle problem. =(\r\n\r\nLooking the train/test distributions again, a good way to split can be:\r\n\r\n 1. **Train:** 80% of train data from day 0 to 10\r\n 2. **Validation:** 20% of train data from day 0 to 10 + day 11 + day 12",
      "votes": null
    },
    {
      "id": "138892",
      "postDate": "10/11/2016 14:23:31",
      "content": "<p>I was wondering what the organizer's motivation could be to split train and test this way. Would be great to have some understanding on this.</p>",
      "rawMarkdown": "I was wondering what the organizer's motivation could be to split train and test this way. Would be great to have some understanding on this.",
      "votes": null
    },
    {
      "id": "139665",
      "postDate": "10/15/2016 18:00:43",
      "content": "<p>Thank you @Eric Couto. my cv is kind of consistent now</p>\n\n<p>cv: 0.6613 lb: 0.65989</p>\n\n<p>cv: 0.6653 lb: 0.66416</p>\n\n<p>cv: 0.6674 lb: 0.66548</p>",
      "rawMarkdown": "Thank you @Eric Couto. my cv is kind of consistent now\r\n\r\ncv: 0.6613 lb: 0.65989\r\n\r\ncv: 0.6653 lb: 0.66416\r\n\r\ncv: 0.6674 lb: 0.66548",
      "votes": null
    },
    {
      "id": "139777",
      "postDate": "10/16/2016 14:46:30",
      "content": "<p>Great, @rcarson! I'll give it a try!</p>",
      "rawMarkdown": "Great, @rcarson! I'll give it a try!",
      "votes": null
    },
    {
      "id": "140805",
      "postDate": "10/23/2016 10:07:42",
      "content": "<p>@rcarson Can you show me how to evaluate Mean Average Precision on CV train?</p>",
      "rawMarkdown": "rcarson Can you show me how to evaluate Mean Average Precision on CV train?",
      "votes": null
    },
    {
      "id": "141077",
      "postDate": "10/25/2016 02:27:53",
      "content": "<p>Thanks @ericcouto ! With this split my CV and LB are now nicely aligned. #brazilianforce</p>",
      "rawMarkdown": "Thanks @ericcouto ! With this split my CV and LB are now nicely aligned. #brazilianforce",
      "votes": null
    },
    {
      "id": "144293",
      "postDate": "11/13/2016 16:28:52",
      "content": "<p>Our Val and LB scores are</p>\n\n<ol>\n<li>Val = 0.6680 and LB = 0.6745</li>\n<li>Val = 0.6651 and LB = 0.6697 </li>\n</ol>\n\n<p>Our validation set is a mix of random and time based split to mimic the test data as shown by @Eric. </p>",
      "rawMarkdown": "Our Val and LB scores are\r\n\r\n1. Val = 0.6680 and LB = 0.6745\r\n2. Val = 0.6651 and LB = 0.6697 \r\n\r\nOur validation set is a mix of random and time based split to mimic the test data as shown by @Eric.",
      "votes": null
    },
    {
      "id": "151151",
      "postDate": "12/19/2016 03:34:07",
      "content": "<p>@eric i am using a validation set as discussed above, even then my cv seems to be inconsistent with the lb. \nIm doing the following -&gt; display_id % 5. \n Validation set  -&gt; display_mod = 0 or days &gt; 10 and (for 20 % of present and future)\n Train set -&gt; display_mod != 0 and days &lt;= 10 (80 % present data).</p>\n\n<p>Should I also considering training the model separately for future and present with different validation sets? </p>\n\n<p>Could you please let me know if I am doing anything wrong?</p>",
      "rawMarkdown": "eric i am using a validation set as discussed above, even then my cv seems to be inconsistent with the lb. \r\nIm doing the following -> display_id % 5. \r\n Validation set  -> display_mod = 0 or days > 10 and (for 20 % of present and future)\r\n Train set -> display_mod != 0 and days <= 10 (80 % present data).\r\n\r\nShould I also considering training the model separately for future and present with different validation sets? \r\n\r\nCould you please let me know if I am doing anything wrong?",
      "votes": null
    },
    {
      "id": "151562",
      "postDate": "12/21/2016 10:14:09",
      "content": "<p>@Keerath Jaggi,</p>\n\n<p>I think you're doing right. Can you give the CV/LB MAPs?</p>\n\n<blockquote>\n  <p>Should I also considering training the model separately for future and present with different validation sets?</p>\n</blockquote>\n\n<p>Right now, I am not training my models separately, but it's a good next step! </p>",
      "rawMarkdown": "Keerath Jaggi,\r\n\r\nI think you're doing right. Can you give the CV/LB MAPs?\r\n\r\n> Should I also considering training the model separately for future and present with different validation sets?\r\n\r\nRight now, I am not training my models separately, but it's a good next step!",
      "votes": null
    },
    {
      "id": "151570",
      "postDate": "12/21/2016 11:07:02",
      "content": "<p>@eric thanks for responding there was this small bug in my code because of which my CV was inconsistent  and ya I'm going to try training the models on different validation sets present and future and let u know if that was helpful.</p>",
      "rawMarkdown": "eric thanks for responding there was this small bug in my code because of which my CV was inconsistent  and ya I'm going to try training the models on different validation sets present and future and let u know if that was helpful.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 138818,
      "author_name": "ericcouto",
      "author_url": "",
      "post_date": "10/11/2016 01:10:34",
      "content": "<p>The problem: normal CV is not a representative sample from test data</p>\n\n<p>Look the train/test distributions (@joconnor kernel):</p>\n\n<p><img src=\"https://www.kaggle.io/svf/387845/4dde313fc3ec72f1a13a6f4638501efa/__results___files/__results___6_1.png\" alt=\"Test Distribution\" title></p>\n\n<p>What about this approach?</p>\n\n<ol>\n<li>CV using random split (estimation for 'present' data)</li>\n<li>CV using time based split (estimation for 'future' data)</li>\n<li>Final CV = weighted mean from 1 and 2</li>\n</ol>",
      "votes": null,
      "replies": []
    },
    {
      "id": 138844,
      "author_name": "sonnylaskar",
      "author_url": "",
      "post_date": "10/11/2016 07:03:05",
      "content": "<p>@rcarson Interesting.</p>\n\n<p>Are you using a time based split or random split?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 138884,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "10/11/2016 13:13:01",
      "content": "<p>@Eric, thank you! I don't know why I can't upvote you, maybe something wrong with my browser.</p>\n\n<p>@Sonny, I'm using naive split. First 60% rows in click_train as train and the rest 40% as validation.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 138886,
      "author_name": "ericcouto",
      "author_url": "",
      "post_date": "10/11/2016 13:38:30",
      "content": "<p>@rcarson, same with me.  I think is a Kaggle problem. =(</p>\n\n<p>Looking the train/test distributions again, a good way to split can be:</p>\n\n<ol>\n<li><strong>Train:</strong> 80% of train data from day 0 to 10</li>\n<li><strong>Validation:</strong> 20% of train data from day 0 to 10 + day 11 + day 12</li>\n</ol>",
      "votes": null,
      "replies": []
    },
    {
      "id": 138892,
      "author_name": "sangxia",
      "author_url": "",
      "post_date": "10/11/2016 14:23:31",
      "content": "<p>I was wondering what the organizer's motivation could be to split train and test this way. Would be great to have some understanding on this.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 139665,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "10/15/2016 18:00:43",
      "content": "<p>Thank you @Eric Couto. my cv is kind of consistent now</p>\n\n<p>cv: 0.6613 lb: 0.65989</p>\n\n<p>cv: 0.6653 lb: 0.66416</p>\n\n<p>cv: 0.6674 lb: 0.66548</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 139777,
      "author_name": "ericcouto",
      "author_url": "",
      "post_date": "10/16/2016 14:46:30",
      "content": "<p>Great, @rcarson! I'll give it a try!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 140805,
      "author_name": "nhuantd",
      "author_url": "",
      "post_date": "10/23/2016 10:07:42",
      "content": "<p>@rcarson Can you show me how to evaluate Mean Average Precision on CV train?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 141077,
      "author_name": "gspmoreira",
      "author_url": "",
      "post_date": "10/25/2016 02:27:53",
      "content": "<p>Thanks @ericcouto ! With this split my CV and LB are now nicely aligned. #brazilianforce</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 144293,
      "author_name": "sudalairajkumar",
      "author_url": "",
      "post_date": "11/13/2016 16:28:52",
      "content": "<p>Our Val and LB scores are</p>\n\n<ol>\n<li>Val = 0.6680 and LB = 0.6745</li>\n<li>Val = 0.6651 and LB = 0.6697 </li>\n</ol>\n\n<p>Our validation set is a mix of random and time based split to mimic the test data as shown by @Eric. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 151151,
      "author_name": "keerath",
      "author_url": "",
      "post_date": "12/19/2016 03:34:07",
      "content": "<p>@eric i am using a validation set as discussed above, even then my cv seems to be inconsistent with the lb. \nIm doing the following -&gt; display_id % 5. \n Validation set  -&gt; display_mod = 0 or days &gt; 10 and (for 20 % of present and future)\n Train set -&gt; display_mod != 0 and days &lt;= 10 (80 % present data).</p>\n\n<p>Should I also considering training the model separately for future and present with different validation sets? </p>\n\n<p>Could you please let me know if I am doing anything wrong?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 151562,
      "author_name": "ericcouto",
      "author_url": "",
      "post_date": "12/21/2016 10:14:09",
      "content": "<p>@Keerath Jaggi,</p>\n\n<p>I think you're doing right. Can you give the CV/LB MAPs?</p>\n\n<blockquote>\n  <p>Should I also considering training the model separately for future and present with different validation sets?</p>\n</blockquote>\n\n<p>Right now, I am not training my models separately, but it's a good next step! </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 151570,
      "author_name": "keerath",
      "author_url": "",
      "post_date": "12/21/2016 11:07:02",
      "content": "<p>@eric thanks for responding there was this small bug in my code because of which my CV was inconsistent  and ya I'm going to try training the models on different validation sets present and future and let u know if that was helpful.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "138741": "Classic Thread : )\r\n\r\nI made three submissions:\r\n\r\n1) cv: 0.632. lb: 0.652\r\n\r\n2) cv: 0.640. lb: 0.659\r\n\r\n3) cv: 0.649. lb: 0.664\r\n\r\nIt is kind of surprising to me that 3)-2) cv improvement is 2x of lb improvement.",
    "138818": "The problem: normal CV is not a representative sample from test data\r\n\r\nLook the train/test distributions (@joconnor kernel):\r\n\r\n![Test Distribution][1]\r\n\r\nWhat about this approach?\r\n\r\n 1. CV using random split (estimation for 'present' data)\r\n 2. CV using time based split (estimation for 'future' data)\r\n 3. Final CV = weighted mean from 1 and 2\r\n\r\n  [1]: https://www.kaggle.io/svf/387845/4dde313fc3ec72f1a13a6f4638501efa/__results___files/__results___6_1.png",
    "138844": "rcarson Interesting.\r\n\r\nAre you using a time based split or random split?",
    "138884": "Eric, thank you! I don't know why I can't upvote you, maybe something wrong with my browser.\r\n\r\n@Sonny, I'm using naive split. First 60% rows in click_train as train and the rest 40% as validation.",
    "138886": "rcarson, same with me.  I think is a Kaggle problem. =(\r\n\r\nLooking the train/test distributions again, a good way to split can be:\r\n\r\n 1. **Train:** 80% of train data from day 0 to 10\r\n 2. **Validation:** 20% of train data from day 0 to 10 + day 11 + day 12",
    "138892": "I was wondering what the organizer's motivation could be to split train and test this way. Would be great to have some understanding on this.",
    "139665": "Thank you @Eric Couto. my cv is kind of consistent now\r\n\r\ncv: 0.6613 lb: 0.65989\r\n\r\ncv: 0.6653 lb: 0.66416\r\n\r\ncv: 0.6674 lb: 0.66548",
    "139777": "Great, @rcarson! I'll give it a try!",
    "140805": "rcarson Can you show me how to evaluate Mean Average Precision on CV train?",
    "141077": "Thanks @ericcouto ! With this split my CV and LB are now nicely aligned. #brazilianforce",
    "144293": "Our Val and LB scores are\r\n\r\n1. Val = 0.6680 and LB = 0.6745\r\n2. Val = 0.6651 and LB = 0.6697 \r\n\r\nOur validation set is a mix of random and time based split to mimic the test data as shown by @Eric.",
    "151151": "eric i am using a validation set as discussed above, even then my cv seems to be inconsistent with the lb. \r\nIm doing the following -> display_id % 5. \r\n Validation set  -> display_mod = 0 or days > 10 and (for 20 % of present and future)\r\n Train set -> display_mod != 0 and days <= 10 (80 % present data).\r\n\r\nShould I also considering training the model separately for future and present with different validation sets? \r\n\r\nCould you please let me know if I am doing anything wrong?",
    "151562": "Keerath Jaggi,\r\n\r\nI think you're doing right. Can you give the CV/LB MAPs?\r\n\r\n> Should I also considering training the model separately for future and present with different validation sets?\r\n\r\nRight now, I am not training my models separately, but it's a good next step!",
    "151570": "eric thanks for responding there was this small bug in my code because of which my CV was inconsistent  and ya I'm going to try training the models on different validation sets present and future and let u know if that was helpful."
  },
  "source": "meta"
}