{
  "id": 79483,
  "title": "not just trust local cv",
  "url": "/competitions/quora-insincere-questions-classification/discussion/79483",
  "author_name": "",
  "post_date": "2019-02-04T15:08:41.178903300Z",
  "votes": 3,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Many kagglers have found that local cv is not correlate with LB. My best local cv is 0.712, but its LB is just 0.712. Which cause this phenomenon to appear: </p>\n\n<ol>\n<li>overfit training set</li>\n<li>public test set is too small</li>\n<li>it is not enough diversity in differect models</li>\n<li>random seed???</li>\n<li>other reasons u consider?</li>\n</ol>\n\n<p>In my opinion, local cv represent to single model preforment, not ensumble result. We should not just focus on local cv.\nHappy chinese new year!</p>\n\n<hr>\n\n<p>EDIT: local cv is 0.712, but its LB is 0.699</p>",
  "messages": [
    {
      "id": "466040",
      "postDate": "02/04/2019 15:08:41",
      "content": "<p>Many kagglers have found that local cv is not correlate with LB. My best local cv is 0.712, but its LB is just 0.712. Which cause this phenomenon to appear: </p>\n\n<ol>\n<li>overfit training set</li>\n<li>public test set is too small</li>\n<li>it is not enough diversity in differect models</li>\n<li>random seed???</li>\n<li>other reasons u consider?</li>\n</ol>\n\n<p>In my opinion, local cv represent to single model preforment, not ensumble result. We should not just focus on local cv.\nHappy chinese new year!</p>\n\n<hr>\n\n<p>EDIT: local cv is 0.712, but its LB is 0.699</p>",
      "rawMarkdown": "Many kagglers have found that local cv is not correlate with LB. My best local cv is 0.712, but its LB is just 0.712. Which cause this phenomenon to appear: \n\n1. overfit training set\n2. public test set is too small\n3. it is not enough diversity in differect models\n4. random seed???\n5. other reasons u consider?\n\nIn my opinion, local cv represent to single model preforment, not ensumble result. We should not just focus on local cv.\nHappy chinese new year!\n\n\n------\n\nEDIT: local cv is 0.712, but its LB is 0.699",
      "votes": null
    },
    {
      "id": "466049",
      "postDate": "02/04/2019 15:36:41",
      "content": "<p>Are you sure that you got LB score as 0.712 because currently, your lb score is 0.703 ?</p>",
      "rawMarkdown": "Are you sure that you got LB score as 0.712 because currently, your lb score is 0.703 ?",
      "votes": null
    },
    {
      "id": "466071",
      "postDate": "02/04/2019 16:08:55",
      "content": "<p>Sorry. lb score just 0.699</p>",
      "rawMarkdown": "Sorry. lb score just 0.699",
      "votes": null
    },
    {
      "id": "466307",
      "postDate": "02/05/2019 04:59:35",
      "content": "<p>A move from 0.712 to 0.699 is not out-of-the-ordinary for moving between CV and LB. It could go up or down by around that much and that wouldn't be notable. CV score is correlated with LB score to some extent, but CV is considered more robust since training datasets are usually much larger than LB test sets. If you trust in your CV and used proper methodology in your model generation, that'll likely be reflected in the private leaderboard. Local CV does not only represent single-model performance unless you're using CV incorrectly, when done properly it can be a robust measure of the performance of your estimator. </p>",
      "rawMarkdown": "A move from 0.712 to 0.699 is not out-of-the-ordinary for moving between CV and LB. It could go up or down by around that much and that wouldn't be notable. CV score is correlated with LB score to some extent, but CV is considered more robust since training datasets are usually much larger than LB test sets. If you trust in your CV and used proper methodology in your model generation, that'll likely be reflected in the private leaderboard. Local CV does not only represent single-model performance unless you're using CV incorrectly, when done properly it can be a robust measure of the performance of your estimator.",
      "votes": null
    },
    {
      "id": "466337",
      "postDate": "02/05/2019 06:34:36",
      "content": "<p>Same. Local CV of 0.6991 gives LB of 0.685. Trusting in local CV only. Fingers crossed for Private LB. Hoping all effort doesn't go to waste.</p>",
      "rawMarkdown": "Same. Local CV of 0.6991 gives LB of 0.685. Trusting in local CV only. Fingers crossed for Private LB. Hoping all effort doesn't go to waste.",
      "votes": null
    },
    {
      "id": "466385",
      "postDate": "02/05/2019 09:17:19",
      "content": "<p>F1 score is very sensitive. Most high LB public kernels overfit the training set because F1 scores becomes less sensitive when you train enough epochs.</p>",
      "rawMarkdown": "F1 score is very sensitive. Most high LB public kernels overfit the training set because F1 scores becomes less sensitive when you train enough epochs.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 466049,
      "author_name": "suchith0312",
      "author_url": "",
      "post_date": "02/04/2019 15:36:41",
      "content": "<p>Are you sure that you got LB score as 0.712 because currently, your lb score is 0.703 ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 466071,
          "author_name": "salonsai",
          "author_url": "",
          "post_date": "02/04/2019 16:08:55",
          "content": "<p>Sorry. lb score just 0.699</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 466307,
      "author_name": "alecthekulak",
      "author_url": "",
      "post_date": "02/05/2019 04:59:35",
      "content": "<p>A move from 0.712 to 0.699 is not out-of-the-ordinary for moving between CV and LB. It could go up or down by around that much and that wouldn't be notable. CV score is correlated with LB score to some extent, but CV is considered more robust since training datasets are usually much larger than LB test sets. If you trust in your CV and used proper methodology in your model generation, that'll likely be reflected in the private leaderboard. Local CV does not only represent single-model performance unless you're using CV incorrectly, when done properly it can be a robust measure of the performance of your estimator. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 466337,
      "author_name": "mlwhiz",
      "author_url": "",
      "post_date": "02/05/2019 06:34:36",
      "content": "<p>Same. Local CV of 0.6991 gives LB of 0.685. Trusting in local CV only. Fingers crossed for Private LB. Hoping all effort doesn't go to waste.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 466385,
      "author_name": "jihangz",
      "author_url": "",
      "post_date": "02/05/2019 09:17:19",
      "content": "<p>F1 score is very sensitive. Most high LB public kernels overfit the training set because F1 scores becomes less sensitive when you train enough epochs.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "466040": "Many kagglers have found that local cv is not correlate with LB. My best local cv is 0.712, but its LB is just 0.712. Which cause this phenomenon to appear: \n\n1. overfit training set\n2. public test set is too small\n3. it is not enough diversity in differect models\n4. random seed???\n5. other reasons u consider?\n\nIn my opinion, local cv represent to single model preforment, not ensumble result. We should not just focus on local cv.\nHappy chinese new year!\n\n\n------\n\nEDIT: local cv is 0.712, but its LB is 0.699",
    "466049": "Are you sure that you got LB score as 0.712 because currently, your lb score is 0.703 ?",
    "466071": "Sorry. lb score just 0.699",
    "466307": "A move from 0.712 to 0.699 is not out-of-the-ordinary for moving between CV and LB. It could go up or down by around that much and that wouldn't be notable. CV score is correlated with LB score to some extent, but CV is considered more robust since training datasets are usually much larger than LB test sets. If you trust in your CV and used proper methodology in your model generation, that'll likely be reflected in the private leaderboard. Local CV does not only represent single-model performance unless you're using CV incorrectly, when done properly it can be a robust measure of the performance of your estimator.",
    "466337": "Same. Local CV of 0.6991 gives LB of 0.685. Trusting in local CV only. Fingers crossed for Private LB. Hoping all effort doesn't go to waste.",
    "466385": "F1 score is very sensitive. Most high LB public kernels overfit the training set because F1 scores becomes less sensitive when you train enough epochs."
  },
  "source": "meta"
}