{
  "id": 92212,
  "title": "What are the properties of a good CV strategy?",
  "url": "/competitions/LANL-Earthquake-Prediction/discussion/92212",
  "author_name": "",
  "post_date": "2019-05-14T11:57:25.378953900Z",
  "votes": 10,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I guess a good CV strategy is really important in this competition. So I would like to hear and learn from others how they think a good CV should be. </p>\n\n<p>IMHO, a good CV strategy has the following properties:\n1. robust to irrelevant changes, e.g., models, random states, number of folds (within a small range), how the folds are splitted, etc.\n2. have a good correlation between local CV and test scores, so that we can trust CV when selecting features/models\n3. model trained on each fold can be averaged to further decrease bias on the test set\n4. in order to find out if a CV strategy is good or bad, we need to do experiments on the training data, or sometimes on simulated data</p>\n\n<p>Please feel free to leave your suggestions or thoughts, thanks :)</p>",
  "messages": [
    {
      "id": "531155",
      "postDate": "05/14/2019 11:57:25",
      "content": "<p>I guess a good CV strategy is really important in this competition. So I would like to hear and learn from others how they think a good CV should be. </p>\n\n<p>IMHO, a good CV strategy has the following properties:\n1. robust to irrelevant changes, e.g., models, random states, number of folds (within a small range), how the folds are splitted, etc.\n2. have a good correlation between local CV and test scores, so that we can trust CV when selecting features/models\n3. model trained on each fold can be averaged to further decrease bias on the test set\n4. in order to find out if a CV strategy is good or bad, we need to do experiments on the training data, or sometimes on simulated data</p>\n\n<p>Please feel free to leave your suggestions or thoughts, thanks :)</p>",
      "rawMarkdown": "I guess a good CV strategy is really important in this competition. So I would like to hear and learn from others how they think a good CV should be. \n\nIMHO, a good CV strategy has the following properties:\n1. robust to irrelevant changes, e.g., models, random states, number of folds (within a small range), how the folds are splitted, etc.\n2. have a good correlation between local CV and test scores, so that we can trust CV when selecting features/models\n3. model trained on each fold can be averaged to further decrease bias on the test set\n4. in order to find out if a CV strategy is good or bad, we need to do experiments on the training data, or sometimes on simulated data\n\nPlease feel free to leave your suggestions or thoughts, thanks :)",
      "votes": null
    },
    {
      "id": "531180",
      "postDate": "05/14/2019 12:39:31",
      "content": "<p>Agree on what you describe except for this:</p>\n\n<blockquote>\n  <p>robust to irrelevant changes, e.g., models, random states, number of folds (within a small range), how the folds are splitted, etc.</p>\n</blockquote>\n\n<p>Number of folds (within a small range), and how the folds are split is the crux of the CV strategy.  Any change of how folds are defined is a change in CV strategy.  Or I don't understand what you call a CV strategy.</p>",
      "rawMarkdown": "Agree on what you describe except for this:\n\n&gt; robust to irrelevant changes, e.g., models, random states, number of folds (within a small range), how the folds are splitted, etc.\n\nNumber of folds (within a small range), and how the folds are split is the crux of the CV strategy.  Any change of how folds are defined is a change in CV strategy.  Or I don't understand what you call a CV strategy.",
      "votes": null
    },
    {
      "id": "531333",
      "postDate": "05/14/2019 16:58:15",
      "content": "<p>Yes, you are right! Thanks for the comment. When I wrote this, I was only thinking of kfold split, in which case if  k is slightly changed the resulting scores should be stable. But in time series, it becomes very tricky. Maybe the choice of validation and training data is essential here. </p>\n\n<p>Actually, many issues should be taken into account when choosing the train and validation data:\n- shuffle or not\n- augmentation or not\n- 1 EQ per fold or randomly split train/validation</p>\n\n<p>But there is one thing that I don't understand fully, how the number of folds impacts CV? From my previous experience, slightly changing k only has a marginal effect on the averaged/ensemble test prediction.</p>",
      "rawMarkdown": "Yes, you are right! Thanks for the comment. When I wrote this, I was only thinking of kfold split, in which case if  k is slightly changed the resulting scores should be stable. But in time series, it becomes very tricky. Maybe the choice of validation and training data is essential here. \n\nActually, many issues should be taken into account when choosing the train and validation data:\n- shuffle or not\n- augmentation or not\n- 1 EQ per fold or randomly split train/validation\n\nBut there is one thing that I don't understand fully, how the number of folds impacts CV? From my previous experience, slightly changing k only has a marginal effect on the averaged/ensemble test prediction.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 531180,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "05/14/2019 12:39:31",
      "content": "<p>Agree on what you describe except for this:</p>\n\n<blockquote>\n  <p>robust to irrelevant changes, e.g., models, random states, number of folds (within a small range), how the folds are splitted, etc.</p>\n</blockquote>\n\n<p>Number of folds (within a small range), and how the folds are split is the crux of the CV strategy.  Any change of how folds are defined is a change in CV strategy.  Or I don't understand what you call a CV strategy.</p>",
      "votes": null,
      "replies": [
        {
          "id": 531333,
          "author_name": "lucaskg",
          "author_url": "",
          "post_date": "05/14/2019 16:58:15",
          "content": "<p>Yes, you are right! Thanks for the comment. When I wrote this, I was only thinking of kfold split, in which case if  k is slightly changed the resulting scores should be stable. But in time series, it becomes very tricky. Maybe the choice of validation and training data is essential here. </p>\n\n<p>Actually, many issues should be taken into account when choosing the train and validation data:\n- shuffle or not\n- augmentation or not\n- 1 EQ per fold or randomly split train/validation</p>\n\n<p>But there is one thing that I don't understand fully, how the number of folds impacts CV? From my previous experience, slightly changing k only has a marginal effect on the averaged/ensemble test prediction.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "531155": "I guess a good CV strategy is really important in this competition. So I would like to hear and learn from others how they think a good CV should be. \n\nIMHO, a good CV strategy has the following properties:\n1. robust to irrelevant changes, e.g., models, random states, number of folds (within a small range), how the folds are splitted, etc.\n2. have a good correlation between local CV and test scores, so that we can trust CV when selecting features/models\n3. model trained on each fold can be averaged to further decrease bias on the test set\n4. in order to find out if a CV strategy is good or bad, we need to do experiments on the training data, or sometimes on simulated data\n\nPlease feel free to leave your suggestions or thoughts, thanks :)",
    "531180": "Agree on what you describe except for this:\n\n&gt; robust to irrelevant changes, e.g., models, random states, number of folds (within a small range), how the folds are splitted, etc.\n\nNumber of folds (within a small range), and how the folds are split is the crux of the CV strategy.  Any change of how folds are defined is a change in CV strategy.  Or I don't understand what you call a CV strategy.",
    "531333": "Yes, you are right! Thanks for the comment. When I wrote this, I was only thinking of kfold split, in which case if  k is slightly changed the resulting scores should be stable. But in time series, it becomes very tricky. Maybe the choice of validation and training data is essential here. \n\nActually, many issues should be taken into account when choosing the train and validation data:\n- shuffle or not\n- augmentation or not\n- 1 EQ per fold or randomly split train/validation\n\nBut there is one thing that I don't understand fully, how the number of folds impacts CV? From my previous experience, slightly changing k only has a marginal effect on the averaged/ensemble test prediction."
  },
  "source": "meta"
}