{
  "id": 91726,
  "title": "Explaining difference between CV, CV shuffling and LB scores",
  "url": "/competitions/LANL-Earthquake-Prediction/discussion/91726",
  "author_name": "Carlos Prades K.",
  "post_date": "2019-05-08T12:10:54.564000",
  "votes": 2,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I have seen discussions about why CV is different from CV, and questions about why when doing CV with shuffling you get a similar score in different folds, but when you don't shuffle the folds scores are different.</p>\n\n<p>I think the reason is clear for many of you, but for those that are still confused, this how I understand it (please tell me if I am missing something).</p>\n\n<p>Some facts:\n- The size of the different cycles is variable. Some of them are short, some medium and some of the are long (due to release of energy during the cycle).\n- The error of predictions is greater in the longer periods, since it is difficult for the model that there is gonna be a release of energy that is not going to be strong enough to be considered an earthquake.</p>\n\n<p>Implications:\n- If you don't shuffle for doing CV, some folds are going to contain the longer cycles, and those folds are going to get greater mse (&gt;2) (See attached image).\n- If you shuffle you are going to get similar scores because in each fold you are going to have a mix of \"easy\" and \"tough\" to predict cycles, so you get an mse around 2.\n- The LB scores are usually &lt;2. That means to me that there are not long cycles there. Of course, that doesn't mean that there are not long cycles in the PB (I bet there are!).</p>\n\n<p>Make sense?\nI hope this helps ;)</p>",
  "messages": [
    {
      "id": 528702,
      "postDate": "2019-05-08T12:10:54.563Z",
      "content": "<p>I have seen discussions about why CV is different from CV, and questions about why when doing CV with shuffling you get a similar score in different folds, but when you don't shuffle the folds scores are different.</p>\n\n<p>I think the reason is clear for many of you, but for those that are still confused, this how I understand it (please tell me if I am missing something).</p>\n\n<p>Some facts:\n- The size of the different cycles is variable. Some of them are short, some medium and some of the are long (due to release of energy during the cycle).\n- The error of predictions is greater in the longer periods, since it is difficult for the model that there is gonna be a release of energy that is not going to be strong enough to be considered an earthquake.</p>\n\n<p>Implications:\n- If you don't shuffle for doing CV, some folds are going to contain the longer cycles, and those folds are going to get greater mse (&gt;2) (See attached image).\n- If you shuffle you are going to get similar scores because in each fold you are going to have a mix of \"easy\" and \"tough\" to predict cycles, so you get an mse around 2.\n- The LB scores are usually &lt;2. That means to me that there are not long cycles there. Of course, that doesn't mean that there are not long cycles in the PB (I bet there are!).</p>\n\n<p>Make sense?\nI hope this helps ;)</p>",
      "rawMarkdown": "I have seen discussions about why CV is different from CV, and questions about why when doing CV with shuffling you get a similar score in different folds, but when you don't shuffle the folds scores are different.\n\nI think the reason is clear for many of you, but for those that are still confused, this how I understand it (please tell me if I am missing something).\n\nSome facts:\n- The size of the different cycles is variable. Some of them are short, some medium and some of the are long (due to release of energy during the cycle).\n- The error of predictions is greater in the longer periods, since it is difficult for the model that there is gonna be a release of energy that is not going to be strong enough to be considered an earthquake.\n\nImplications:\n- If you don't shuffle for doing CV, some folds are going to contain the longer cycles, and those folds are going to get greater mse (&gt;2) (See attached image).\n- If you shuffle you are going to get similar scores because in each fold you are going to have a mix of \"easy\" and \"tough\" to predict cycles, so you get an mse around 2.\n- The LB scores are usually &lt;2. That means to me that there are not long cycles there. Of course, that doesn't mean that there are not long cycles in the PB (I bet there are!).\n\nMake sense?\nI hope this helps ;)",
      "votes": 1
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "528702": "I have seen discussions about why CV is different from CV, and questions about why when doing CV with shuffling you get a similar score in different folds, but when you don't shuffle the folds scores are different.\n\nI think the reason is clear for many of you, but for those that are still confused, this how I understand it (please tell me if I am missing something).\n\nSome facts:\n- The size of the different cycles is variable. Some of them are short, some medium and some of the are long (due to release of energy during the cycle).\n- The error of predictions is greater in the longer periods, since it is difficult for the model that there is gonna be a release of energy that is not going to be strong enough to be considered an earthquake.\n\nImplications:\n- If you don't shuffle for doing CV, some folds are going to contain the longer cycles, and those folds are going to get greater mse (&gt;2) (See attached image).\n- If you shuffle you are going to get similar scores because in each fold you are going to have a mix of \"easy\" and \"tough\" to predict cycles, so you get an mse around 2.\n- The LB scores are usually &lt;2. That means to me that there are not long cycles there. Of course, that doesn't mean that there are not long cycles in the PB (I bet there are!).\n\nMake sense?\nI hope this helps ;)"
  }
}