{
  "id": 92012,
  "title": "How to Trust Your CV Score??",
  "url": "/competitions/LANL-Earthquake-Prediction/discussion/92012",
  "author_name": "",
  "post_date": "2019-05-12T02:11:40.979588300Z",
  "votes": 1,
  "comment_count": 5,
  "views": 0,
  "content": "<p>In my experiment, I got a large difference CV score between trainset, validateset and public LB score, even the trend of cv and  public LB are different, I mean a lower cv score got a higher  public LB score. so I guess there 2 reason lead to this problem:\n1. There is indeed a large distribution difference between train &amp; test (many discuss prove this problem)\n2. overfit to train</p>\n\n<p>it seems like a same problem.\nmy solution is 1. give up early stop 2.use less features , but it cause a new problem: how to balance the point between cv score &amp; public PL score, which one should I trust, and How to get a credible CV?\nwhat's you good suggest on this problem?</p>",
  "messages": [
    {
      "id": "530175",
      "postDate": "05/12/2019 02:11:40",
      "content": "<p>In my experiment, I got a large difference CV score between trainset, validateset and public LB score, even the trend of cv and  public LB are different, I mean a lower cv score got a higher  public LB score. so I guess there 2 reason lead to this problem:\n1. There is indeed a large distribution difference between train &amp; test (many discuss prove this problem)\n2. overfit to train</p>\n\n<p>it seems like a same problem.\nmy solution is 1. give up early stop 2.use less features , but it cause a new problem: how to balance the point between cv score &amp; public PL score, which one should I trust, and How to get a credible CV?\nwhat's you good suggest on this problem?</p>",
      "rawMarkdown": "In my experiment, I got a large difference CV score between trainset, validateset and public LB score, even the trend of cv and  public LB are different, I mean a lower cv score got a higher  public LB score. so I guess there 2 reason lead to this problem:\n1. There is indeed a large distribution difference between train &amp; test (many discuss prove this problem)\n2. overfit to train\n\nit seems like a same problem.\nmy solution is 1. give up early stop 2.use less features , but it cause a new problem: how to balance the point between cv score &amp; public PL score, which one should I trust, and How to get a credible CV?\nwhat's you good suggest on this problem?",
      "votes": null
    },
    {
      "id": "530179",
      "postDate": "05/12/2019 02:19:09",
      "content": "<p>I think the public PL is not credible neither, It contains too few samples, what I consider about is the big shock on LB </p>",
      "rawMarkdown": "I think the public PL is not credible neither, It contains too few samples, what I consider about is the big shock on LB",
      "votes": null
    },
    {
      "id": "530274",
      "postDate": "05/12/2019 10:59:07",
      "content": "<p>I spent my first week on finding a decent CV setting.  That's the most important piece of work to be done in any competition.</p>",
      "rawMarkdown": "I spent my first week on finding a decent CV setting.  That's the most important piece of work to be done in any competition.",
      "votes": null
    },
    {
      "id": "530528",
      "postDate": "05/13/2019 05:36:20",
      "content": "<p>Thank you for reply!\nDo you mind to share some suggestion?</p>",
      "rawMarkdown": "Thank you for reply!\nDo you mind to share some suggestion?",
      "votes": null
    },
    {
      "id": "530567",
      "postDate": "05/13/2019 07:27:48",
      "content": "<p>Read all the forum, CV strategies have been discussed at length.  Basically people discussed three main choices: base folds on quakes, standard kfold unshuffled, standard k fold shuffled.  </p>",
      "rawMarkdown": "Read all the forum, CV strategies have been discussed at length.  Basically people discussed three main choices: base folds on quakes, standard kfold unshuffled, standard k fold shuffled.",
      "votes": null
    },
    {
      "id": "531879",
      "postDate": "05/15/2019 17:41:36",
      "content": "<p>Good Work.</p>",
      "rawMarkdown": "Good Work.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 530179,
      "author_name": "rentiansky",
      "author_url": "",
      "post_date": "05/12/2019 02:19:09",
      "content": "<p>I think the public PL is not credible neither, It contains too few samples, what I consider about is the big shock on LB </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 530274,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "05/12/2019 10:59:07",
      "content": "<p>I spent my first week on finding a decent CV setting.  That's the most important piece of work to be done in any competition.</p>",
      "votes": null,
      "replies": [
        {
          "id": 530528,
          "author_name": "rentiansky",
          "author_url": "",
          "post_date": "05/13/2019 05:36:20",
          "content": "<p>Thank you for reply!\nDo you mind to share some suggestion?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 530567,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "05/13/2019 07:27:48",
          "content": "<p>Read all the forum, CV strategies have been discussed at length.  Basically people discussed three main choices: base folds on quakes, standard kfold unshuffled, standard k fold shuffled.  </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 531879,
      "author_name": "bulent7",
      "author_url": "",
      "post_date": "05/15/2019 17:41:36",
      "content": "<p>Good Work.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "530175": "In my experiment, I got a large difference CV score between trainset, validateset and public LB score, even the trend of cv and  public LB are different, I mean a lower cv score got a higher  public LB score. so I guess there 2 reason lead to this problem:\n1. There is indeed a large distribution difference between train &amp; test (many discuss prove this problem)\n2. overfit to train\n\nit seems like a same problem.\nmy solution is 1. give up early stop 2.use less features , but it cause a new problem: how to balance the point between cv score &amp; public PL score, which one should I trust, and How to get a credible CV?\nwhat's you good suggest on this problem?",
    "530179": "I think the public PL is not credible neither, It contains too few samples, what I consider about is the big shock on LB",
    "530274": "I spent my first week on finding a decent CV setting.  That's the most important piece of work to be done in any competition.",
    "530528": "Thank you for reply!\nDo you mind to share some suggestion?",
    "530567": "Read all the forum, CV strategies have been discussed at length.  Basically people discussed three main choices: base folds on quakes, standard kfold unshuffled, standard k fold shuffled.",
    "531879": "Good Work."
  },
  "source": "meta"
}