{
  "id": 88799,
  "title": "Test Set Too Small",
  "url": "/competitions/LANL-Earthquake-Prediction/discussion/88799",
  "author_name": "",
  "post_date": "2019-04-10T15:42:34.590635600Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I think the score with the whole test set will have high variance in general  unless there is a a very strong model.  There is a high probability that luck would play big in this competition.</p>",
  "messages": [
    {
      "id": "512134",
      "postDate": "04/10/2019 15:42:34",
      "content": "<p>I think the score with the whole test set will have high variance in general  unless there is a a very strong model.  There is a high probability that luck would play big in this competition.</p>",
      "rawMarkdown": "I think the score with the whole test set will have high variance in general  unless there is a a very strong model.  There is a high probability that luck would play big in this competition.",
      "votes": null
    },
    {
      "id": "512473",
      "postDate": "04/10/2019 19:20:56",
      "content": "<p>In that case, it is our responsibility to make models that can generalise as much as possible! Certainly I agree that the public LB is unhelpful, as it's clearly an unrepresentative sample of the whole data. Variance will definitely be an issue there.</p>\n\n<p>The researchers helpfully pointed out that both train and test are from the same experiment. Therefore, I think our best approach is to minimise the CV score for our models across the training data, rather than trying to climb the leaderboard (though it's hard to resist...).</p>",
      "rawMarkdown": "In that case, it is our responsibility to make models that can generalise as much as possible! Certainly I agree that the public LB is unhelpful, as it's clearly an unrepresentative sample of the whole data. Variance will definitely be an issue there.\n\nThe researchers helpfully pointed out that both train and test are from the same experiment. Therefore, I think our best approach is to minimise the CV score for our models across the training data, rather than trying to climb the leaderboard (though it's hard to resist...).",
      "votes": null
    },
    {
      "id": "514168",
      "postDate": "04/11/2019 13:42:45",
      "content": "<p>That's not an unusual problem. You'll want to augment the data a bit.</p>\n\n<p>Take the train data and break it into 150k sized pieces, but you can step every 75k, 50k.... Break it into too many sections and you'll overfit. Finding the sweet spot is part of the puzzle.</p>",
      "rawMarkdown": "That's not an unusual problem. You'll want to augment the data a bit.\n\nTake the train data and break it into 150k sized pieces, but you can step every 75k, 50k.... Break it into too many sections and you'll overfit. Finding the sweet spot is part of the puzzle.",
      "votes": null
    },
    {
      "id": "515639",
      "postDate": "04/12/2019 21:51:56",
      "content": "<p>As many great Kaggler's have said, trust your CV score, not the public LB.</p>",
      "rawMarkdown": "As many great Kaggler's have said, trust your CV score, not the public LB.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 512473,
      "author_name": "bigironsphere",
      "author_url": "",
      "post_date": "04/10/2019 19:20:56",
      "content": "<p>In that case, it is our responsibility to make models that can generalise as much as possible! Certainly I agree that the public LB is unhelpful, as it's clearly an unrepresentative sample of the whole data. Variance will definitely be an issue there.</p>\n\n<p>The researchers helpfully pointed out that both train and test are from the same experiment. Therefore, I think our best approach is to minimise the CV score for our models across the training data, rather than trying to climb the leaderboard (though it's hard to resist...).</p>",
      "votes": null,
      "replies": [
        {
          "id": 515639,
          "author_name": "halldalton94",
          "author_url": "",
          "post_date": "04/12/2019 21:51:56",
          "content": "<p>As many great Kaggler's have said, trust your CV score, not the public LB.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 514168,
      "author_name": "wrinkledtime",
      "author_url": "",
      "post_date": "04/11/2019 13:42:45",
      "content": "<p>That's not an unusual problem. You'll want to augment the data a bit.</p>\n\n<p>Take the train data and break it into 150k sized pieces, but you can step every 75k, 50k.... Break it into too many sections and you'll overfit. Finding the sweet spot is part of the puzzle.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "512134": "I think the score with the whole test set will have high variance in general  unless there is a a very strong model.  There is a high probability that luck would play big in this competition.",
    "512473": "In that case, it is our responsibility to make models that can generalise as much as possible! Certainly I agree that the public LB is unhelpful, as it's clearly an unrepresentative sample of the whole data. Variance will definitely be an issue there.\n\nThe researchers helpfully pointed out that both train and test are from the same experiment. Therefore, I think our best approach is to minimise the CV score for our models across the training data, rather than trying to climb the leaderboard (though it's hard to resist...).",
    "514168": "That's not an unusual problem. You'll want to augment the data a bit.\n\nTake the train data and break it into 150k sized pieces, but you can step every 75k, 50k.... Break it into too many sections and you'll overfit. Finding the sweet spot is part of the puzzle.",
    "515639": "As many great Kaggler's have said, trust your CV score, not the public LB."
  },
  "source": "meta"
}