{
  "id": 75821,
  "title": "One way to simulate shakeup from the stage 2 test set",
  "url": "/competitions/quora-insincere-questions-classification/discussion/75821",
  "author_name": "bilal2vec",
  "post_date": "2018-12-26T22:39:34.563000",
  "votes": 18,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hi,</p>\n\n<p>Since the stage 1 test data is so small compared to the stage-2 data (50k rows vs 400k rows) the public LB score isn't going to be a good indicator of your score on the private LB, so using part of the training set to simulate the test set might give you a better idea of how well your model generalizes to more data.</p>\n\n<p>One way to do this is: Use 850k rows of the train set to train your model and split the other 450k rows into a 50k row stage 1 test set and a 400k row stage 2 test set. Run your model multiple times with different random seeds to simulate the likely range of the shakeup.</p>\n\n<p>This technique was used by the winner of the Planet competition. His full discussion post showing his approach to simulating the private test set is available here: <a href=\"https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/discussion/36809\">https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/discussion/36809</a></p>",
  "messages": [
    {
      "id": 445670,
      "postDate": "2018-12-26T22:39:34.563Z",
      "content": "<p>Hi,</p>\n\n<p>Since the stage 1 test data is so small compared to the stage-2 data (50k rows vs 400k rows) the public LB score isn't going to be a good indicator of your score on the private LB, so using part of the training set to simulate the test set might give you a better idea of how well your model generalizes to more data.</p>\n\n<p>One way to do this is: Use 850k rows of the train set to train your model and split the other 450k rows into a 50k row stage 1 test set and a 400k row stage 2 test set. Run your model multiple times with different random seeds to simulate the likely range of the shakeup.</p>\n\n<p>This technique was used by the winner of the Planet competition. His full discussion post showing his approach to simulating the private test set is available here: <a href=\"https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/discussion/36809\">https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/discussion/36809</a></p>",
      "rawMarkdown": "Hi,\n\nSince the stage 1 test data is so small compared to the stage-2 data (50k rows vs 400k rows) the public LB score isn't going to be a good indicator of your score on the private LB, so using part of the training set to simulate the test set might give you a better idea of how well your model generalizes to more data.\n\nOne way to do this is: Use 850k rows of the train set to train your model and split the other 450k rows into a 50k row stage 1 test set and a 400k row stage 2 test set. Run your model multiple times with different random seeds to simulate the likely range of the shakeup.\n\nThis technique was used by the winner of the Planet competition. His full discussion post showing his approach to simulating the private test set is available here: https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/discussion/36809",
      "votes": 17
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "445670": "Hi,\n\nSince the stage 1 test data is so small compared to the stage-2 data (50k rows vs 400k rows) the public LB score isn't going to be a good indicator of your score on the private LB, so using part of the training set to simulate the test set might give you a better idea of how well your model generalizes to more data.\n\nOne way to do this is: Use 850k rows of the train set to train your model and split the other 450k rows into a 50k row stage 1 test set and a 400k row stage 2 test set. Run your model multiple times with different random seeds to simulate the likely range of the shakeup.\n\nThis technique was used by the winner of the Planet competition. His full discussion post showing his approach to simulating the private test set is available here: https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/discussion/36809"
  }
}