{
  "id": 550879,
  "title": "Do you use a validation set?",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/550879",
  "author_name": "",
  "post_date": "2024-12-10T03:50:40.550421500Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Do you guys leave out the last parquet or two as a validation set? Or train without validation set and use the submission as a \"validation set\"?</p>\n<p>With validation set</p>\n<ul>\n<li>can test changes very quickly, able to see during model training if it helped</li>\n<li>model learns from lesser data</li>\n</ul>\n<p>Without validation set</p>\n<ul>\n<li>only 5 submissions a day, so only 5 times a day to test your improvements</li>\n<li>\"faster iteration\" since you dont have to retrain the model on the full dataset without validation before submitting</li>\n</ul>",
  "messages": [
    {
      "id": "3068180",
      "postDate": "12/10/2024 03:50:40",
      "content": "<p>Do you guys leave out the last parquet or two as a validation set? Or train without validation set and use the submission as a \"validation set\"?</p>\n<p>With validation set</p>\n<ul>\n<li>can test changes very quickly, able to see during model training if it helped</li>\n<li>model learns from lesser data</li>\n</ul>\n<p>Without validation set</p>\n<ul>\n<li>only 5 submissions a day, so only 5 times a day to test your improvements</li>\n<li>\"faster iteration\" since you dont have to retrain the model on the full dataset without validation before submitting</li>\n</ul>",
      "rawMarkdown": "Do you guys leave out the last parquet or two as a validation set? Or train without validation set and use the submission as a \"validation set\"?\n\nWith validation set\n- can test changes very quickly, able to see during model training if it helped\n- model learns from lesser data\n\nWithout validation set\n- only 5 submissions a day, so only 5 times a day to test your improvements\n- \"faster iteration\" since you dont have to retrain the model on the full dataset without validation before submitting",
      "votes": null
    },
    {
      "id": "3068213",
      "postDate": "12/10/2024 04:58:10",
      "content": "<p>I use last parquet as a validation for early stopping to determine the number of booster rounds before overfit then retrain with all data and no early stopping using the optimum number of booster rounds found from validation </p>",
      "rawMarkdown": "I use last parquet as a validation for early stopping to determine the number of booster rounds before overfit then retrain with all data and no early stopping using the optimum number of booster rounds found from validation",
      "votes": null
    },
    {
      "id": "3068421",
      "postDate": "12/10/2024 08:32:46",
      "content": "<p>it is hard to use a valid set cuz there is a huge gap between result from valid set and public board</p>",
      "rawMarkdown": "it is hard to use a valid set cuz there is a huge gap between result from valid set and public board",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3068213,
      "author_name": "lgreig",
      "author_url": "",
      "post_date": "12/10/2024 04:58:10",
      "content": "<p>I use last parquet as a validation for early stopping to determine the number of booster rounds before overfit then retrain with all data and no early stopping using the optimum number of booster rounds found from validation </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3068421,
      "author_name": "nauynixgnay",
      "author_url": "",
      "post_date": "12/10/2024 08:32:46",
      "content": "<p>it is hard to use a valid set cuz there is a huge gap between result from valid set and public board</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3068180": "Do you guys leave out the last parquet or two as a validation set? Or train without validation set and use the submission as a \"validation set\"?\n\nWith validation set\n- can test changes very quickly, able to see during model training if it helped\n- model learns from lesser data\n\nWithout validation set\n- only 5 submissions a day, so only 5 times a day to test your improvements\n- \"faster iteration\" since you dont have to retrain the model on the full dataset without validation before submitting",
    "3068213": "I use last parquet as a validation for early stopping to determine the number of booster rounds before overfit then retrain with all data and no early stopping using the optimum number of booster rounds found from validation",
    "3068421": "it is hard to use a valid set cuz there is a huge gap between result from valid set and public board"
  },
  "source": "meta"
}