{
  "id": 556421,
  "title": "Selecting the Best Model: Local Validation vs. Public Test Scores",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/556421",
  "author_name": "",
  "post_date": "2025-01-13T08:25:55.449419600Z",
  "votes": 2,
  "comment_count": 1,
  "views": 0,
  "content": "<p>In most competitions, we rely on the local validation score to select a robust model for test data.<br>\nHowever, in this competition, several strategies can be considered due to the characteristics of the time-series API (Online Training) and the fact that the private test set comes after the public test set, which itself is ahead of the training set.<br>\nI believe we should select the model with the best public test score for the final submission. This is because the public test set is the closest to the private test set and has a sufficiently large amount of data (4.5M).<br>\nWhat strategy would you choose?</p>",
  "messages": [
    {
      "id": "3095329",
      "postDate": "01/13/2025 08:25:55",
      "content": "<p>In most competitions, we rely on the local validation score to select a robust model for test data.<br>\nHowever, in this competition, several strategies can be considered due to the characteristics of the time-series API (Online Training) and the fact that the private test set comes after the public test set, which itself is ahead of the training set.<br>\nI believe we should select the model with the best public test score for the final submission. This is because the public test set is the closest to the private test set and has a sufficiently large amount of data (4.5M).<br>\nWhat strategy would you choose?</p>",
      "rawMarkdown": "In most competitions, we rely on the local validation score to select a robust model for test data.\nHowever, in this competition, several strategies can be considered due to the characteristics of the time-series API (Online Training) and the fact that the private test set comes after the public test set, which itself is ahead of the training set.\nI believe we should select the model with the best public test score for the final submission. This is because the public test set is the closest to the private test set and has a sufficiently large amount of data (4.5M).\nWhat strategy would you choose?",
      "votes": null
    },
    {
      "id": "3095766",
      "postDate": "01/13/2025 17:17:31",
      "content": "<p>Some other approaches could be:</p>\n<ul>\n<li>Only pick models that improves both your local validation and the public LB, maybe by some margin</li>\n<li>Do a weighted average of the local validation and public LB</li>\n</ul>",
      "rawMarkdown": "Some other approaches could be:\n\n- Only pick models that improves both your local validation and the public LB, maybe by some margin\n- Do a weighted average of the local validation and public LB",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3095766,
      "author_name": "redfoongus",
      "author_url": "",
      "post_date": "01/13/2025 17:17:31",
      "content": "<p>Some other approaches could be:</p>\n<ul>\n<li>Only pick models that improves both your local validation and the public LB, maybe by some margin</li>\n<li>Do a weighted average of the local validation and public LB</li>\n</ul>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3095329": "In most competitions, we rely on the local validation score to select a robust model for test data.\nHowever, in this competition, several strategies can be considered due to the characteristics of the time-series API (Online Training) and the fact that the private test set comes after the public test set, which itself is ahead of the training set.\nI believe we should select the model with the best public test score for the final submission. This is because the public test set is the closest to the private test set and has a sufficiently large amount of data (4.5M).\nWhat strategy would you choose?",
    "3095766": "Some other approaches could be:\n\n- Only pick models that improves both your local validation and the public LB, maybe by some margin\n- Do a weighted average of the local validation and public LB"
  },
  "source": "meta"
}