{
  "id": 554668,
  "title": "Why not use all of the historical data to make a prediction?",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/554668",
  "author_name": "",
  "post_date": "2025-01-02T15:57:13.384924700Z",
  "votes": 1,
  "comment_count": 5,
  "views": 0,
  "content": "<p>This is my first competition and I'm a confused beginner…</p>\n<p>In all of the public notebooks I've examined, the models just use the 79 attributes (and lags) from the current time stamp to predict responder_6.</p>\n<p>It seems to me that a model that used the historical data from all previous time stamps would be clearly better but I do not see any public notebooks that do that.</p>\n<p>What am I missing?</p>",
  "messages": [
    {
      "id": "3086707",
      "postDate": "01/02/2025 15:57:13",
      "content": "<p>This is my first competition and I'm a confused beginner…</p>\n<p>In all of the public notebooks I've examined, the models just use the 79 attributes (and lags) from the current time stamp to predict responder_6.</p>\n<p>It seems to me that a model that used the historical data from all previous time stamps would be clearly better but I do not see any public notebooks that do that.</p>\n<p>What am I missing?</p>",
      "rawMarkdown": "This is my first competition and I'm a confused beginner...\n\nIn all of the public notebooks I've examined, the models just use the 79 attributes (and lags) from the current time stamp to predict responder_6.\n\nIt seems to me that a model that used the historical data from all previous time stamps would be clearly better but I do not see any public notebooks that do that.\n\nWhat am I missing?",
      "votes": null
    },
    {
      "id": "3086926",
      "postDate": "01/02/2025 21:10:29",
      "content": "<p>Public notebooks avoid using all historical data to simplify computation, prevent overfitting, and focus on real-time predictions. However, using historical data can capture long-term patterns, especially with models like LSTMs or Transformers. You can try a hybrid approach: use a recent window of historical data (e.g., last 100 time steps) or extract rolling statistics to keep the model efficient while leveraging past information. But I am not sure whether it will work or not.</p>",
      "rawMarkdown": "Public notebooks avoid using all historical data to simplify computation, prevent overfitting, and focus on real-time predictions. However, using historical data can capture long-term patterns, especially with models like LSTMs or Transformers. You can try a hybrid approach: use a recent window of historical data (e.g., last 100 time steps) or extract rolling statistics to keep the model efficient while leveraging past information. But I am not sure whether it will work or not.",
      "votes": null
    },
    {
      "id": "3086964",
      "postDate": "01/02/2025 22:45:34",
      "content": "<p>What you see in public is what people publish.<br>\nI can assure you that top places are using LSTM/GRU etc. on past time stamps.</p>",
      "rawMarkdown": "What you see in public is what people publish.\nI can assure you that top places are using LSTM/GRU etc. on past time stamps.",
      "votes": null
    },
    {
      "id": "3087523",
      "postDate": "01/03/2025 15:27:35",
      "content": "<p>Thank you, this is helpful.</p>",
      "rawMarkdown": "Thank you, this is helpful.",
      "votes": null
    },
    {
      "id": "3087524",
      "postDate": "01/03/2025 15:28:01",
      "content": "<p>Thanks for the helpful explanation.</p>",
      "rawMarkdown": "Thanks for the helpful explanation.",
      "votes": null
    },
    {
      "id": "3088119",
      "postDate": "01/04/2025 10:56:38",
      "content": "<p>We has not used past time stamps to participate in the prediction, and I am sure that using only current timestamps, such as a simple mlp, also can achieve a higher score</p>",
      "rawMarkdown": "We has not used past time stamps to participate in the prediction, and I am sure that using only current timestamps, such as a simple mlp, also can achieve a higher score",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3086926,
      "author_name": "rishavrajverma",
      "author_url": "",
      "post_date": "01/02/2025 21:10:29",
      "content": "<p>Public notebooks avoid using all historical data to simplify computation, prevent overfitting, and focus on real-time predictions. However, using historical data can capture long-term patterns, especially with models like LSTMs or Transformers. You can try a hybrid approach: use a recent window of historical data (e.g., last 100 time steps) or extract rolling statistics to keep the model efficient while leveraging past information. But I am not sure whether it will work or not.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3087524,
          "author_name": "michaeldelamaza",
          "author_url": "",
          "post_date": "01/03/2025 15:28:01",
          "content": "<p>Thanks for the helpful explanation.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3086964,
      "author_name": "shlomoron",
      "author_url": "",
      "post_date": "01/02/2025 22:45:34",
      "content": "<p>What you see in public is what people publish.<br>\nI can assure you that top places are using LSTM/GRU etc. on past time stamps.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3087523,
          "author_name": "michaeldelamaza",
          "author_url": "",
          "post_date": "01/03/2025 15:27:35",
          "content": "<p>Thank you, this is helpful.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 3088119,
          "author_name": "lechengyan",
          "author_url": "",
          "post_date": "01/04/2025 10:56:38",
          "content": "<p>We has not used past time stamps to participate in the prediction, and I am sure that using only current timestamps, such as a simple mlp, also can achieve a higher score</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3086707": "This is my first competition and I'm a confused beginner...\n\nIn all of the public notebooks I've examined, the models just use the 79 attributes (and lags) from the current time stamp to predict responder_6.\n\nIt seems to me that a model that used the historical data from all previous time stamps would be clearly better but I do not see any public notebooks that do that.\n\nWhat am I missing?",
    "3086926": "Public notebooks avoid using all historical data to simplify computation, prevent overfitting, and focus on real-time predictions. However, using historical data can capture long-term patterns, especially with models like LSTMs or Transformers. You can try a hybrid approach: use a recent window of historical data (e.g., last 100 time steps) or extract rolling statistics to keep the model efficient while leveraging past information. But I am not sure whether it will work or not.",
    "3086964": "What you see in public is what people publish.\nI can assure you that top places are using LSTM/GRU etc. on past time stamps.",
    "3087523": "Thank you, this is helpful.",
    "3087524": "Thanks for the helpful explanation.",
    "3088119": "We has not used past time stamps to participate in the prediction, and I am sure that using only current timestamps, such as a simple mlp, also can achieve a higher score"
  },
  "source": "meta"
}