{
  "id": 551359,
  "title": "date_id for Test Dataset",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/551359",
  "author_name": "",
  "post_date": "2024-12-12T19:37:25.645287400Z",
  "votes": 3,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Question: Should the date_id only be used for lagged feature creation and not as a direct feature? I am confused by the fact that in the test dataset, date_id starts at zero once again. I am trying to understand how this date_id compares to the training date_id and what the time difference is from the final value in the training set to the first in the test set. Given it starts at zero, I suspect that while many time series models capture a linear trend, it may not be able to translate this into the future using the date_id.</p>",
  "messages": [
    {
      "id": "3070527",
      "postDate": "12/12/2024 19:37:25",
      "content": "<p>Question: Should the date_id only be used for lagged feature creation and not as a direct feature? I am confused by the fact that in the test dataset, date_id starts at zero once again. I am trying to understand how this date_id compares to the training date_id and what the time difference is from the final value in the training set to the first in the test set. Given it starts at zero, I suspect that while many time series models capture a linear trend, it may not be able to translate this into the future using the date_id.</p>",
      "rawMarkdown": "Question: Should the date_id only be used for lagged feature creation and not as a direct feature? I am confused by the fact that in the test dataset, date_id starts at zero once again. I am trying to understand how this date_id compares to the training date_id and what the time difference is from the final value in the training set to the first in the test set. Given it starts at zero, I suspect that while many time series models capture a linear trend, it may not be able to translate this into the future using the date_id.",
      "votes": null
    },
    {
      "id": "3072035",
      "postDate": "12/14/2024 15:53:57",
      "content": "<p>Avoid Using date_id Directly: Instead, use derived features or lagged features that are invariant to the reset in date_id:<br>\nLagged values of the target variable.<br>\nRolling averages or differences.<br>\nTime-related categorical features (e.g., day of the week, month).<br>\nUnderstand the Gap: Determine the temporal gap between training and test datasets to properly interpret any time-dependent features.<br>\nFeature Engineering:<br>\nCreate a normalized or cumulative date_id that spans both training and test datasets.<br>\nUse additional time features (e.g., days_since_start).</p>",
      "rawMarkdown": "Avoid Using date_id Directly: Instead, use derived features or lagged features that are invariant to the reset in date_id:\nLagged values of the target variable.\nRolling averages or differences.\nTime-related categorical features (e.g., day of the week, month).\nUnderstand the Gap: Determine the temporal gap between training and test datasets to properly interpret any time-dependent features.\nFeature Engineering:\nCreate a normalized or cumulative date_id that spans both training and test datasets.\nUse additional time features (e.g., days_since_start).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3072035,
      "author_name": "amunsentom",
      "author_url": "",
      "post_date": "12/14/2024 15:53:57",
      "content": "<p>Avoid Using date_id Directly: Instead, use derived features or lagged features that are invariant to the reset in date_id:<br>\nLagged values of the target variable.<br>\nRolling averages or differences.<br>\nTime-related categorical features (e.g., day of the week, month).<br>\nUnderstand the Gap: Determine the temporal gap between training and test datasets to properly interpret any time-dependent features.<br>\nFeature Engineering:<br>\nCreate a normalized or cumulative date_id that spans both training and test datasets.<br>\nUse additional time features (e.g., days_since_start).</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3070527": "Question: Should the date_id only be used for lagged feature creation and not as a direct feature? I am confused by the fact that in the test dataset, date_id starts at zero once again. I am trying to understand how this date_id compares to the training date_id and what the time difference is from the final value in the training set to the first in the test set. Given it starts at zero, I suspect that while many time series models capture a linear trend, it may not be able to translate this into the future using the date_id.",
    "3072035": "Avoid Using date_id Directly: Instead, use derived features or lagged features that are invariant to the reset in date_id:\nLagged values of the target variable.\nRolling averages or differences.\nTime-related categorical features (e.g., day of the week, month).\nUnderstand the Gap: Determine the temporal gap between training and test datasets to properly interpret any time-dependent features.\nFeature Engineering:\nCreate a normalized or cumulative date_id that spans both training and test datasets.\nUse additional time features (e.g., days_since_start)."
  },
  "source": "meta"
}