{
  "id": 544232,
  "title": "Can we use the historical features for test stage?",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/544232",
  "author_name": "",
  "post_date": "2024-11-04T01:43:03.965357400Z",
  "votes": 1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>We can guarantee that we can use the previous date_id lagged responder. (from data description)</p>\n<p>As this way, can we use the lagged features from train data? (It seems to be not)</p>\n<p>If date_ids are vary and ambiguous in test data, we can't use  the lagged features from train data.</p>\n<p>(This also imply that we can't use algorithm for timeseries data since we can't make sequnece input)</p>",
  "messages": [
    {
      "id": "3035904",
      "postDate": "11/04/2024 01:43:03",
      "content": "<p>We can guarantee that we can use the previous date_id lagged responder. (from data description)</p>\n<p>As this way, can we use the lagged features from train data? (It seems to be not)</p>\n<p>If date_ids are vary and ambiguous in test data, we can't use  the lagged features from train data.</p>\n<p>(This also imply that we can't use algorithm for timeseries data since we can't make sequnece input)</p>",
      "rawMarkdown": "We can guarantee that we can use the previous date_id lagged responder. (from data description)\n\nAs this way, can we use the lagged features from train data? (It seems to be not)\n\nIf date_ids are vary and ambiguous in test data, we can't use  the lagged features from train data.\n\n(This also imply that we can't use algorithm for timeseries data since we can't make sequnece input)",
      "votes": null
    },
    {
      "id": "3036092",
      "postDate": "11/04/2024 08:01:03",
      "content": "<p>I'd be careful about this part:</p>\n<blockquote>\n  <p>We can guarantee that we can use the previous date_id lagged responder. (from data description)</p>\n</blockquote>\n<p>In the training data, we see that the symbol_ids available for each date_id are different, so we can expect this to be the case in the test data too. It's best to assume that lagged responder values may be missing for some symbol_ids. I actually run a submission to check that and confirmed that lags for some symbol_ids may be missing in the test data.</p>",
      "rawMarkdown": "I'd be careful about this part:\n\n> We can guarantee that we can use the previous date_id lagged responder. (from data description)\n\nIn the training data, we see that the symbol_ids available for each date_id are different, so we can expect this to be the case in the test data too. It's best to assume that lagged responder values may be missing for some symbol_ids. I actually run a submission to check that and confirmed that lags for some symbol_ids may be missing in the test data.",
      "votes": null
    },
    {
      "id": "3036827",
      "postDate": "11/05/2024 00:41:02",
      "content": "<p>Yes, I am agree with you. We assume some symbol_id is missing and we may had better train with averaged responder by symbol_id.<br>\nBut this let me get another question.. If symbol_id is missing, what is the symbol_id of first timestep value (time_id=0) from lags?<br>\nAveraged ? or just first one ?<br>\nThere are some things confused..</p>",
      "rawMarkdown": "Yes, I am agree with you. We assume some symbol_id is missing and we may had better train with averaged responder by symbol_id.\nBut this let me get another question.. If symbol_id is missing, what is the symbol_id of first timestep value (time_id=0) from lags?\nAveraged ? or just first one ?\nThere are some things confused..",
      "votes": null
    },
    {
      "id": "3036920",
      "postDate": "11/05/2024 03:55:09",
      "content": "<p><a href=\"https://www.kaggle.com/cafelatte1\" target=\"_blank\">@cafelatte1</a> I cannot understand how the lags will matter if the symbol id does not exist on a given date. The lagged value won't be used at all in this case. </p>",
      "rawMarkdown": "cafelatte1 I cannot understand how the lags will matter if the symbol id does not exist on a given date. The lagged value won't be used at all in this case.",
      "votes": null
    },
    {
      "id": "3036990",
      "postDate": "11/05/2024 07:12:48",
      "content": "<p>Each entry will have the symbol_id attached to it, and it will be a smallest symbol_id present on the previous day.</p>",
      "rawMarkdown": "Each entry will have the symbol_id attached to it, and it will be a smallest symbol_id present on the previous day.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3036092,
      "author_name": "shuthdar",
      "author_url": "",
      "post_date": "11/04/2024 08:01:03",
      "content": "<p>I'd be careful about this part:</p>\n<blockquote>\n  <p>We can guarantee that we can use the previous date_id lagged responder. (from data description)</p>\n</blockquote>\n<p>In the training data, we see that the symbol_ids available for each date_id are different, so we can expect this to be the case in the test data too. It's best to assume that lagged responder values may be missing for some symbol_ids. I actually run a submission to check that and confirmed that lags for some symbol_ids may be missing in the test data.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3036827,
          "author_name": "cafelatte1",
          "author_url": "",
          "post_date": "11/05/2024 00:41:02",
          "content": "<p>Yes, I am agree with you. We assume some symbol_id is missing and we may had better train with averaged responder by symbol_id.<br>\nBut this let me get another question.. If symbol_id is missing, what is the symbol_id of first timestep value (time_id=0) from lags?<br>\nAveraged ? or just first one ?<br>\nThere are some things confused..</p>",
          "votes": null,
          "replies": [
            {
              "id": 3036920,
              "author_name": "ravi20076",
              "author_url": "",
              "post_date": "11/05/2024 03:55:09",
              "content": "<p><a href=\"https://www.kaggle.com/cafelatte1\" target=\"_blank\">@cafelatte1</a> I cannot understand how the lags will matter if the symbol id does not exist on a given date. The lagged value won't be used at all in this case. </p>",
              "votes": null,
              "replies": []
            },
            {
              "id": 3036990,
              "author_name": "shuthdar",
              "author_url": "",
              "post_date": "11/05/2024 07:12:48",
              "content": "<p>Each entry will have the symbol_id attached to it, and it will be a smallest symbol_id present on the previous day.</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3035904": "We can guarantee that we can use the previous date_id lagged responder. (from data description)\n\nAs this way, can we use the lagged features from train data? (It seems to be not)\n\nIf date_ids are vary and ambiguous in test data, we can't use  the lagged features from train data.\n\n(This also imply that we can't use algorithm for timeseries data since we can't make sequnece input)",
    "3036092": "I'd be careful about this part:\n\n> We can guarantee that we can use the previous date_id lagged responder. (from data description)\n\nIn the training data, we see that the symbol_ids available for each date_id are different, so we can expect this to be the case in the test data too. It's best to assume that lagged responder values may be missing for some symbol_ids. I actually run a submission to check that and confirmed that lags for some symbol_ids may be missing in the test data.",
    "3036827": "Yes, I am agree with you. We assume some symbol_id is missing and we may had better train with averaged responder by symbol_id.\nBut this let me get another question.. If symbol_id is missing, what is the symbol_id of first timestep value (time_id=0) from lags?\nAveraged ? or just first one ?\nThere are some things confused..",
    "3036920": "cafelatte1 I cannot understand how the lags will matter if the symbol id does not exist on a given date. The lagged value won't be used at all in this case.",
    "3036990": "Each entry will have the symbol_id attached to it, and it will be a smallest symbol_id present on the previous day."
  },
  "source": "meta"
}