{ "id": 547710, "title": "How will the test data be delivered during the private board evaluation period", "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/547710", "author_name": "", "post_date": "2024-11-23T05:54:45.870846500Z", "votes": 2, "comment_count": 6, "views": 0, "content": "

Hi, I just joined the competition and would like to get a clarification on the test data set during the private board evaluation period. Suppose:

\n
    \n

  1. We are at the beginning of this period, then the public test set will be provided in test and its corresponding labels in lags?

    2. \n

  2. Then two weeks later we are at the second evaluation, and the public test set + first-two-week private test set will be provided in test and their corresponding labels in lags? And for the third evaluation it will be public test set + first-one-month private test set, and so on for forth, fifth, …

    3. \n
\n

Thanks!

", "messages": [ { "id": "3053074", "postDate": "11/23/2024 05:54:45", "content": "

Hi, I just joined the competition and would like to get a clarification on the test data set during the private board evaluation period. Suppose:

\n
    \n

  1. We are at the beginning of this period, then the public test set will be provided in test and its corresponding labels in lags?

    2. \n

  2. Then two weeks later we are at the second evaluation, and the public test set + first-two-week private test set will be provided in test and their corresponding labels in lags? And for the third evaluation it will be public test set + first-one-month private test set, and so on for forth, fifth, …

    3. \n
\n

Thanks!

", "rawMarkdown": "Hi, I just joined the competition and would like to get a clarification on the test data set during the private board evaluation period. Suppose:\n\n1. We are at the beginning of this period, then the **public test set** will be provided in **test** and its corresponding labels in **lags**?\n\n2. Then two weeks later we are at the second evaluation, and the **public test set + first-two-week private test set** will be provided in **test** and their corresponding labels in **lags**? And for the third evaluation it will be **public test set + first-one-month private test set**, and so on for forth, fifth, ...\n\nThanks!", "votes": null }, { "id": "3054495", "postDate": "11/24/2024 18:18:46", "content": "

Not sure if it answers your question but I understand it as follows: the cut-off date is 13th Jan 2025.

\n

There are roughly 3 components:

\n
    \n
  1. Existing test set (for the current leaderboard): ~100-150 days worth of data (based on other discussions)
    2. \n
  2. \"Extended\" test set: This is (1) + data collected closer to the cut-off date - they will extend the test set but this will not be scored or counted to the current leaderboard - it is rather for you to check your notebook runs in time and you don't get any runtime errors etc.
    3. \n
  3. Private test set: Data collected from 13th Jan 2025 to around July 2025 (i.e. around half a year worth of data)
    4. \n
\n

The notebook will then be run on (1) + (2) + (3) but the private score will be on (3).

\n

For the lags: the way the API works is as follows (also based on my own understanding).
\nFor each date_id and time_id, test set is served in form of test and lag. So if in (1), there are 150 days and 900 time_id's, then you will have 150x900 = 135,000 iterations.

\n", "rawMarkdown": "Not sure if it answers your question but I understand it as follows: the cut-off date is 13th Jan 2025.\n\nThere are roughly 3 components:\n1. Existing test set (for the current leaderboard): ~100-150 days worth of data (based on other discussions)\n2. \"Extended\" test set: This is (1) + data collected closer to the cut-off date - they will extend the test set but this will not be scored or counted to the current leaderboard - it is rather for you to check your notebook runs in time and you don't get any runtime errors etc.\n3. Private test set: Data collected from 13th Jan 2025 to around July 2025 (i.e. around half a year worth of data)\n\nThe notebook will then be run on (1) + (2) + (3) but the private score will be on (3).\n\nFor the lags: the way the API works is as follows (also based on my own understanding).\nFor each date_id and time_id, test set is served in form of `test` and `lag`. So if in (1), there are 150 days and 900 time_id's, then you will have 150x900 = 135,000 iterations.\n- When time_id = 0, lags is a data frame with responder values from the previous day. So e.g. date_id = 1701 and time_id = 0, then you will get the responder_values for all time_id's and symbol_id's from date_id=1700.\n- When time_id != 0, lags = None.", "votes": null }, { "id": "3055572", "postDate": "11/25/2024 21:51:15", "content": "

Thanks for sharing your understanding👍 This looks very clear to me.

\n

Just want to clarify that:

\n
\n

The notebook will then be run on (1) + (2) + (3) but the private score will be on (3)

\n
\n

If we are in the middle May and (1) has 150 days, (2) has 150 days, and (3) has 120 days (around 4 months), then we will be provided 420 days' data in the test data set, and only the most recent 14 days (an assumption as the private set will be updated around every two weeks) will be scored?

", "rawMarkdown": "Thanks for sharing your understanding👍 This looks very clear to me.\n\nJust want to clarify that:\n>The notebook will then be run on (1) + (2) + (3) but the private score will be on (3)\n\nIf we are in the middle May and (1) has 150 days, (2) has 150 days, and (3) has 120 days (around 4 months), then we will be provided 420 days' data in the test data set, and only the most recent 14 days (an assumption as the private set will be updated around every two weeks) will be scored?", "votes": null }, { "id": "3055589", "postDate": "11/25/2024 22:14:00", "content": "
\n

The notebook will then be run on (1) + (2) + (3) but the private score will be on (3).

\n
\n

So our code should be expected to predict on ~ a year's worth of data in 8 hours, where the predict for each day must be < 1 min?

", "rawMarkdown": "> The notebook will then be run on (1) + (2) + (3) but the private score will be on (3).\n\nSo our code should be expected to predict on ~ a year's worth of data in 8 hours, where the predict for each day must be < 1 min?", "votes": null }, { "id": "3055643", "postDate": "11/26/2024 00:37:13", "content": "

No, I think in that case the score will be on the (3), i.e. on the entire 120 days. It is just that they add new data every 2 weeks up until July or so, so you can continually check on how your model is performing as new data comes in.

", "rawMarkdown": "No, I **think** in that case the score will be on the (3), i.e. on the entire 120 days. It is just that they add new data every 2 weeks up until July or so, so you can continually check on how your model is performing as new data comes in.", "votes": null }, { "id": "3055644", "postDate": "11/26/2024 00:39:03", "content": "

Yeah I think so. The predict itself doesn't need to run within 1 minute for each day, but I guess it would be smart to do so (e.g. 360 days -> 360 minutes = 6 hours)

", "rawMarkdown": "Yeah I think so. The predict itself doesn't need to run within 1 minute for each day, but I guess it would be smart to do so (e.g. 360 days -> 360 minutes = 6 hours)", "votes": null }, { "id": "3055738", "postDate": "11/26/2024 04:29:23", "content": "

Only half a year needed to be predicted (from Jan. to Jul.) and the previous data can be ignored I guess.

", "rawMarkdown": "Only half a year needed to be predicted (from Jan. to Jul.) and the previous data can be ignored I guess.", "votes": null } ], "comments": [ { "id": 3054495, "author_name": "kevinlam", "author_url": "", "post_date": "11/24/2024 18:18:46", "content": "

Not sure if it answers your question but I understand it as follows: the cut-off date is 13th Jan 2025.

\n

There are roughly 3 components:

\n
    \n
  1. Existing test set (for the current leaderboard): ~100-150 days worth of data (based on other discussions)
    2. \n
  2. \"Extended\" test set: This is (1) + data collected closer to the cut-off date - they will extend the test set but this will not be scored or counted to the current leaderboard - it is rather for you to check your notebook runs in time and you don't get any runtime errors etc.
    3. \n
  3. Private test set: Data collected from 13th Jan 2025 to around July 2025 (i.e. around half a year worth of data)
    4. \n
\n

The notebook will then be run on (1) + (2) + (3) but the private score will be on (3).

\n

For the lags: the way the API works is as follows (also based on my own understanding).
\nFor each date_id and time_id, test set is served in form of test and lag. So if in (1), there are 150 days and 900 time_id's, then you will have 150x900 = 135,000 iterations.

\n", "votes": null, "replies": [ { "id": 3055572, "author_name": "makeli", "author_url": "", "post_date": "11/25/2024 21:51:15", "content": "

Thanks for sharing your understanding👍 This looks very clear to me.

\n

Just want to clarify that:

\n
\n

The notebook will then be run on (1) + (2) + (3) but the private score will be on (3)

\n
\n

If we are in the middle May and (1) has 150 days, (2) has 150 days, and (3) has 120 days (around 4 months), then we will be provided 420 days' data in the test data set, and only the most recent 14 days (an assumption as the private set will be updated around every two weeks) will be scored?

", "votes": null, "replies": [ { "id": 3055643, "author_name": "kevinlam", "author_url": "", "post_date": "11/26/2024 00:37:13", "content": "

No, I think in that case the score will be on the (3), i.e. on the entire 120 days. It is just that they add new data every 2 weeks up until July or so, so you can continually check on how your model is performing as new data comes in.

", "votes": null, "replies": [] } ] }, { "id": 3055589, "author_name": "redfoongus", "author_url": "", "post_date": "11/25/2024 22:14:00", "content": "
\n

The notebook will then be run on (1) + (2) + (3) but the private score will be on (3).

\n
\n

So our code should be expected to predict on ~ a year's worth of data in 8 hours, where the predict for each day must be < 1 min?

", "votes": null, "replies": [ { "id": 3055644, "author_name": "kevinlam", "author_url": "", "post_date": "11/26/2024 00:39:03", "content": "

Yeah I think so. The predict itself doesn't need to run within 1 minute for each day, but I guess it would be smart to do so (e.g. 360 days -> 360 minutes = 6 hours)

", "votes": null, "replies": [] }, { "id": 3055738, "author_name": "makeli", "author_url": "", "post_date": "11/26/2024 04:29:23", "content": "

Only half a year needed to be predicted (from Jan. to Jul.) and the previous data can be ignored I guess.

", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3053074": "Hi, I just joined the competition and would like to get a clarification on the test data set during the private board evaluation period. Suppose:\n\n1. We are at the beginning of this period, then the **public test set** will be provided in **test** and its corresponding labels in **lags**?\n\n2. Then two weeks later we are at the second evaluation, and the **public test set + first-two-week private test set** will be provided in **test** and their corresponding labels in **lags**? And for the third evaluation it will be **public test set + first-one-month private test set**, and so on for forth, fifth, ...\n\nThanks!", "3054495": "Not sure if it answers your question but I understand it as follows: the cut-off date is 13th Jan 2025.\n\nThere are roughly 3 components:\n1. Existing test set (for the current leaderboard): ~100-150 days worth of data (based on other discussions)\n2. \"Extended\" test set: This is (1) + data collected closer to the cut-off date - they will extend the test set but this will not be scored or counted to the current leaderboard - it is rather for you to check your notebook runs in time and you don't get any runtime errors etc.\n3. Private test set: Data collected from 13th Jan 2025 to around July 2025 (i.e. around half a year worth of data)\n\nThe notebook will then be run on (1) + (2) + (3) but the private score will be on (3).\n\nFor the lags: the way the API works is as follows (also based on my own understanding).\nFor each date_id and time_id, test set is served in form of `test` and `lag`. So if in (1), there are 150 days and 900 time_id's, then you will have 150x900 = 135,000 iterations.\n- When time_id = 0, lags is a data frame with responder values from the previous day. So e.g. date_id = 1701 and time_id = 0, then you will get the responder_values for all time_id's and symbol_id's from date_id=1700.\n- When time_id != 0, lags = None.", "3055572": "Thanks for sharing your understanding👍 This looks very clear to me.\n\nJust want to clarify that:\n>The notebook will then be run on (1) + (2) + (3) but the private score will be on (3)\n\nIf we are in the middle May and (1) has 150 days, (2) has 150 days, and (3) has 120 days (around 4 months), then we will be provided 420 days' data in the test data set, and only the most recent 14 days (an assumption as the private set will be updated around every two weeks) will be scored?", "3055589": "> The notebook will then be run on (1) + (2) + (3) but the private score will be on (3).\n\nSo our code should be expected to predict on ~ a year's worth of data in 8 hours, where the predict for each day must be < 1 min?", "3055643": "No, I **think** in that case the score will be on the (3), i.e. on the entire 120 days. It is just that they add new data every 2 weeks up until July or so, so you can continually check on how your model is performing as new data comes in.", "3055644": "Yeah I think so. The predict itself doesn't need to run within 1 minute for each day, but I guess it would be smart to do so (e.g. 360 days -> 360 minutes = 6 hours)", "3055738": "Only half a year needed to be predicted (from Jan. to Jul.) and the previous data can be ignored I guess." }, "source": "meta" }