{ "id": 544838, "title": "The lags feature in API is HARD to understand", "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/544838", "author_name": "", "post_date": "2024-11-07T06:58:31.676423600Z", "votes": 7, "comment_count": 11, "views": 0, "content": "

I am trying to use the lag feature, but there are several problems.

\n

1. Lags can be None?

\n

The lags served some time are None, because when my code merge the lag feature into the test dataframe, after submission, it will error out.
\nAnd I took a test in the debug with this https://www.kaggle.com/code/chumajin/janestreet-updated-simulator-for-time-series-api, the lag can be None. At least in this mock api, it will only serve one lags dataframe and later loops are all None.
\nIts hard to debug this problem, any insights?

\n

2. Lags date_id

\n

This part is really unclear to me. Assume the test's date_id be 5, will the lag date_id be 4 or is it maching 5? since it is lagged.
\nI am assuming date_id will match, but who knows.. any insights?

", "messages": [ { "id": "3038632", "postDate": "11/07/2024 06:58:31", "content": "

I am trying to use the lag feature, but there are several problems.

\n

1. Lags can be None?

\n

The lags served some time are None, because when my code merge the lag feature into the test dataframe, after submission, it will error out.
\nAnd I took a test in the debug with this https://www.kaggle.com/code/chumajin/janestreet-updated-simulator-for-time-series-api, the lag can be None. At least in this mock api, it will only serve one lags dataframe and later loops are all None.
\nIts hard to debug this problem, any insights?

\n

2. Lags date_id

\n

This part is really unclear to me. Assume the test's date_id be 5, will the lag date_id be 4 or is it maching 5? since it is lagged.
\nI am assuming date_id will match, but who knows.. any insights?

", "rawMarkdown": "I am trying to use the lag feature, but there are several problems.\n\n**1. Lags can be None?**\n\nThe lags served some time are None, because when my code merge the lag feature into the test dataframe, after submission, it will error out. \nAnd I took a test in the debug with this [https://www.kaggle.com/code/chumajin/janestreet-updated-simulator-for-time-series-api](url), the lag can be None. At least in this mock api, it will only serve one lags dataframe and later loops are all None. \nIts hard to debug this problem, any insights?\n\n**2. Lags date_id**\n\nThis part is really unclear to me. Assume the test's date_id be 5, will the lag date_id be 4 or is it maching 5? since it is lagged.\nI am assuming date_id will match, but who knows.. any insights?", "votes": null }, { "id": "3038634", "postDate": "11/07/2024 07:02:24", "content": "

Check this thread to clear it out - https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/543567

", "rawMarkdown": "Check this thread to clear it out - https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/543567", "votes": null }, { "id": "3038678", "postDate": "11/07/2024 08:02:18", "content": "

I think you are the referring to the lags argument passed to the predict function, if so then yes the lags can be none but first you need to understand how the API works.
\nthe API call the predict function multiple times , and it does not pass all the test data and lags at once. instead each call to the predict function passes one time_id worth of test data.
\nthe lags are passes in a different manner. each time the test data passed to the predict function reaches time_id ==0 (ie new day ) the API passes all the lags data that correspond to that day. so if you want to use lags then you should store them in a local variable like this
\nglobal lags_\n if lags is not None:\n lags_ = lags
\nthis makes the global lags_ updated each day (ie when it's provided by the API) and you can use then in later calls with the other time_ids till it gets updated again.
\n2 in the data provided by the competition the test date_id and lags test_id appears to be the same. however since the lags and test passed to the predict function are for the date_id you can join them based on time_id and symbol_id only.

", "rawMarkdown": "I think you are the referring to the lags argument passed to the predict function, if so then yes the lags can be none but first you need to understand how the API works.\nthe API call the predict function multiple times , and it does not pass all the test data and lags at once. instead each call to the predict function passes one time_id worth of test data. \nthe lags are passes in a different manner. each time the test data passed to the predict function reaches time_id ==0 (ie new day ) the API passes all the lags data that correspond to that day. so if you want to use lags then you should store them in a local variable like this \n`global lags_\n if lags is not None:\n lags_ = lags`\nthis makes the global lags_ updated each day (ie when it's provided by the API) and you can use then in later calls with the other time_ids till it gets updated again. \n2 in the data provided by the competition the test date_id and lags test_id appears to be the same. however since the lags and test passed to the predict function are for the date_id you can join them based on time_id and symbol_id only.", "votes": null }, { "id": "3038806", "postDate": "11/07/2024 11:27:01", "content": "

thanks for your explaination. it is helpful! I saw that lags_ there but never know the reason. 😁

", "rawMarkdown": "thanks for your explaination. it is helpful! I saw that lags_ there but never know the reason. 😁", "votes": null }, { "id": "3038807", "postDate": "11/07/2024 11:27:45", "content": "

Great! Thank you

", "rawMarkdown": "Great! Thank you", "votes": null }, { "id": "3038824", "postDate": "11/07/2024 12:04:39", "content": "

So for example, at
\ndate_id == 1, and time_id==0, the API will pass \"lag\" and \"test\".
\ndate_id == 1, and time_id==1, we will only get \"test\" for time==1, and it will keep calling different time_id
\n….
\n…. until
\ndate_id == 2, and time_id==0, we will get \"lag\" and \"test\"

\n

I hope I got the above correct

", "rawMarkdown": "So for example, at \ndate_id == 1, and time_id==0, the API will pass \"lag\" and \"test\".\ndate_id == 1, and time_id==1, we will only get \"test\" for time==1, and it will keep calling different time_id\n....\n.... until\ndate_id == 2, and time_id==0, we will get \"lag\" and \"test\"\n\nI hope I got the above correct", "votes": null }, { "id": "3039138", "postDate": "11/07/2024 17:42:29", "content": "

yes that's it. one more thing to add there are some time limits you need to keep in mind.

\n\n

try to measure the execution time for a single predict call from the API to get a sense on how long it takes to score a submission.

", "rawMarkdown": "yes that's it. one more thing to add there are some time limits you need to keep in mind.\n- The run time of the notebook should not pass 8h.\n- the API should be called before 15 mins. so it's better to make a separate notebook for training and another for inference.\n- each call of the API must take less than 10s except the first call has no time limit so you load your models during the first call if the first 15 mins is not enough.\n- even if the time limit for a single predict call is 10s you should try to keep the execution time of a single predict call around 150 ms else you will exceed the 8h time limit.\n\ntry to measure the execution time for a single predict call from the API to get a sense on how long it takes to score a submission.", "votes": null }, { "id": "3039320", "postDate": "11/07/2024 23:03:19", "content": "

good to know the 150 ms! That could be one of the reason I am keep failling the submission! I took 535ms !

", "rawMarkdown": "good to know the 150 ms! That could be one of the reason I am keep failling the submission! I took 535ms !", "votes": null }, { "id": "3039333", "postDate": "11/07/2024 23:30:46", "content": "

tbh i test it on the a draft session by executing the cell multiple times to see how it does on average. cause one execution can be misleading. especially the first one. sometimes i get like 5 seconds then when i rerun the cell it get down to 200 ms!

", "rawMarkdown": "tbh i test it on the a draft session by executing the cell multiple times to see how it does on average. cause one execution can be misleading. especially the first one. sometimes i get like 5 seconds then when i rerun the cell it get down to 200 ms!", "votes": null }, { "id": "3039346", "postDate": "11/08/2024 00:08:35", "content": "

you are right…. I cant figure out why I am failing the submission. nothing turns wrong. debug test ran alright.!

\n

\"\"

\n

\"\"

\n

\"\"

", "rawMarkdown": "you are right.... I cant figure out why I am failing the submission. nothing turns wrong. debug test ran alright.!\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4858569%2F5d8295a5c7f3b2378f6a59a8aa1123d6%2Fwechat_2024-11-08_081022_238.png?generation=1731024633951394&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4858569%2Fb9077c08ff4f81cb72d8c99a08f33c97%2F2024-11-08_075648_126.png?generation=1731024645420480&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4858569%2F14524ccd5a83a1e1ea677d048de0073c%2F2024-11-08_080752_379.png?generation=1731024651867644&alt=media)", "votes": null }, { "id": "3044850", "postDate": "11/13/2024 22:24:46", "content": "

@younesbenalia

\n

Could you clarify if I understood correctly? At time_id == 0, does the lags parameter provide a DataFrame containing different lag values for each responder_id across various time_ids? Or do we just receive a DataFrame with lags specifically for time_id == 0?

", "rawMarkdown": "younesbenalia\n\nCould you clarify if I understood correctly? At time_id == 0, does the lags parameter provide a DataFrame containing different lag values for each responder_id across various time_ids? Or do we just receive a DataFrame with lags specifically for time_id == 0?", "votes": null }, { "id": "3045123", "postDate": "11/14/2024 07:29:17", "content": "

at time_id == 0, we get all the time_ids, possiblely from 0 to 970ish, which is lagged by 1 date_id, which means its yesterday's all time_id data

", "rawMarkdown": "at time_id == 0, we get all the time_ids, possiblely from 0 to 970ish, which is lagged by 1 date_id, which means its yesterday's all time_id data", "votes": null } ], "comments": [ { "id": 3038634, "author_name": "eu1234", "author_url": "", "post_date": "11/07/2024 07:02:24", "content": "

Check this thread to clear it out - https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/543567

", "votes": null, "replies": [ { "id": 3038807, "author_name": "zoutain", "author_url": "", "post_date": "11/07/2024 11:27:45", "content": "

Great! Thank you

", "votes": null, "replies": [] } ] }, { "id": 3038678, "author_name": "younesbenalia", "author_url": "", "post_date": "11/07/2024 08:02:18", "content": "

I think you are the referring to the lags argument passed to the predict function, if so then yes the lags can be none but first you need to understand how the API works.
\nthe API call the predict function multiple times , and it does not pass all the test data and lags at once. instead each call to the predict function passes one time_id worth of test data.
\nthe lags are passes in a different manner. each time the test data passed to the predict function reaches time_id ==0 (ie new day ) the API passes all the lags data that correspond to that day. so if you want to use lags then you should store them in a local variable like this
\nglobal lags_\n if lags is not None:\n lags_ = lags
\nthis makes the global lags_ updated each day (ie when it's provided by the API) and you can use then in later calls with the other time_ids till it gets updated again.
\n2 in the data provided by the competition the test date_id and lags test_id appears to be the same. however since the lags and test passed to the predict function are for the date_id you can join them based on time_id and symbol_id only.

", "votes": null, "replies": [ { "id": 3038806, "author_name": "zoutain", "author_url": "", "post_date": "11/07/2024 11:27:01", "content": "

thanks for your explaination. it is helpful! I saw that lags_ there but never know the reason. 😁

", "votes": null, "replies": [] }, { "id": 3038824, "author_name": "zoutain", "author_url": "", "post_date": "11/07/2024 12:04:39", "content": "

So for example, at
\ndate_id == 1, and time_id==0, the API will pass \"lag\" and \"test\".
\ndate_id == 1, and time_id==1, we will only get \"test\" for time==1, and it will keep calling different time_id
\n….
\n…. until
\ndate_id == 2, and time_id==0, we will get \"lag\" and \"test\"

\n

I hope I got the above correct

", "votes": null, "replies": [ { "id": 3039138, "author_name": "younesbenalia", "author_url": "", "post_date": "11/07/2024 17:42:29", "content": "

yes that's it. one more thing to add there are some time limits you need to keep in mind.

\n\n

try to measure the execution time for a single predict call from the API to get a sense on how long it takes to score a submission.

", "votes": null, "replies": [ { "id": 3039320, "author_name": "zoutain", "author_url": "", "post_date": "11/07/2024 23:03:19", "content": "

good to know the 150 ms! That could be one of the reason I am keep failling the submission! I took 535ms !

", "votes": null, "replies": [ { "id": 3039333, "author_name": "younesbenalia", "author_url": "", "post_date": "11/07/2024 23:30:46", "content": "

tbh i test it on the a draft session by executing the cell multiple times to see how it does on average. cause one execution can be misleading. especially the first one. sometimes i get like 5 seconds then when i rerun the cell it get down to 200 ms!

", "votes": null, "replies": [ { "id": 3039346, "author_name": "zoutain", "author_url": "", "post_date": "11/08/2024 00:08:35", "content": "

you are right…. I cant figure out why I am failing the submission. nothing turns wrong. debug test ran alright.!

\n

\"\"

\n

\"\"

\n

\"\"

", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 3044850, "author_name": "nuinashco", "author_url": "", "post_date": "11/13/2024 22:24:46", "content": "

@younesbenalia

\n

Could you clarify if I understood correctly? At time_id == 0, does the lags parameter provide a DataFrame containing different lag values for each responder_id across various time_ids? Or do we just receive a DataFrame with lags specifically for time_id == 0?

", "votes": null, "replies": [ { "id": 3045123, "author_name": "zoutain", "author_url": "", "post_date": "11/14/2024 07:29:17", "content": "

at time_id == 0, we get all the time_ids, possiblely from 0 to 970ish, which is lagged by 1 date_id, which means its yesterday's all time_id data

", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3038632": "I am trying to use the lag feature, but there are several problems.\n\n**1. Lags can be None?**\n\nThe lags served some time are None, because when my code merge the lag feature into the test dataframe, after submission, it will error out. \nAnd I took a test in the debug with this [https://www.kaggle.com/code/chumajin/janestreet-updated-simulator-for-time-series-api](url), the lag can be None. At least in this mock api, it will only serve one lags dataframe and later loops are all None. \nIts hard to debug this problem, any insights?\n\n**2. Lags date_id**\n\nThis part is really unclear to me. Assume the test's date_id be 5, will the lag date_id be 4 or is it maching 5? since it is lagged.\nI am assuming date_id will match, but who knows.. any insights?", "3038634": "Check this thread to clear it out - https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/543567", "3038678": "I think you are the referring to the lags argument passed to the predict function, if so then yes the lags can be none but first you need to understand how the API works.\nthe API call the predict function multiple times , and it does not pass all the test data and lags at once. instead each call to the predict function passes one time_id worth of test data. \nthe lags are passes in a different manner. each time the test data passed to the predict function reaches time_id ==0 (ie new day ) the API passes all the lags data that correspond to that day. so if you want to use lags then you should store them in a local variable like this \n`global lags_\n if lags is not None:\n lags_ = lags`\nthis makes the global lags_ updated each day (ie when it's provided by the API) and you can use then in later calls with the other time_ids till it gets updated again. \n2 in the data provided by the competition the test date_id and lags test_id appears to be the same. however since the lags and test passed to the predict function are for the date_id you can join them based on time_id and symbol_id only.", "3038806": "thanks for your explaination. it is helpful! I saw that lags_ there but never know the reason. 😁", "3038807": "Great! Thank you", "3038824": "So for example, at \ndate_id == 1, and time_id==0, the API will pass \"lag\" and \"test\".\ndate_id == 1, and time_id==1, we will only get \"test\" for time==1, and it will keep calling different time_id\n....\n.... until\ndate_id == 2, and time_id==0, we will get \"lag\" and \"test\"\n\nI hope I got the above correct", "3039138": "yes that's it. one more thing to add there are some time limits you need to keep in mind.\n- The run time of the notebook should not pass 8h.\n- the API should be called before 15 mins. so it's better to make a separate notebook for training and another for inference.\n- each call of the API must take less than 10s except the first call has no time limit so you load your models during the first call if the first 15 mins is not enough.\n- even if the time limit for a single predict call is 10s you should try to keep the execution time of a single predict call around 150 ms else you will exceed the 8h time limit.\n\ntry to measure the execution time for a single predict call from the API to get a sense on how long it takes to score a submission.", "3039320": "good to know the 150 ms! That could be one of the reason I am keep failling the submission! I took 535ms !", "3039333": "tbh i test it on the a draft session by executing the cell multiple times to see how it does on average. cause one execution can be misleading. especially the first one. sometimes i get like 5 seconds then when i rerun the cell it get down to 200 ms!", "3039346": "you are right.... I cant figure out why I am failing the submission. nothing turns wrong. debug test ran alright.!\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4858569%2F5d8295a5c7f3b2378f6a59a8aa1123d6%2Fwechat_2024-11-08_081022_238.png?generation=1731024633951394&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4858569%2Fb9077c08ff4f81cb72d8c99a08f33c97%2F2024-11-08_075648_126.png?generation=1731024645420480&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4858569%2F14524ccd5a83a1e1ea677d048de0073c%2F2024-11-08_080752_379.png?generation=1731024651867644&alt=media)", "3044850": "younesbenalia\n\nCould you clarify if I understood correctly? At time_id == 0, does the lags parameter provide a DataFrame containing different lag values for each responder_id across various time_ids? Or do we just receive a DataFrame with lags specifically for time_id == 0?", "3045123": "at time_id == 0, we get all the time_ids, possiblely from 0 to 970ish, which is lagged by 1 date_id, which means its yesterday's all time_id data" }, "source": "meta" }