I have some questions about how to apply the mapping between date_id and time_id for the responder_{0…8} values from the previous day. I understand that the goal is to take these values from the previous date_id and use them in the next date_id. However, I’m not entirely clear on how this mapping should occur between date_id and the different time_id values.
\nSpecifically, my question is: should the responder_{0…8} values from the last time_id of a date_id be applied only to the first time_id of the new date_id, or should they be applied to all time_id values of the new date_id? Alternatively, should we replicate the values from each time_id of the previous date_id to the corresponding time_id in the new date_id? In other words, should time_id=0 from the previous date_id map to time_id=0 in the new date_id, time_id=1 to time_id=1, and so on?
\nThis is my question in relation to the description of lags.parquet:
\n\"lags.parquet - Values of responder_{0…8} lagged by one date_id. The evaluation API serves the entirety of the lagged responders for a date_id on that date_id’s first time_id. In other words, all of the previous date's responders will be served at the first time step of the succeeding date.\"
\nIs this the correct way to understand how to map the responder_{0…8} values between date_id and time_id?
", "messages": [ { "id": "3044943", "postDate": "11/14/2024 02:25:28", "content": "I have some questions about how to apply the mapping between date_id and time_id for the responder_{0…8} values from the previous day. I understand that the goal is to take these values from the previous date_id and use them in the next date_id. However, I’m not entirely clear on how this mapping should occur between date_id and the different time_id values.
\nSpecifically, my question is: should the responder_{0…8} values from the last time_id of a date_id be applied only to the first time_id of the new date_id, or should they be applied to all time_id values of the new date_id? Alternatively, should we replicate the values from each time_id of the previous date_id to the corresponding time_id in the new date_id? In other words, should time_id=0 from the previous date_id map to time_id=0 in the new date_id, time_id=1 to time_id=1, and so on?
\nThis is my question in relation to the description of lags.parquet:
\n\"lags.parquet - Values of responder_{0…8} lagged by one date_id. The evaluation API serves the entirety of the lagged responders for a date_id on that date_id’s first time_id. In other words, all of the previous date's responders will be served at the first time step of the succeeding date.\"
\nIs this the correct way to understand how to map the responder_{0…8} values between date_id and time_id?
", "rawMarkdown": "I have some questions about how to apply the mapping between date_id and time_id for the responder_{0...8} values from the previous day. I understand that the goal is to take these values from the previous date_id and use them in the next date_id. However, I’m not entirely clear on how this mapping should occur between date_id and the different time_id values.\n\nSpecifically, my question is: should the responder_{0...8} values from the last time_id of a date_id be applied only to the first time_id of the new date_id, or should they be applied to all time_id values of the new date_id? Alternatively, should we replicate the values from each time_id of the previous date_id to the corresponding time_id in the new date_id? In other words, should time_id=0 from the previous date_id map to time_id=0 in the new date_id, time_id=1 to time_id=1, and so on?\n\nThis is my question in relation to the description of lags.parquet:\n\n\"lags.parquet - Values of responder_{0...8} lagged by one date_id. The evaluation API serves the entirety of the lagged responders for a date_id on that date_id’s first time_id. In other words, all of the previous date's responders will be served at the first time step of the succeeding date.\"\n\nIs this the correct way to understand how to map the responder_{0...8} values between date_id and time_id?", "votes": null }, { "id": "3045040", "postDate": "11/14/2024 05:39:24", "content": "@matiastorresr there is no correct/ incorrect way on Kaggle as we don't consider domain specifics here. I suggest you to try out different options and check the CV score - whatever improves your CV score is a winning approach among other candidates in your case.
", "rawMarkdown": "matiastorresr there is no correct/ incorrect way on Kaggle as we don't consider domain specifics here. I suggest you to try out different options and check the CV score - whatever improves your CV score is a winning approach among other candidates in your case.", "votes": null }, { "id": "3045116", "postDate": "11/14/2024 07:24:13", "content": "i had a post talking about lag, take a look
\nhttps://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/545073
Hi John, I read your discussion and thanks for posting! We are a team of 3 and would love to join forces. We have various degree of expertise. There are interesting results on lags we are happy to share. Let me know if interested :). All the best
", "rawMarkdown": "Hi John, I read your discussion and thanks for posting! We are a team of 3 and would love to join forces. We have various degree of expertise. There are interesting results on lags we are happy to share. Let me know if interested :). All the best", "votes": null }, { "id": "3045292", "postDate": "11/14/2024 11:14:04", "content": "\n\nIn other words, should time_id=0 from the previous date_id map to time_id=0 in the new date_id, time_id=1 to time_id=1, and so on?
\n
Not necessarily. You can map any time_id from the previous day to any time_id of today as you like, if you believe it helps your model. You can also map statistical features, specific time_id, or whatever features calculated from the previous day's lags to today's data.
\nBear in mind though. It is not always true that two consecutive days have the same symbol_ids. When the symbol_id in lags_ does not match with the symbol id in test, it may lead to NAN values in some cases when merge/join lags_ to the test using symbol id as the key.
", "rawMarkdown": ">In other words, should time_id=0 from the previous date_id map to time_id=0 in the new date_id, time_id=1 to time_id=1, and so on?\n\nNot necessarily. You can map any time_id from the previous day to any time_id of today as you like, if you believe it helps your model. You can also map statistical features, specific time_id, or whatever features calculated from the previous day's lags to today's data. \n\nBear in mind though. It is not always true that two consecutive days have the same symbol_ids. When the symbol_id in lags_ does not match with the symbol id in test, it may lead to NAN values in some cases when merge/join lags_ to the test using symbol id as the key.", "votes": null }, { "id": "3046349", "postDate": "11/15/2024 11:55:01", "content": "awesome, sure, id love to join
", "rawMarkdown": "awesome, sure, id love to join", "votes": null } ], "comments": [ { "id": 3045040, "author_name": "ravi20076", "author_url": "", "post_date": "11/14/2024 05:39:24", "content": "@matiastorresr there is no correct/ incorrect way on Kaggle as we don't consider domain specifics here. I suggest you to try out different options and check the CV score - whatever improves your CV score is a winning approach among other candidates in your case.
", "votes": null, "replies": [] }, { "id": 3045116, "author_name": "zoutain", "author_url": "", "post_date": "11/14/2024 07:24:13", "content": "i had a post talking about lag, take a look
\nhttps://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/545073
Hi John, I read your discussion and thanks for posting! We are a team of 3 and would love to join forces. We have various degree of expertise. There are interesting results on lags we are happy to share. Let me know if interested :). All the best
", "votes": null, "replies": [ { "id": 3046349, "author_name": "zoutain", "author_url": "", "post_date": "11/15/2024 11:55:01", "content": "awesome, sure, id love to join
", "votes": null, "replies": [] } ] } ] }, { "id": 3045292, "author_name": "shiyili", "author_url": "", "post_date": "11/14/2024 11:14:04", "content": "\n\nIn other words, should time_id=0 from the previous date_id map to time_id=0 in the new date_id, time_id=1 to time_id=1, and so on?
\n
Not necessarily. You can map any time_id from the previous day to any time_id of today as you like, if you believe it helps your model. You can also map statistical features, specific time_id, or whatever features calculated from the previous day's lags to today's data.
\nBear in mind though. It is not always true that two consecutive days have the same symbol_ids. When the symbol_id in lags_ does not match with the symbol id in test, it may lead to NAN values in some cases when merge/join lags_ to the test using symbol id as the key.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3044943": "I have some questions about how to apply the mapping between date_id and time_id for the responder_{0...8} values from the previous day. I understand that the goal is to take these values from the previous date_id and use them in the next date_id. However, I’m not entirely clear on how this mapping should occur between date_id and the different time_id values.\n\nSpecifically, my question is: should the responder_{0...8} values from the last time_id of a date_id be applied only to the first time_id of the new date_id, or should they be applied to all time_id values of the new date_id? Alternatively, should we replicate the values from each time_id of the previous date_id to the corresponding time_id in the new date_id? In other words, should time_id=0 from the previous date_id map to time_id=0 in the new date_id, time_id=1 to time_id=1, and so on?\n\nThis is my question in relation to the description of lags.parquet:\n\n\"lags.parquet - Values of responder_{0...8} lagged by one date_id. The evaluation API serves the entirety of the lagged responders for a date_id on that date_id’s first time_id. In other words, all of the previous date's responders will be served at the first time step of the succeeding date.\"\n\nIs this the correct way to understand how to map the responder_{0...8} values between date_id and time_id?", "3045040": "matiastorresr there is no correct/ incorrect way on Kaggle as we don't consider domain specifics here. I suggest you to try out different options and check the CV score - whatever improves your CV score is a winning approach among other candidates in your case.", "3045116": "i had a post talking about lag, take a look\nhttps://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/545073", "3045227": "Hi John, I read your discussion and thanks for posting! We are a team of 3 and would love to join forces. We have various degree of expertise. There are interesting results on lags we are happy to share. Let me know if interested :). All the best", "3045292": ">In other words, should time_id=0 from the previous date_id map to time_id=0 in the new date_id, time_id=1 to time_id=1, and so on?\n\nNot necessarily. You can map any time_id from the previous day to any time_id of today as you like, if you believe it helps your model. You can also map statistical features, specific time_id, or whatever features calculated from the previous day's lags to today's data. \n\nBear in mind though. It is not always true that two consecutive days have the same symbol_ids. When the symbol_id in lags_ does not match with the symbol id in test, it may lead to NAN values in some cases when merge/join lags_ to the test using symbol id as the key.", "3046349": "awesome, sure, id love to join" }, "source": "meta" }