![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Nov-2022/leak.png\")
In this competition we are predicting test events occurring within a one week period.
\nWhen we predict events occurring in first day of test data, we can use events occurring in the next 6 days of test data which are from the future to make better predictions. Original post asked \"Can we train our models using test data from the future?\" and the host replied below \"yes we can\".
\nUsing test data will boost our CV LB as shown in this notebook here where original notebook had LB 0.539 and adding test data to train achieves LB 0.541.
\nBelow is a diagram of train and test sessions from OTTO's GitHub here. And in the comments below, i post histograms of test session start times and test session end times.
\n
In this competition we are predicting test events occurring within a one week period.
\nWhen we predict events occurring in first day of test data, we can use events occurring in the next 6 days of test data which are from the future to make better predictions. Original post asked \"Can we train our models using test data from the future?\" and the host replied below \"yes we can\".
\nUsing test data will boost our CV LB as shown in this notebook here where original notebook had LB 0.539 and adding test data to train achieves LB 0.541.
\nBelow is a diagram of train and test sessions from OTTO's GitHub here. And in the comments below, i post histograms of test session start times and test session end times.
\n
I don't see how this could be disallowed to be honest 🙂 Tracking this in solutions would be a nightmare, to make sure people didn't train on the test set (essentially, impossible). Plus this is not very different from pseudo-labeling, etc.
\nThat is a great question though @cdeotte! Would be great if there could be an official confirmation that this is okay 🙂
", "rawMarkdown": "I don't see how this could be disallowed to be honest 🙂 Tracking this in solutions would be a nightmare, to make sure people didn't train on the test set (essentially, impossible). Plus this is not very different from pseudo-labeling, etc.\n\nThat is a great question though @cdeotte! Would be great if there could be an official confirmation that this is okay 🙂", "votes": null }, { "id": "2016230", "postDate": "11/03/2022 21:05:54", "content": "BTW I am not sure your phrasing is correct? It doesn't line up with my understanding -- maybe I am getting something wrong here.
\nWe are never predicting the first day of the test set for any of the events. We are always predicting the events that come after the truncated sessions in the test set (always day 7+) AFAIU.
\nWould be great if someone could verify if that reasoning is right -- conceptually the situation is different from what you describe.
", "rawMarkdown": "BTW I am not sure your phrasing is correct? It doesn't line up with my understanding -- maybe I am getting something wrong here.\n\nWe are never predicting the first day of the test set for any of the events. We are always predicting the events that come after the truncated sessions in the test set (always day 7+) AFAIU.\n\nWould be great if someone could verify if that reasoning is right -- conceptually the situation is different from what you describe.", "votes": null }, { "id": "2016265", "postDate": "11/03/2022 22:01:25", "content": "Hey @cdeotte, thanks for your question and congrats on the current 1st place! In practice, we would not be able to train our models on the truncated futures sessions, as you suggest. However, as @radek1 mentioned, it is impractical to prevent this for the scope of this competition, and you are allowed to use all the data we provided.
", "rawMarkdown": "Hey @cdeotte, thanks for your question and congrats on the current 1st place! In practice, we would not be able to train our models on the truncated futures sessions, as you suggest. However, as @radek1 mentioned, it is impractical to prevent this for the scope of this competition, and you are allowed to use all the data we provided.", "votes": null }, { "id": "2016280", "postDate": "11/03/2022 22:31:15", "content": "I see why you consider it impractical, I feel that this question was brought up because of developements in the RecSys Challenge 2022 where a strict rule on usage of test set was added:
\n\n\nWhen predicting, treat each test session independently of all other test sessions (i.e., when predicting for test session B, the model should not have any knowledge of test session A. Even if that came before it in terms of time-stamp)
\n
I had the same concern in a recently closed community competition where I posted a similar thread.
\nThose kind of rules usually get in place so that the code is more realistically close to a real-world recommender system.
But, for sure, enforcing this kind of rules requires a lot of work and in addition to that it is difficult to detect future data leakage (one could just use future data for hyperparameter tuning and no one could know).
\n( In recsys challenge 2022 i think around top 20-30 teams were required to submit \"reproducible code\" for their solutions, someone had a lot of work to perform for sure. Just going through my own solution was a nightmare, I can't imagine how an external reader that validated the code must have felt. )
Luckly we already have confirmation that it is possible to use future data.
\nI just wanted to add some context to the discussion for future readers reference.
", "rawMarkdown": "I see why you consider it impractical, I feel that this question was brought up because of developements in the [RecSys Challenge 2022](http://www.recsyschallenge.com/2022/) where a strict rule on usage of test set was added: \n\n> When predicting, treat each test session independently of all other test sessions (i.e., when predicting for test session B, the model should not have any knowledge of test session A. Even if that came before it in terms of time-stamp)\n\nI had the same concern in a recently closed community competition where I posted a [similar thread](https://www.kaggle.com/competitions/what-card-should-i-select-next/discussion/353776).\nThose kind of rules usually get in place so that the code is more realistically close to a real-world recommender system. \n\nBut, for sure, enforcing this kind of rules requires a lot of work and in addition to that it is difficult to detect future data leakage (one could just use future data for hyperparameter tuning and no one could know).\n( In recsys challenge 2022 i think around top 20-30 teams were required to submit \"reproducible code\" for their solutions, someone had a lot of work to perform for sure. Just going through my own solution was a nightmare, I can't imagine how an external reader that validated the code must have felt. )\n\nLuckly we already have confirmation that it is possible to use future data. \n\nI just wanted to add some context to the discussion for future readers reference.", "votes": null }, { "id": "2016413", "postDate": "11/04/2022 01:37:09", "content": "I added a diagram to my original post to illustrate. And below are histograms of the 1,671,803 test session end times and start times. We see that roughly 1/7 of the sessions end on Tuesday Aug 30th 2022 and then 6/7 of the test sessions start on the following 6 days. The median session length is 30 seconds.
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Nov-2022/date_hist.png\")
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Nov-2022/date_hist2.png\")
I really like your question @cdeotte, I really don't know its exact answer. I have got some answer from comment section. I just wanted you to share the conclusive answer if possible. Thanks for sharing.
", "rawMarkdown": "I really like your question @cdeotte, I really don't know its exact answer. I have got some answer from comment section. I just wanted you to share the conclusive answer if possible. Thanks for sharing.", "votes": null }, { "id": "2017211", "postDate": "11/04/2022 15:53:20", "content": "Os dados para teste precisam ser publicados para que mais Cientistas e Analistas consigam realizar treinamentos e se tornarem profissionais melhores.
", "rawMarkdown": "Os dados para teste precisam ser publicados para que mais Cientistas e Analistas consigam realizar treinamentos e se tornarem profissionais melhores.", "votes": null }, { "id": "2017432", "postDate": "11/04/2022 19:54:49", "content": "Nice visualization, @cdeotte 👌 But I agree with @radek1 that the phrasings regarding the evaluation are somewhat misleading. The task is neither to predict the 7+ day nor the next day of a truncated session. Instead, your task is to predict the next events of the randomly truncated sessions, which can range over the whole test period but never exceed it. I hope this clears up any misconceptions 😌
", "rawMarkdown": "Nice visualization, @cdeotte 👌 But I agree with @radek1 that the phrasings regarding the evaluation are somewhat misleading. The task is neither to predict the 7+ day nor the next day of a truncated session. Instead, your task is to predict the next events of the randomly truncated sessions, which can range over the whole test period but never exceed it. I hope this clears up any misconceptions 😌", "votes": null }, { "id": "2017433", "postDate": "11/04/2022 19:55:18", "content": "The entire dataset, including the untruncated test sessions, will be published once the competition is finalized.
", "rawMarkdown": "The entire dataset, including the untruncated test sessions, will be published once the competition is finalized.", "votes": null }, { "id": "2017465", "postDate": "11/04/2022 20:31:48", "content": "The host Philipp has said
\n\n\nyou are allowed to use all the data we provided.
\n
in another comment. So, yes, we may train with test data.
", "rawMarkdown": "The host Philipp has said \n> you are allowed to use all the data we provided.\n\nin another comment. So, yes, we may train with test data.", "votes": null }, { "id": "2017485", "postDate": "11/04/2022 20:49:20", "content": "Thanks for clarification @pnormann . Good point, we are predicting events (not days). I updated my original post to use the word \"events\". Hopefully this together with your explanation makes evaluation clearer.
", "rawMarkdown": "Thanks for clarification @pnormann . Good point, we are predicting events (not days). I updated my original post to use the word \"events\". Hopefully this together with your explanation makes evaluation clearer.", "votes": null }, { "id": "2072682", "postDate": "12/22/2022 10:46:46", "content": "Hi @pnormann
\nIs there a reason why you haven't made available the historical data of the test sessions for dates covering the train set? Conscious that in a real use-case, we would have access to that data for inference, so what was your motivation for truncating the test sessions like this?
Hi @cdeotte
\nWhy would you call your finding a data \"leakage\"? When training a model on a train set, and you want to make a prediction at a random datetime (still within train set's date range), there are going to be train sessions whose last actions took place X days before this reference datetime, and therefore you can create features that look at what other items were clicked/carted/ordered after the session's last action and before the reference datetime. This way you can incorporate information from the \"future\" of a session, but without cheating - it's not leakage. For example, if I made my last action as a customer 2 days ago, and I went on your website today, your product recommender could also look at what other items have been popular since 2 days ago.
\nUnless I have misunderstood on how you're using the Test data.
\n\nIs there a reason why you haven't made available the historical data of the test sessions for dates covering the train set? Conscious that in a real use-case, we would have access to that data for inference, so what was your motivation for truncating the test sessions like this?
\n
There is no historical data for test users. The users in test data are the users from the week 5 who have no history data. They are not truncated. The train users are truncated. Any user with history data before week 5 is added to train data and their activity is truncated to weeks 1 thru 4.
\n\n\nWhy would you call your finding a data \"leakage\"?
\n
Imagine today is Wednesday and I need to make a recommendation for you. It is true that i can view your activity from Monday and Tuesday. In this competition, i can also view the future of Thursday and Friday (which has not occurred yet) and make you a recommendation of the most popular item from the future dates of Thursday and Friday.
", "rawMarkdown": "ikogias \n>Is there a reason why you haven't made available the historical data of the test sessions for dates covering the train set? Conscious that in a real use-case, we would have access to that data for inference, so what was your motivation for truncating the test sessions like this?\n\nThere is no historical data for test users. The users in test data are the users from the week 5 who have no history data. They are not truncated. The train users are truncated. Any user with history data before week 5 is added to train data and their activity is truncated to weeks 1 thru 4.\n\n>Why would you call your finding a data \"leakage\"?\n\nImagine today is Wednesday and I need to make a recommendation for you. It is true that i can view your activity from Monday and Tuesday. In this competition, i can also view the future of Thursday and Friday (which has not occurred yet) and make you a recommendation of the most popular item from the future dates of Thursday and Friday.", "votes": null }, { "id": "2072936", "postDate": "12/22/2022 14:29:32", "content": "\n\n\nThere is no historical data for test users.
\n
Didn't realise that, thanks.
\n\n\nImagine today is Wednesday and I need to make a recommendation for you. It is true that i can view your activity from Monday and Tuesday. In this competition, i can also view the future of Thursday and Friday (which has not occurred yet) and make you a recommendation of the most popular item from the future dates of Thursday and Friday.
\n
I agree with your interpretation given that OTTO has truncated the sessions. What I had in mind was that test sessions were intact and the \"time of prediction\" was the very end of the test set. Had they done it like this, there wouldn't be any leakage..
", "rawMarkdown": "cdeotte \n\n>There is no historical data for test users. \n\nDidn't realise that, thanks.\n\n>Imagine today is Wednesday and I need to make a recommendation for you. It is true that i can view your activity from Monday and Tuesday. In this competition, i can also view the future of Thursday and Friday (which has not occurred yet) and make you a recommendation of the most popular item from the future dates of Thursday and Friday.\n\nI agree with your interpretation given that OTTO has truncated the sessions. What I had in mind was that test sessions were intact and the \"time of prediction\" was the very end of the test set. Had they done it like this, there wouldn't be any leakage..", "votes": null }, { "id": "2072955", "postDate": "12/22/2022 14:43:51", "content": "@ikogias The leakage is about using test User B to predict test User A. I agree with you if we only use information from User A to predict User A then there is no leakage. But since Kaggle gave us all the test data, we have info from 1.8 million test users. If I use info from the 1,799,999 test users other than the user i am predicting then it is leakage because in real life when we are predicting User A we do not have future information from User B, User C (whom have activity in the future), etc
Note we are not predicting events that occur in week 6 which take place after all test user sessions. We are predicting events taking place during week 5. So for User A we have what they did Monday and Tuesday and we need to predict Wednesday, Thursday, Friday.
\nAt the same time, we have User B with events from Monday, Tuesday, Wednesday, Thursday. And we need to predict what User B does on Friday.
\nThe leakage is using User B events from Wednesday and Thursday to predict User A events on Wednesday, Thursday, Friday.
", "rawMarkdown": "Note we are not predicting events that occur in week 6 which take place after all test user sessions. We are predicting events taking place during week 5. So for User A we have what they did Monday and Tuesday and we need to predict Wednesday, Thursday, Friday.\n\nAt the same time, we have User B with events from Monday, Tuesday, Wednesday, Thursday. And we need to predict what User B does on Friday.\n\nThe leakage is using User B events from Wednesday and Thursday to predict User A events on Wednesday, Thursday, Friday.", "votes": null }, { "id": "2072982", "postDate": "12/22/2022 15:01:49", "content": "@cdeotte Agreed. All I was saying was, had OTTO asked us to predict events that occur in week 6 (with week 5 test sessions intact) then there wouldn't be any possibility of data leakage.
", "rawMarkdown": "cdeotte Agreed. All I was saying was, had OTTO asked us to predict events that occur in week 6 (with week 5 test sessions intact) then there wouldn't be any possibility of data leakage.", "votes": null } ], "comments": [ { "id": 2016227, "author_name": "radek1", "author_url": "", "post_date": "11/03/2022 21:01:09", "content": "I don't see how this could be disallowed to be honest 🙂 Tracking this in solutions would be a nightmare, to make sure people didn't train on the test set (essentially, impossible). Plus this is not very different from pseudo-labeling, etc.
\nThat is a great question though @cdeotte! Would be great if there could be an official confirmation that this is okay 🙂
", "votes": null, "replies": [ { "id": 2016280, "author_name": "pietromaldini1", "author_url": "", "post_date": "11/03/2022 22:31:15", "content": "I see why you consider it impractical, I feel that this question was brought up because of developements in the RecSys Challenge 2022 where a strict rule on usage of test set was added:
\n\n\nWhen predicting, treat each test session independently of all other test sessions (i.e., when predicting for test session B, the model should not have any knowledge of test session A. Even if that came before it in terms of time-stamp)
\n
I had the same concern in a recently closed community competition where I posted a similar thread.
\nThose kind of rules usually get in place so that the code is more realistically close to a real-world recommender system.
But, for sure, enforcing this kind of rules requires a lot of work and in addition to that it is difficult to detect future data leakage (one could just use future data for hyperparameter tuning and no one could know).
\n( In recsys challenge 2022 i think around top 20-30 teams were required to submit \"reproducible code\" for their solutions, someone had a lot of work to perform for sure. Just going through my own solution was a nightmare, I can't imagine how an external reader that validated the code must have felt. )
Luckly we already have confirmation that it is possible to use future data.
\nI just wanted to add some context to the discussion for future readers reference.
", "votes": null, "replies": [] } ] }, { "id": 2016230, "author_name": "radek1", "author_url": "", "post_date": "11/03/2022 21:05:54", "content": "BTW I am not sure your phrasing is correct? It doesn't line up with my understanding -- maybe I am getting something wrong here.
\nWe are never predicting the first day of the test set for any of the events. We are always predicting the events that come after the truncated sessions in the test set (always day 7+) AFAIU.
\nWould be great if someone could verify if that reasoning is right -- conceptually the situation is different from what you describe.
", "votes": null, "replies": [ { "id": 2016413, "author_name": "cdeotte", "author_url": "", "post_date": "11/04/2022 01:37:09", "content": "I added a diagram to my original post to illustrate. And below are histograms of the 1,671,803 test session end times and start times. We see that roughly 1/7 of the sessions end on Tuesday Aug 30th 2022 and then 6/7 of the test sessions start on the following 6 days. The median session length is 30 seconds.
\n
\n
Nice visualization, @cdeotte 👌 But I agree with @radek1 that the phrasings regarding the evaluation are somewhat misleading. The task is neither to predict the 7+ day nor the next day of a truncated session. Instead, your task is to predict the next events of the randomly truncated sessions, which can range over the whole test period but never exceed it. I hope this clears up any misconceptions 😌
", "votes": null, "replies": [] }, { "id": 2017485, "author_name": "cdeotte", "author_url": "", "post_date": "11/04/2022 20:49:20", "content": "Thanks for clarification @pnormann . Good point, we are predicting events (not days). I updated my original post to use the word \"events\". Hopefully this together with your explanation makes evaluation clearer.
", "votes": null, "replies": [] } ] }, { "id": 2016265, "author_name": "pnormann", "author_url": "", "post_date": "11/03/2022 22:01:25", "content": "Hey @cdeotte, thanks for your question and congrats on the current 1st place! In practice, we would not be able to train our models on the truncated futures sessions, as you suggest. However, as @radek1 mentioned, it is impractical to prevent this for the scope of this competition, and you are allowed to use all the data we provided.
", "votes": null, "replies": [] }, { "id": 2016771, "author_name": "rahul029", "author_url": "", "post_date": "11/04/2022 08:51:30", "content": "I really like your question @cdeotte, I really don't know its exact answer. I have got some answer from comment section. I just wanted you to share the conclusive answer if possible. Thanks for sharing.
", "votes": null, "replies": [ { "id": 2017465, "author_name": "cdeotte", "author_url": "", "post_date": "11/04/2022 20:31:48", "content": "The host Philipp has said
\n\n\nyou are allowed to use all the data we provided.
\n
in another comment. So, yes, we may train with test data.
", "votes": null, "replies": [] } ] }, { "id": 2017211, "author_name": "leandrocarrinho", "author_url": "", "post_date": "11/04/2022 15:53:20", "content": "Os dados para teste precisam ser publicados para que mais Cientistas e Analistas consigam realizar treinamentos e se tornarem profissionais melhores.
", "votes": null, "replies": [ { "id": 2017433, "author_name": "pnormann", "author_url": "", "post_date": "11/04/2022 19:55:18", "content": "The entire dataset, including the untruncated test sessions, will be published once the competition is finalized.
", "votes": null, "replies": [] } ] }, { "id": 2072682, "author_name": "ikogias", "author_url": "", "post_date": "12/22/2022 10:46:46", "content": "Hi @pnormann
\nIs there a reason why you haven't made available the historical data of the test sessions for dates covering the train set? Conscious that in a real use-case, we would have access to that data for inference, so what was your motivation for truncating the test sessions like this?
Hi @cdeotte
\nWhy would you call your finding a data \"leakage\"? When training a model on a train set, and you want to make a prediction at a random datetime (still within train set's date range), there are going to be train sessions whose last actions took place X days before this reference datetime, and therefore you can create features that look at what other items were clicked/carted/ordered after the session's last action and before the reference datetime. This way you can incorporate information from the \"future\" of a session, but without cheating - it's not leakage. For example, if I made my last action as a customer 2 days ago, and I went on your website today, your product recommender could also look at what other items have been popular since 2 days ago.
\nUnless I have misunderstood on how you're using the Test data.
\n\nIs there a reason why you haven't made available the historical data of the test sessions for dates covering the train set? Conscious that in a real use-case, we would have access to that data for inference, so what was your motivation for truncating the test sessions like this?
\n
There is no historical data for test users. The users in test data are the users from the week 5 who have no history data. They are not truncated. The train users are truncated. Any user with history data before week 5 is added to train data and their activity is truncated to weeks 1 thru 4.
\n\n\nWhy would you call your finding a data \"leakage\"?
\n
Imagine today is Wednesday and I need to make a recommendation for you. It is true that i can view your activity from Monday and Tuesday. In this competition, i can also view the future of Thursday and Friday (which has not occurred yet) and make you a recommendation of the most popular item from the future dates of Thursday and Friday.
", "votes": null, "replies": [ { "id": 2072936, "author_name": "ikogias", "author_url": "", "post_date": "12/22/2022 14:29:32", "content": "\n\n\nThere is no historical data for test users.
\n
Didn't realise that, thanks.
\n\n\nImagine today is Wednesday and I need to make a recommendation for you. It is true that i can view your activity from Monday and Tuesday. In this competition, i can also view the future of Thursday and Friday (which has not occurred yet) and make you a recommendation of the most popular item from the future dates of Thursday and Friday.
\n
I agree with your interpretation given that OTTO has truncated the sessions. What I had in mind was that test sessions were intact and the \"time of prediction\" was the very end of the test set. Had they done it like this, there wouldn't be any leakage..
", "votes": null, "replies": [ { "id": 2072955, "author_name": "cdeotte", "author_url": "", "post_date": "12/22/2022 14:43:51", "content": "@ikogias The leakage is about using test User B to predict test User A. I agree with you if we only use information from User A to predict User A then there is no leakage. But since Kaggle gave us all the test data, we have info from 1.8 million test users. If I use info from the 1,799,999 test users other than the user i am predicting then it is leakage because in real life when we are predicting User A we do not have future information from User B, User C (whom have activity in the future), etc
Note we are not predicting events that occur in week 6 which take place after all test user sessions. We are predicting events taking place during week 5. So for User A we have what they did Monday and Tuesday and we need to predict Wednesday, Thursday, Friday.
\nAt the same time, we have User B with events from Monday, Tuesday, Wednesday, Thursday. And we need to predict what User B does on Friday.
\nThe leakage is using User B events from Wednesday and Thursday to predict User A events on Wednesday, Thursday, Friday.
", "votes": null, "replies": [ { "id": 2072982, "author_name": "ikogias", "author_url": "", "post_date": "12/22/2022 15:01:49", "content": "@cdeotte Agreed. All I was saying was, had OTTO asked us to predict events that occur in week 6 (with week 5 test sessions intact) then there wouldn't be any possibility of data leakage.
", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2016122": "In this competition we are predicting test events occurring within a one week period. \n\nWhen we predict events occurring in first day of test data, we can use events occurring in the next 6 days of test data which are from the future to make better predictions. Original post asked \"Can we train our models using test data from the future?\" and the host replied below \"yes we can\".\n\nUsing test data will boost our CV LB as shown in this notebook [here][1] where original notebook had LB 0.539 and adding test data to train achieves LB 0.541. \n\nBelow is a diagram of train and test sessions from OTTO's GitHub [here][2]. And in the comments below, i post histograms of test session start times and test session end times.\n\n\n\n[1]: https://www.kaggle.com/code/cdeotte/test-data-leak-lb-boost\n[2]: https://github.com/otto-de/recsys-dataset", "2016227": "I don't see how this could be disallowed to be honest 🙂 Tracking this in solutions would be a nightmare, to make sure people didn't train on the test set (essentially, impossible). Plus this is not very different from pseudo-labeling, etc.\n\nThat is a great question though @cdeotte! Would be great if there could be an official confirmation that this is okay 🙂", "2016230": "BTW I am not sure your phrasing is correct? It doesn't line up with my understanding -- maybe I am getting something wrong here.\n\nWe are never predicting the first day of the test set for any of the events. We are always predicting the events that come after the truncated sessions in the test set (always day 7+) AFAIU.\n\nWould be great if someone could verify if that reasoning is right -- conceptually the situation is different from what you describe.", "2016265": "Hey @cdeotte, thanks for your question and congrats on the current 1st place! In practice, we would not be able to train our models on the truncated futures sessions, as you suggest. However, as @radek1 mentioned, it is impractical to prevent this for the scope of this competition, and you are allowed to use all the data we provided.", "2016280": "I see why you consider it impractical, I feel that this question was brought up because of developements in the [RecSys Challenge 2022](http://www.recsyschallenge.com/2022/) where a strict rule on usage of test set was added: \n\n> When predicting, treat each test session independently of all other test sessions (i.e., when predicting for test session B, the model should not have any knowledge of test session A. Even if that came before it in terms of time-stamp)\n\nI had the same concern in a recently closed community competition where I posted a [similar thread](https://www.kaggle.com/competitions/what-card-should-i-select-next/discussion/353776).\nThose kind of rules usually get in place so that the code is more realistically close to a real-world recommender system. \n\nBut, for sure, enforcing this kind of rules requires a lot of work and in addition to that it is difficult to detect future data leakage (one could just use future data for hyperparameter tuning and no one could know).\n( In recsys challenge 2022 i think around top 20-30 teams were required to submit \"reproducible code\" for their solutions, someone had a lot of work to perform for sure. Just going through my own solution was a nightmare, I can't imagine how an external reader that validated the code must have felt. )\n\nLuckly we already have confirmation that it is possible to use future data. \n\nI just wanted to add some context to the discussion for future readers reference.", "2016413": "I added a diagram to my original post to illustrate. And below are histograms of the 1,671,803 test session end times and start times. We see that roughly 1/7 of the sessions end on Tuesday Aug 30th 2022 and then 6/7 of the test sessions start on the following 6 days. The median session length is 30 seconds.\n\n", "2016771": "I really like your question @cdeotte, I really don't know its exact answer. I have got some answer from comment section. I just wanted you to share the conclusive answer if possible. Thanks for sharing.", "2017211": "Os dados para teste precisam ser publicados para que mais Cientistas e Analistas consigam realizar treinamentos e se tornarem profissionais melhores.", "2017432": "Nice visualization, @cdeotte 👌 But I agree with @radek1 that the phrasings regarding the evaluation are somewhat misleading. The task is neither to predict the 7+ day nor the next day of a truncated session. Instead, your task is to predict the next events of the randomly truncated sessions, which can range over the whole test period but never exceed it. I hope this clears up any misconceptions 😌", "2017433": "The entire dataset, including the untruncated test sessions, will be published once the competition is finalized.", "2017465": "The host Philipp has said \n> you are allowed to use all the data we provided.\n\nin another comment. So, yes, we may train with test data.", "2017485": "Thanks for clarification @pnormann . Good point, we are predicting events (not days). I updated my original post to use the word \"events\". Hopefully this together with your explanation makes evaluation clearer.", "2072682": "Hi @pnormann \nIs there a reason why you haven't made available the historical data of the test sessions for dates covering the train set? Conscious that in a real use-case, we would have access to that data for inference, so what was your motivation for truncating the test sessions like this?\n\nHi @cdeotte\nWhy would you call your finding a data \"leakage\"? When training a model on a train set, and you want to make a prediction at a random datetime (still within train set's date range), there are going to be train sessions whose last actions took place X days before this reference datetime, and therefore you can create features that look at what other items were clicked/carted/ordered after the session's last action and before the reference datetime. This way you can incorporate information from the \"future\" of a session, but without cheating - it's not leakage. For example, if I made my last action as a customer 2 days ago, and I went on your website today, your product recommender could also look at what other items have been popular since 2 days ago.\nUnless I have misunderstood on how you're using the Test data.", "2072890": "ikogias \n>Is there a reason why you haven't made available the historical data of the test sessions for dates covering the train set? Conscious that in a real use-case, we would have access to that data for inference, so what was your motivation for truncating the test sessions like this?\n\nThere is no historical data for test users. The users in test data are the users from the week 5 who have no history data. They are not truncated. The train users are truncated. Any user with history data before week 5 is added to train data and their activity is truncated to weeks 1 thru 4.\n\n>Why would you call your finding a data \"leakage\"?\n\nImagine today is Wednesday and I need to make a recommendation for you. It is true that i can view your activity from Monday and Tuesday. In this competition, i can also view the future of Thursday and Friday (which has not occurred yet) and make you a recommendation of the most popular item from the future dates of Thursday and Friday.", "2072936": "cdeotte \n\n>There is no historical data for test users. \n\nDidn't realise that, thanks.\n\n>Imagine today is Wednesday and I need to make a recommendation for you. It is true that i can view your activity from Monday and Tuesday. In this competition, i can also view the future of Thursday and Friday (which has not occurred yet) and make you a recommendation of the most popular item from the future dates of Thursday and Friday.\n\nI agree with your interpretation given that OTTO has truncated the sessions. What I had in mind was that test sessions were intact and the \"time of prediction\" was the very end of the test set. Had they done it like this, there wouldn't be any leakage..", "2072955": "ikogias The leakage is about using `test User B` to predict `test User A`. I agree with you if we only use information from `User A` to predict `User A` then there is no leakage. But since Kaggle gave us all the test data, we have info from 1.8 million test users. If I use info from the 1,799,999 test users other than the user i am predicting then it is leakage because in real life when we are predicting `User A` we do not have future information from `User B`, `User C` (whom have activity in the future), etc", "2072959": "Note we are not predicting events that occur in week 6 which take place after all test user sessions. We are predicting events taking place during week 5. So for User A we have what they did Monday and Tuesday and we need to predict Wednesday, Thursday, Friday.\n\nAt the same time, we have User B with events from Monday, Tuesday, Wednesday, Thursday. And we need to predict what User B does on Friday.\n\nThe leakage is using User B events from Wednesday and Thursday to predict User A events on Wednesday, Thursday, Friday.", "2072982": "cdeotte Agreed. All I was saying was, had OTTO asked us to predict events that occur in week 6 (with week 5 test sessions intact) then there wouldn't be any possibility of data leakage." }, "source": "meta" }