![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2F457ff5950918744c841a83ac4c0ece1a%2Fdownload.png?generation=1667989795036578&alt=media\")
Hey!
\nI ran a couple of experiments and local validation tracks public LB perfectly! Here is what the situation looks like:
\n
The results are slightly poorer in local validation, but that is not surprising -- we are running on one week less of train data!
\nHere's the setup:
\nThere has been a lot of conversation on the forums on the metric, I think we now have a good understanding of how it works π (one can also always confirm one's understanding by checking out the organizer's repo!). Interesting discussions on the forums here and here.
\nHere is the code I am using for calculating the metric in pandas, starting from a valid submission DataFrame, in case this might be of help:
\n submission['session'] = submission.session_type.apply(lambda x: int(x.split('_')[0]))\n submission['type'] = submission.session_type.apply(lambda x: x.split('_')[1])\n submission.labels = submission.labels.apply(lambda x: [int(i) for i in x.split(' ')[:20]])\n\n test_labels = pd.read_parquet('out_processed/test_labels.parquet')\n test_labels = test_labels.merge(submission, how='left', on=['session', 'type'])\n test_labels['hits'] = test_labels.apply(lambda df: len(set(df.ground_truth).intersection(set(df.labels))), axis=1)\n test_labels['gt_count'] = test_labels.ground_truth.str.len().clip(0,20)\n\n recall_per_type = test_labels.groupby(['type'])['hits'].sum() / test_labels.groupby(['type'])['gt_count'].sum() \n\n score = (recall_per_type * pd.Series({'clicks': 0.10, 'carts': 0.30, 'orders': 0.60})).sum()\nCompetitions where local validation tracks public LB are much more fun, IMHO π
\nWishing you all the best in the challenge!
\nRadek
IMPORTANT CAVEAT: Please note that in my dataset, in order to minimize its size without losing information (unless you care about milliseconds, but I assume seconds should be enough π), I divide the ts column by 1000. Notebooks on Kaggle by other people use unmodified data, so if you would want to run this validation with that code, you need to either make changes to the code or multiply the ts column by 1000.
A couple of related resources you might find useful:
\nHey!
\nI ran a couple of experiments and local validation tracks public LB perfectly! Here is what the situation looks like:
\n
The results are slightly poorer in local validation, but that is not surprising -- we are running on one week less of train data!
\nHere's the setup:
\nThere has been a lot of conversation on the forums on the metric, I think we now have a good understanding of how it works π (one can also always confirm one's understanding by checking out the organizer's repo!). Interesting discussions on the forums here and here.
\nHere is the code I am using for calculating the metric in pandas, starting from a valid submission DataFrame, in case this might be of help:
\n submission['session'] = submission.session_type.apply(lambda x: int(x.split('_')[0]))\n submission['type'] = submission.session_type.apply(lambda x: x.split('_')[1])\n submission.labels = submission.labels.apply(lambda x: [int(i) for i in x.split(' ')[:20]])\n\n test_labels = pd.read_parquet('out_processed/test_labels.parquet')\n test_labels = test_labels.merge(submission, how='left', on=['session', 'type'])\n test_labels['hits'] = test_labels.apply(lambda df: len(set(df.ground_truth).intersection(set(df.labels))), axis=1)\n test_labels['gt_count'] = test_labels.ground_truth.str.len().clip(0,20)\n\n recall_per_type = test_labels.groupby(['type'])['hits'].sum() / test_labels.groupby(['type'])['gt_count'].sum() \n\n score = (recall_per_type * pd.Series({'clicks': 0.10, 'carts': 0.30, 'orders': 0.60})).sum()\nCompetitions where local validation tracks public LB are much more fun, IMHO π
\nWishing you all the best in the challenge!
\nRadek
IMPORTANT CAVEAT: Please note that in my dataset, in order to minimize its size without losing information (unless you care about milliseconds, but I assume seconds should be enough π), I divide the ts column by 1000. Notebooks on Kaggle by other people use unmodified data, so if you would want to run this validation with that code, you need to either make changes to the code or multiply the ts column by 1000.
A couple of related resources you might find useful:
\nFYI, I published a version of Radek's validation data as multiple parquets and restored columns ts and type back to their original. Radek's dataset is more memory and disk efficient, but I published this alternative dataset for anyone who has a pipeline using a Kaggle dataset with multiple parquets and original ts and type columns.
Now you can run your exact pipeline with validation data and compute CV score without making any changes to your pipeline. The Kaggle dataset is here. I plan to publish a notebook which computes validation score for my notebook soon.
", "rawMarkdown": "FYI, I published a version of Radek's validation data as multiple parquets and restored columns `ts` and `type` back to their original. Radek's dataset is more memory and disk efficient, but I published this alternative dataset for anyone who has a pipeline using a Kaggle dataset with multiple parquets and original `ts` and `type` columns. \n\nNow you can run your exact pipeline with validation data and compute CV score without making any changes to your pipeline. The Kaggle dataset is [here][1]. I plan to publish a notebook which computes validation score for my notebook soon.\n\n[1]: https://www.kaggle.com/datasets/cdeotte/otto-validation", "votes": 9, "replies": [ { "id": 2024873, "postDate": "2022-11-10T20:03:44.977Z", "content": "UPDATE: I published a train notebook that achieves LB 0.573 and corresponding validation notebook \"CV\" 0.563 using Radek's validation data (in my Kaggle dataset).
", "rawMarkdown": "UPDATE: I published a train notebook that achieves LB 0.573 and corresponding validation notebook \"CV\" 0.563 using Radek's validation data (in my Kaggle dataset).", "votes": 5 }, { "id": 2025144, "postDate": "2022-11-11T02:37:13.650Z", "content": "After publishing the notebook, i made two changes that boosted CV +0.003 and after submission, the LB boosted +0.003. The CV LB correlate very well.
Your CV scheme (data and metric code) has perfect correlation for me too. Thank you for publishing your setup. The first two data points are my public notebook version 4 and 9 here (with LB 0.556 and LB 0.562 respectively). Then next data point is Pietro's notebook here (when it was/is LB 0.565 version 1). The next data point is Ingvaras' notebook here (when it was/is LB 0.570 version 3). We see that each notebook improves both CV and LB. The last data point is my current LB score using some unpublished tricks.
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Nov-2022/cv_lb2.png\")
wonderful data, thank you for sharing this @cdeotte! π
", "rawMarkdown": "wonderful data, thank you for sharing this @cdeotte! π ", "votes": 2 }, { "id": 2075047, "postDate": "2022-12-25T01:16:59.220Z", "content": "Hi,Chris.I used your version 9 notebook to calculate the local CV. I suspect something went wrong with my idea of calculating local CV scores. In the notebook, I generated candidates for val A's validation set and then calculated the local CV score, I only made the following modifications, but the CV score looks so outrageous. What went wrong with me? I am looking forward to hearing from you.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11356689%2Ffc4f117d4f91aae5da0b8f807e71ad6a%2F1.png?generation=1671930944213119&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11356689%2Fa1e7af94b592aba55f2e8df7ad1383b5%2F2.png?generation=1671930967340151&alt=media\")
Thanks for sharing, this saves a lot of tine!
", "rawMarkdown": "Thanks for sharing, this saves a lot of tine!", "votes": 5, "replies": [ { "id": 2054296, "postDate": "2022-12-04T02:24:31.563Z", "content": "Thank you, JFP π Really glad to be of help
", "rawMarkdown": "Thank you, JFP π Really glad to be of help", "votes": 1 } ] }, { "id": 2023511, "postDate": "2022-11-09T19:43:42.153Z", "content": "This is very helpful thank you. I used your files and code to compute the CV score for my notebook version 4 with LB 0.556. The computed CV score is 0.474 so I think there is a bug in my code somewhere. It appears that you have a smaller gap (between LB and CV) when using your CV files. I will review my code and try againβ¦
", "rawMarkdown": "This is very helpful thank you. I used your files and code to compute the CV score for my notebook version 4 with LB 0.556. The computed CV score is 0.474 so I think there is a bug in my code somewhere. It appears that you have a smaller gap (between LB and CV) when using your CV files. I will review my code and try again...", "votes": 3, "replies": [ { "id": 2023514, "postDate": "2022-11-09T19:51:17.187Z", "content": "I may have found the problem. Your parquet files have ts column with original_ts / 1000. So my notebook's pipeline which adds weight based on ts doesn't work correctly. I will update and check CV score.
You should add a note to your discussion here. Anyone who uses code based on original ts like my Chris' notebook, or Vladimir's notebook, or Pietro's notebook, or Ingvaras's notebook will run into this problem. For example the following line of code:
df.query('abs(ts_x - ts_y) < 24 * 60 * 60 * 1000 and aid_x != aid_y')\nwill waste many hours running code with your pipeline before discovering the error. Also type conversion will fail
\n.astype({\"ts\": \"datetime64[ms]\"})\nYes Chris, that is a great point! Apologies for causing all this trouble on your end. You are absolutely right.
\nI included this bit of information in the other dataset that I published, in the description, but didn't do so here. Very sorry bout that. Will correct this in a second and add a note.
\nMy apologies for all the trouble.
", "rawMarkdown": "Yes Chris, that is a great point! Apologies for causing all this trouble on your end. You are absolutely right.\n\nI included this bit of information in the other dataset that I published, in the description, but didn't do so here. Very sorry bout that. Will correct this in a second and add a note.\n\nMy apologies for all the trouble.", "votes": 1 }, { "id": 2023578, "postDate": "2022-11-09T21:35:50.963Z", "content": "Great. Looks good now. My computed CV is 0.548 for version 4 of my notebook with LB 0.556. That's pretty close. I will compute more CV scores for different versions and check the correlation. I will post results here.
\nThen i will begin boosting my CV LB with improved logic! and/or trained models!
", "rawMarkdown": "Great. Looks good now. My computed CV is 0.548 for version 4 of my notebook with LB 0.556. That's pretty close. I will compute more CV scores for different versions and check the correlation. I will post results here.\n\nThen i will begin boosting my CV LB with improved logic! and/or trained models!", "votes": 3 }, { "id": 2023731, "postDate": "2022-11-10T01:43:16.273Z", "content": "It's really cool how you are sharing openly how you plan to approach this competition, I think that is great learning material! π
\nMy plan is very similar, hoping to share some stuff on Matrix Factorization maybe with using some of the Merlin stuff, have some really cool ideas, but the only problem is finding the time π
\nGreat there are that many weeks still left in this competition! π
", "rawMarkdown": "It's really cool how you are sharing openly how you plan to approach this competition, I think that is great learning material! π\n\nMy plan is very similar, hoping to share some stuff on Matrix Factorization maybe with using some of the Merlin stuff, have some really cool ideas, but the only problem is finding the time π\n\nGreat there are that many weeks still left in this competition! π", "votes": 2 } ] }, { "id": 2029433, "postDate": "2022-11-14T16:31:57.847Z", "content": "Thanks for the dataset. I've spotted a problem with validation dataset. There are \"new\" AIDs (18785) in the validation test set comparing to validations train set). For the original train and test set this is not the case.
\nI've added notebook with the calculation to the dataset: https://www.kaggle.com/code/piotrekga/new-aids-in-test-set
\nI guess there was more logic in train/test split. @pnormann, could you please give us information on what to do with sessions and ground truth with new AIDs?
", "rawMarkdown": "Thanks for the dataset. I've spotted a problem with validation dataset. There are \"new\" AIDs (18785) in the validation test set comparing to validations train set). For the original train and test set this is not the case.\n\nI've added notebook with the calculation to the dataset: https://www.kaggle.com/code/piotrekga/new-aids-in-test-set\n\nI guess there was more logic in train/test split. @pnormann, could you please give us information on what to do with sessions and ground truth with new AIDs?", "votes": 4, "replies": [ { "id": 2029810, "postDate": "2022-11-15T00:27:10.533Z", "content": "ha, interesting observation @piotrekga!
\nI think this might just be coincidental, new AIDs might have gotten added to the system that week π€ Or, alternatively, there was some additional logic in pulling down the data.
\nHere is the logic used by the organizer to create the test set. There is nothing there about filtering sessions based on aids AFAICT at quick glance.
That's a helpful discovery Piotr. Differences between CV and LB are a concern.
\nHowever, I will comment that Radek's validation data is perfectly correlating with LB. Every 0.0002+ improvement on Radek's CV gives me a boost on LB!
@radek1 I've rerun the train-test split on train.jsonl before and got exactly the same results. You're right the logic is not included in the code provided by organisers.
About new AIDs: taking into consideration that these are users behaviours and the dataset has long tails I find it very likely that there were no new AIDs in the original test set and thousands of them in new train-test split. Of course I may be wrong π . I really hope organisers will provide us with some additional clarification in that regards.
\n@cdeotte, you're right that now the CV-LB correlation works. I'd really like to validate my model locally as close as possible to the final validation though.
", "rawMarkdown": "@radek1 I've rerun the train-test split on `train.jsonl` before and got exactly the same results. You're right the logic is not included in the code provided by organisers.\n\nAbout new AIDs: taking into consideration that these are users behaviours and the dataset has long tails I find it very likely that there were no new AIDs in the original test set and thousands of them in new train-test split. Of course I may be wrong π . I really hope organisers will provide us with some additional clarification in that regards.\n\n@cdeotte, you're right that now the CV-LB correlation works. I'd really like to validate my model locally as close as possible to the final validation though.", "votes": 2 }, { "id": 2032183, "postDate": "2022-11-16T13:19:34.117Z", "content": "Hey @piotrekga, thanks for pointing this out and sorry for the confusion! This behavior was a bug in our public repo that was inconsistent with how we created the dataset for the competition. We created the actual dataset using a Clojure job, and during the reimplementation in Python, we, unfortunately, forgot to add this preprocessing step. I just pushed an update to our GitHub repo which fixes this issue. Although I'm still not entirely happy with the performance of the Python script, it should be enough for you to create your local validation set, which reflects the logic we used for the LB test set. I'll see what I can do to speed things up π
", "rawMarkdown": "Hey @piotrekga, thanks for pointing this out and sorry for the confusion! This behavior was a bug in our public repo that was inconsistent with how we created the dataset for the competition. We created the actual dataset using a Clojure job, and during the reimplementation in Python, we, unfortunately, forgot to add this preprocessing step. I just pushed an [update](https://github.com/otto-de/recsys-dataset/commit/14eea3938cd9b6200bda9985f2c7ff8bb1de1b18) to our GitHub repo which fixes this issue. Although I'm still not entirely happy with the performance of the Python script, it should be enough for you to create your local validation set, which reflects the logic we used for the LB test set. I'll see what I can do to speed things up π", "votes": 8 }, { "id": 2034542, "postDate": "2022-11-18T08:18:33.893Z", "content": "Thank you @pnormann for the update. At this point logic is far more important than performance.
", "rawMarkdown": "Thank you @pnormann for the update. At this point logic is far more important than performance.", "votes": 2 } ] }, { "id": 2093576, "postDate": "2023-01-10T05:35:13.883Z", "content": "Sorry @radek1 what is the id2type and vice versa pkl files contain? What is the purpose of these 2 data files?
", "rawMarkdown": "Sorry @radek1 what is the id2type and vice versa pkl files contain? What is the purpose of these 2 data files?", "votes": 1, "replies": [ { "id": 2094694, "postDate": "2023-01-11T00:46:19.597Z", "content": "hey @ppujari! They just contain the mapping of types to their ids IIRC, 0 maps to clicks, 1 to carts and 2 to orders. Probably shouldn't have included it, makes it all a bit confusing.
Hi @radek1, I have found something weird of the validation dataset you provided.
\nAccording to the organizer's script for creating the validation dataset, when we split the 4 weeks train into 3-weeks-train (part-1) and 1-week-train (part-2), the part-2's aids should have all been seen in part-1. However, the test.parquet in the part-2 still have more than 2% aids which are not seen by part-1.
I have shown this problem in this version of the notebook here
", "rawMarkdown": "Hi @radek1, I have found something weird of the validation dataset you provided. \n\nAccording to the organizer's script for creating the validation dataset, when we split the 4 weeks train into 3-weeks-train (part-1) and 1-week-train (part-2), the part-2's aids should have all been seen in part-1. However, the `test.parquet` in the part-2 still have more than 2% aids which are not seen by part-1.\n\nI have shown this problem in this version of the notebook [here](https://www.kaggle.com/code/danielliao/reading-otto-recsys-organizer-scripts?scriptVersionId=114730145&cellId=9)", "votes": 1 }, { "id": 2049291, "postDate": "2022-11-30T03:18:22.520Z", "content": "Thank you for sharing!!
", "rawMarkdown": "Thank you for sharing!!", "votes": 1, "replies": [ { "id": 2053433, "postDate": "2022-12-03T09:21:42.247Z", "content": "np @yuyatasaki! happy to be of help π
", "rawMarkdown": "np @yuyatasaki! happy to be of help π" } ] }, { "id": 2025309, "postDate": "2022-11-11T06:29:47.053Z", "content": "Thank you for sharing.
", "rawMarkdown": "Thank you for sharing.", "votes": 1, "replies": [ { "id": 2028983, "postDate": "2022-11-14T11:07:04.497Z", "content": "np @zhenzheng0328! Glad you found this useful! π
", "rawMarkdown": "np @zhenzheng0328! Glad you found this useful! π" } ] }, { "id": 2024958, "postDate": "2022-11-10T22:35:50.763Z", "content": "Awesome Data to work withβ¦ Thanks for sharing thisβ¦
", "rawMarkdown": "Awesome Data to work with... Thanks for sharing this...", "votes": 1, "replies": [ { "id": 2024969, "postDate": "2022-11-10T22:50:34.307Z", "content": "NP @saha8631, glad I could be of help! π
", "rawMarkdown": "NP @saha8631, glad I could be of help! π", "votes": 1 } ] }, { "id": 2069469, "postDate": "2022-12-19T02:44:29.387Z", "content": "Thanks for sharing. I have a question, after you split the first 3 weeks into the train the last week into the test. How did you further split to get the test.parquet and test_labels.parquet?
The process is explained in Otto's GitHub here. These are the steps
\nnp.random.uniform(1,len(user_events)). The first random half of user activity becomes test.parquet and the second random half becomes test_labels.parquet.test_labels.parquet that does not exist in train or test gets removed.Thanks for the explanation @cdeotte , very clear. You saved my day
", "rawMarkdown": "Thanks for the explanation @cdeotte , very clear. You saved my day", "votes": 1 }, { "id": 2088453, "postDate": "2023-01-06T10:27:22.367Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2090215, "postDate": "2023-01-07T05:33:11.877Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 2059672, "postDate": "2022-12-09T05:56:54.437Z", "content": "Thank you for sharing validation strategy. I may be wrong because I haven't thought about this too much yet, but I would like to ask one question.
\nYou adopted the last 7 days of train data for validation set. This means the users are same in train and validation while LB and train is different in users. From your result, domain shift by user doesn't occur, right?
It's a valid concern, but there is no overlap. Any overlap of the sort you mention has been removed.
", "rawMarkdown": "It's a valid concern, but there is no overlap. Any overlap of the sort you mention has been removed.", "votes": 1 }, { "id": 2059802, "postDate": "2022-12-09T08:36:39.703Z", "content": "Thank you so muchοΌ
", "rawMarkdown": "Thank you so muchοΌ" } ] }, { "id": 2057414, "postDate": "2022-12-07T04:52:09.797Z", "content": "Thank you for sharing :)
", "rawMarkdown": "Thank you for sharing :)", "replies": [ { "id": 2057621, "postDate": "2022-12-07T09:04:02.877Z", "content": "Glad to be of help, @songwonho! Thanks for your kind comment π
", "rawMarkdown": "Glad to be of help, @songwonho! Thanks for your kind comment π " } ] }, { "id": 2053420, "postDate": "2022-12-03T08:47:54.650Z", "content": "I was doing sanity check and I didn't understand how ground-truth is created. Can someone verify they are correct or not?
\n![\"1\"](\"https://i.imgur.com/E8pJ5ZK.png\")
![\"2\"](\"https://i.imgur.com/fUYju6U.png\")
Why this one doesn't have a ground-truth for carts? Does the created ground-truth belongs to a timestamp after the cart happens?
\n![\"3\"](\"https://i.imgur.com/dIAYGi4.png\")
I think ground-truth labels belong to highlighted timestamp. Is this an intended behavior?
\nEdit: I figured why it looks like that. Sessions are also randomly split while creating ground-truth. I'm not sure whether this is prone to leakage or not.
", "rawMarkdown": "I was doing sanity check and I didn't understand how ground-truth is created. Can someone verify they are correct or not?\n\n \n\n\n\nWhy this one doesn't have a ground-truth for carts? Does the created ground-truth belongs to a timestamp after the cart happens?\n\n\n\nI think ground-truth labels belong to highlighted timestamp. Is this an intended behavior?\n\nEdit: I figured why it looks like that. Sessions are also randomly split while creating ground-truth. I'm not sure whether this is prone to leakage or not." }, { "id": 2034529, "postDate": "2022-11-18T08:02:06.210Z", "content": "\n\nIMPORTANT CAVEAT: Please note that in my dataset, in order to minimize its size without losing information (unless you care about milliseconds, but I assume seconds should be enough π), I divide the ts column by 1000.
\n
@radek1 is absolutely right about the loss of information. It is only millisecond accuracy is affected, not second level accuracy.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F850197%2Fc9d2ab93252b3f994932fb9e2bd38bd4%2FScreen%20Shot%202022-11-18%20at%2016.03.30.png?generation=1668758635832236&alt=media\")
\n\nIMPORTANT CAVEAT: Please note that in my dataset, in order to minimize its size without losing information (unless you care about milliseconds, but I assume seconds should be enough π), I divide the ts column by 1000. Notebooks on Kaggle by other people use unmodified data, so if you would want to run this validation with that code, you need to either make changes to the code or multiply the ts column by 1000.
\n
Hi @radek1, many thanks for this great post with rich resources to dig into. Sorry to still bother you with the dividing 1000 issue.
\nI am a little confused about which dataset you are referring to in the quotation at the top. In the dataset you said you \"divide the ts column by 1000\", is it the dataset named \"Otto Full Optimized Memory Footprint\" which is processed by \"process_data.ipynb\"? If so, why I can't find the /1000 in the source code of \"process_data.ipynb\"?
If you refer to other dataset, could you paste me the link to the source code where you divide ts by 1000? Thanks
", "rawMarkdown": "> IMPORTANT CAVEAT: Please note that in my dataset, in order to minimize its size without losing information (unless you care about milliseconds, but I assume seconds should be enough π), I divide the ts column by 1000. Notebooks on Kaggle by other people use unmodified data, so if you would want to run this validation with that code, you need to either make changes to the code or multiply the ts column by 1000.\n\nHi @radek1, many thanks for this great post with rich resources to dig into. Sorry to still bother you with the dividing 1000 issue.\n\nI am a little confused about which dataset you are referring to in the quotation at the top. In the dataset you said you \"divide the ts column by 1000\", is it the dataset named \"Otto Full Optimized Memory Footprint\" which is processed by \"process_data.ipynb\"? If so, why I can't find the `/1000` in the [source code](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363843#2015604) of \"process_data.ipynb\"?\n\nIf you refer to other dataset, could you paste me the link to the source code where you divide ts by 1000? Thanks", "replies": [ { "id": 2034279, "postDate": "2022-11-18T02:15:30.467Z", "content": "I have experimented and can confirm that convert type from string to uint8 can save CPU RAM significantly. I don't see the code of dividing 1000 on ts, so I can't experiment to see whether it is really saving RAM.
For validation like outlined here, I created this dataset (it has test labels! π)
\nIf you'd like to check what impact this processing has, you can load the data that doesn't have the ts column modified (you can use an earlier version of my dataset, call memory_usage(deep=True) on the dataframe, do df.ts = (df.ts / 1000).astype(np.int32) and call memory usage as well. You will notice you don't lose information (unless you care about milliseconds) and the data occupies much less memory!
(btw I typed the commands out from memory, so maybe you need to change something here and there for it to work π)
", "rawMarkdown": "For validation like outlined here, I created [this dataset](https://www.kaggle.com/datasets/radek1/otto-train-and-test-data-for-local-validation) (it has test labels! π)\n\nIf you'd like to check what impact this processing has, you can load the data that doesn't have the `ts` column modified (you can use [an earlier version of my dataset](https://www.kaggle.com/datasets/radek1/otto-full-optimized-memory-footprint/versions/1), call `memory_usage(deep=True)` on the dataframe, do `df.ts = (df.ts / 1000).astype(np.int32)` and call memory usage as well. You will notice you don't lose information (unless you care about milliseconds) and the data occupies much less memory!\n\n(btw I typed the commands out from memory, so maybe you need to change something here and there for it to work π) ", "votes": 1 }, { "id": 2034317, "postDate": "2022-11-18T03:30:40.877Z", "content": "Thanks a lot @radek1!
\nI now noticed that there are two versions of otto full, and the second version must have used ts/1000. I tried to add the first version in a notebook but I can only find the version 2 not the version 1. How do I add the version 1 instead of version 2 (the default version)? Thank you
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F850197%2F256cc78d940aa1ccc27d66ea6647013d%2FScreen%20Shot%202022-11-18%20at%2011.28.20.png?generation=1668742201458255&alt=media\")
Well, with a patient and careful look, the way to switch different versions is right there.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F850197%2F9316d4f6815d810f3e155d169b1d4996%2FScreen%20Shot%202022-11-18%20at%2013.53.50.png?generation=1668750890542455&alt=media\")
I just tested with the test set, and you are absolutely right that after df.ts = (df.ts / 1000).astype(np.int32), the RAM usage of ts is halved.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F850197%2Fc6c608210bab954761ebda1db0326b9d%2FScreen%20Shot%202022-11-18%20at%2014.56.11.png?generation=1668756449198493&alt=media\")
Hi, @radek1, thank you for sharing. I am also trying to replicate your steps from scratch but get into trouble in the very first step. May I ask how did you create the train/valid sets using the host's repo? I am trying to do it with Kaggle notebook, what I did is:
\n!git clone https://github.com/otto-de/recsys-dataset.git sys\nsys.path.insert(-, )\n!pipenv run python -m src.testset --train-set ../input/otto-recommender-system/train.jsonl --days 7 --output-path 'out/' --seed 42No module named 'src'. I think I somehow put the host's repo int the wrong directory, but I don't know how to fix it. (Question from a Linux noob).Thank you.
", "rawMarkdown": "Hi, @radek1, thank you for sharing. I am also trying to replicate your steps from scratch but get into trouble in the very first step. May I ask how did you create the train/valid sets using the host's repo? I am trying to do it with Kaggle notebook, what I did is:\n1. `!git clone https://github.com/otto-de/recsys-dataset.git`\n2. Insert this into the system paths\n```python\nimport sys\nsys.path.insert(-1, './recsys-dataset')\n```\n3. Run the script\n`!pipenv run python -m src.testset --train-set ../input/otto-recommender-system/train.jsonl --days 7 --output-path 'out/' --seed 42`\nHowever, I constantly got the error saying `No module named 'src'`. I think I somehow put the host's repo int the wrong directory, but I don't know how to fix it. (Question from a Linux noob).\n\nThank you.", "replies": [ { "id": 2023564, "postDate": "2022-11-09T21:18:40.517Z", "content": "Hey @shinomoriaoshi!
\nI ran the script like this: python -m src.testset --train-set data/train.jsonl --days 7 --output-path 'out/' --seed 42
Hope this helps! π
", "rawMarkdown": "Hey @shinomoriaoshi!\n\nI ran the script like this: `python -m src.testset --train-set data/train.jsonl --days 7 --output-path 'out/' --seed 42`\n\nHope this helps! π", "votes": 2 }, { "id": 2057618, "postDate": "2022-12-07T08:59:23.913Z", "content": "@shinomoriaoshi have you run successfully the module from kaggle? If so hwo?
", "rawMarkdown": "@shinomoriaoshi have you run successfully the module from kaggle? If so hwo?" } ] }, { "id": 2095668, "postDate": "2023-01-11T14:52:50.863Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2029213, "postDate": "2022-11-14T14:07:46.810Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 2029811, "postDate": "2022-11-15T00:28:33.123Z", "content": "I outlined the method here, please see this comment
", "rawMarkdown": "I outlined the method here, please see [this comment](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363843#2015604)", "votes": 1 } ] }, { "id": 2109732, "postDate": "2023-01-21T16:01:39.520Z", "content": "Thanks for your sharing
", "rawMarkdown": "Thanks for your sharing" }, { "id": 2098515, "postDate": "2023-01-13T16:20:44.553Z", "content": "So cool.Thanks for sharing
", "rawMarkdown": "So cool.Thanks for sharing" } ], "comments": [ { "id": 2024516, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-11-10T15:25:34.453000", "content": "FYI, I published a version of Radek's validation data as multiple parquets and restored columns ts and type back to their original. Radek's dataset is more memory and disk efficient, but I published this alternative dataset for anyone who has a pipeline using a Kaggle dataset with multiple parquets and original ts and type columns.
Now you can run your exact pipeline with validation data and compute CV score without making any changes to your pipeline. The Kaggle dataset is here. I plan to publish a notebook which computes validation score for my notebook soon.
", "votes": 9, "replies": [ { "id": 2024873, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-11-10T20:03:44.977000", "content": "UPDATE: I published a train notebook that achieves LB 0.573 and corresponding validation notebook \"CV\" 0.563 using Radek's validation data (in my Kaggle dataset).
", "votes": 5, "replies": [] }, { "id": 2025144, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-11-11T02:37:13.650000", "content": "After publishing the notebook, i made two changes that boosted CV +0.003 and after submission, the LB boosted +0.003. The CV LB correlate very well.
Your CV scheme (data and metric code) has perfect correlation for me too. Thank you for publishing your setup. The first two data points are my public notebook version 4 and 9 here (with LB 0.556 and LB 0.562 respectively). Then next data point is Pietro's notebook here (when it was/is LB 0.565 version 1). The next data point is Ingvaras' notebook here (when it was/is LB 0.570 version 3). We see that each notebook improves both CV and LB. The last data point is my current LB score using some unpublished tricks.
\n
wonderful data, thank you for sharing this @cdeotte! π
", "votes": 2, "replies": [] }, { "id": 2075047, "author_name": "Jensen", "author_url": "", "post_date": "2022-12-25T01:16:59.220000", "content": "Hi,Chris.I used your version 9 notebook to calculate the local CV. I suspect something went wrong with my idea of calculating local CV scores. In the notebook, I generated candidates for val A's validation set and then calculated the local CV score, I only made the following modifications, but the CV score looks so outrageous. What went wrong with me? I am looking forward to hearing from you.
\n
Thanks for sharing, this saves a lot of tine!
", "votes": 5, "replies": [ { "id": 2054296, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-04T02:24:31.563000", "content": "Thank you, JFP π Really glad to be of help
", "votes": 1, "replies": [] } ] }, { "id": 2023511, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-11-09T19:43:42.153000", "content": "This is very helpful thank you. I used your files and code to compute the CV score for my notebook version 4 with LB 0.556. The computed CV score is 0.474 so I think there is a bug in my code somewhere. It appears that you have a smaller gap (between LB and CV) when using your CV files. I will review my code and try againβ¦
", "votes": 3, "replies": [ { "id": 2023514, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-11-09T19:51:17.187000", "content": "I may have found the problem. Your parquet files have ts column with original_ts / 1000. So my notebook's pipeline which adds weight based on ts doesn't work correctly. I will update and check CV score.
You should add a note to your discussion here. Anyone who uses code based on original ts like my Chris' notebook, or Vladimir's notebook, or Pietro's notebook, or Ingvaras's notebook will run into this problem. For example the following line of code:
df.query('abs(ts_x - ts_y) < 24 * 60 * 60 * 1000 and aid_x != aid_y')\nwill waste many hours running code with your pipeline before discovering the error. Also type conversion will fail
\n.astype({\"ts\": \"datetime64[ms]\"})\nYes Chris, that is a great point! Apologies for causing all this trouble on your end. You are absolutely right.
\nI included this bit of information in the other dataset that I published, in the description, but didn't do so here. Very sorry bout that. Will correct this in a second and add a note.
\nMy apologies for all the trouble.
", "votes": 1, "replies": [] }, { "id": 2023578, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-11-09T21:35:50.963000", "content": "Great. Looks good now. My computed CV is 0.548 for version 4 of my notebook with LB 0.556. That's pretty close. I will compute more CV scores for different versions and check the correlation. I will post results here.
\nThen i will begin boosting my CV LB with improved logic! and/or trained models!
", "votes": 3, "replies": [] }, { "id": 2023731, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-10T01:43:16.273000", "content": "It's really cool how you are sharing openly how you plan to approach this competition, I think that is great learning material! π
\nMy plan is very similar, hoping to share some stuff on Matrix Factorization maybe with using some of the Merlin stuff, have some really cool ideas, but the only problem is finding the time π
\nGreat there are that many weeks still left in this competition! π
", "votes": 2, "replies": [] } ] }, { "id": 2029433, "author_name": "Piotr Gabrys", "author_url": "", "post_date": "2022-11-14T16:31:57.847000", "content": "Thanks for the dataset. I've spotted a problem with validation dataset. There are \"new\" AIDs (18785) in the validation test set comparing to validations train set). For the original train and test set this is not the case.
\nI've added notebook with the calculation to the dataset: https://www.kaggle.com/code/piotrekga/new-aids-in-test-set
\nI guess there was more logic in train/test split. @pnormann, could you please give us information on what to do with sessions and ground truth with new AIDs?
", "votes": 4, "replies": [ { "id": 2029810, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-15T00:27:10.533000", "content": "ha, interesting observation @piotrekga!
\nI think this might just be coincidental, new AIDs might have gotten added to the system that week π€ Or, alternatively, there was some additional logic in pulling down the data.
\nHere is the logic used by the organizer to create the test set. There is nothing there about filtering sessions based on aids AFAICT at quick glance.
That's a helpful discovery Piotr. Differences between CV and LB are a concern.
\nHowever, I will comment that Radek's validation data is perfectly correlating with LB. Every 0.0002+ improvement on Radek's CV gives me a boost on LB!
@radek1 I've rerun the train-test split on train.jsonl before and got exactly the same results. You're right the logic is not included in the code provided by organisers.
About new AIDs: taking into consideration that these are users behaviours and the dataset has long tails I find it very likely that there were no new AIDs in the original test set and thousands of them in new train-test split. Of course I may be wrong π . I really hope organisers will provide us with some additional clarification in that regards.
\n@cdeotte, you're right that now the CV-LB correlation works. I'd really like to validate my model locally as close as possible to the final validation though.
", "votes": 2, "replies": [] }, { "id": 2032183, "author_name": "Philipp Normann", "author_url": "", "post_date": "2022-11-16T13:19:34.117000", "content": "Hey @piotrekga, thanks for pointing this out and sorry for the confusion! This behavior was a bug in our public repo that was inconsistent with how we created the dataset for the competition. We created the actual dataset using a Clojure job, and during the reimplementation in Python, we, unfortunately, forgot to add this preprocessing step. I just pushed an update to our GitHub repo which fixes this issue. Although I'm still not entirely happy with the performance of the Python script, it should be enough for you to create your local validation set, which reflects the logic we used for the LB test set. I'll see what I can do to speed things up π
", "votes": 8, "replies": [] }, { "id": 2034542, "author_name": "Piotr Gabrys", "author_url": "", "post_date": "2022-11-18T08:18:33.893000", "content": "Thank you @pnormann for the update. At this point logic is far more important than performance.
", "votes": 2, "replies": [] } ] }, { "id": 2093576, "author_name": "Pradeep Pujari", "author_url": "", "post_date": "2023-01-10T05:35:13.883000", "content": "Sorry @radek1 what is the id2type and vice versa pkl files contain? What is the purpose of these 2 data files?
", "votes": 1, "replies": [ { "id": 2094694, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2023-01-11T00:46:19.597000", "content": "hey @ppujari! They just contain the mapping of types to their ids IIRC, 0 maps to clicks, 1 to carts and 2 to orders. Probably shouldn't have included it, makes it all a bit confusing.
Hi @radek1, I have found something weird of the validation dataset you provided.
\nAccording to the organizer's script for creating the validation dataset, when we split the 4 weeks train into 3-weeks-train (part-1) and 1-week-train (part-2), the part-2's aids should have all been seen in part-1. However, the test.parquet in the part-2 still have more than 2% aids which are not seen by part-1.
I have shown this problem in this version of the notebook here
", "votes": 1, "replies": [] }, { "id": 2049291, "author_name": "tsk", "author_url": "", "post_date": "2022-11-30T03:18:22.520000", "content": "Thank you for sharing!!
", "votes": 1, "replies": [ { "id": 2053433, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-03T09:21:42.247000", "content": "np @yuyatasaki! happy to be of help π
", "votes": 0, "replies": [] } ] }, { "id": 2025309, "author_name": "zhez017", "author_url": "", "post_date": "2022-11-11T06:29:47.053000", "content": "Thank you for sharing.
", "votes": 1, "replies": [ { "id": 2028983, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-14T11:07:04.497000", "content": "np @zhenzheng0328! Glad you found this useful! π
", "votes": 0, "replies": [] } ] }, { "id": 2024958, "author_name": "Raj Saha", "author_url": "", "post_date": "2022-11-10T22:35:50.763000", "content": "Awesome Data to work withβ¦ Thanks for sharing thisβ¦
", "votes": 1, "replies": [ { "id": 2024969, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-10T22:50:34.307000", "content": "NP @saha8631, glad I could be of help! π
", "votes": 1, "replies": [] } ] }, { "id": 2069469, "author_name": "william.wu", "author_url": "", "post_date": "2022-12-19T02:44:29.387000", "content": "Thanks for sharing. I have a question, after you split the first 3 weeks into the train the last week into the test. How did you further split to get the test.parquet and test_labels.parquet?
The process is explained in Otto's GitHub here. These are the steps
\nnp.random.uniform(1,len(user_events)). The first random half of user activity becomes test.parquet and the second random half becomes test_labels.parquet.test_labels.parquet that does not exist in train or test gets removed.Thanks for the explanation @cdeotte , very clear. You saved my day
", "votes": 1, "replies": [] }, { "id": 2088453, "author_name": "", "author_url": "", "post_date": "2023-01-06T10:27:22.367000", "content": "", "votes": 0, "replies": [] }, { "id": 2090215, "author_name": "", "author_url": "", "post_date": "2023-01-07T05:33:11.877000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2059672, "author_name": "wakaka", "author_url": "", "post_date": "2022-12-09T05:56:54.437000", "content": "Thank you for sharing validation strategy. I may be wrong because I haven't thought about this too much yet, but I would like to ask one question.
\nYou adopted the last 7 days of train data for validation set. This means the users are same in train and validation while LB and train is different in users. From your result, domain shift by user doesn't occur, right?
It's a valid concern, but there is no overlap. Any overlap of the sort you mention has been removed.
", "votes": 1, "replies": [] }, { "id": 2059802, "author_name": "wakaka", "author_url": "", "post_date": "2022-12-09T08:36:39.703000", "content": "Thank you so muchοΌ
", "votes": 0, "replies": [] } ] }, { "id": 2057414, "author_name": "Wonho Song", "author_url": "", "post_date": "2022-12-07T04:52:09.797000", "content": "Thank you for sharing :)
", "votes": 0, "replies": [ { "id": 2057621, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-07T09:04:02.877000", "content": "Glad to be of help, @songwonho! Thanks for your kind comment π
", "votes": 0, "replies": [] } ] }, { "id": 2053420, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-12-03T08:47:54.650000", "content": "I was doing sanity check and I didn't understand how ground-truth is created. Can someone verify they are correct or not?
\n
Why this one doesn't have a ground-truth for carts? Does the created ground-truth belongs to a timestamp after the cart happens?
\n
I think ground-truth labels belong to highlighted timestamp. Is this an intended behavior?
\nEdit: I figured why it looks like that. Sessions are also randomly split while creating ground-truth. I'm not sure whether this is prone to leakage or not.
", "votes": 0, "replies": [] }, { "id": 2034529, "author_name": "danielliao", "author_url": "", "post_date": "2022-11-18T08:02:06.210000", "content": "\n\nIMPORTANT CAVEAT: Please note that in my dataset, in order to minimize its size without losing information (unless you care about milliseconds, but I assume seconds should be enough π), I divide the ts column by 1000.
\n
@radek1 is absolutely right about the loss of information. It is only millisecond accuracy is affected, not second level accuracy.
\n
\n\nIMPORTANT CAVEAT: Please note that in my dataset, in order to minimize its size without losing information (unless you care about milliseconds, but I assume seconds should be enough π), I divide the ts column by 1000. Notebooks on Kaggle by other people use unmodified data, so if you would want to run this validation with that code, you need to either make changes to the code or multiply the ts column by 1000.
\n
Hi @radek1, many thanks for this great post with rich resources to dig into. Sorry to still bother you with the dividing 1000 issue.
\nI am a little confused about which dataset you are referring to in the quotation at the top. In the dataset you said you \"divide the ts column by 1000\", is it the dataset named \"Otto Full Optimized Memory Footprint\" which is processed by \"process_data.ipynb\"? If so, why I can't find the /1000 in the source code of \"process_data.ipynb\"?
If you refer to other dataset, could you paste me the link to the source code where you divide ts by 1000? Thanks
", "votes": 0, "replies": [ { "id": 2034279, "author_name": "danielliao", "author_url": "", "post_date": "2022-11-18T02:15:30.467000", "content": "I have experimented and can confirm that convert type from string to uint8 can save CPU RAM significantly. I don't see the code of dividing 1000 on ts, so I can't experiment to see whether it is really saving RAM.
For validation like outlined here, I created this dataset (it has test labels! π)
\nIf you'd like to check what impact this processing has, you can load the data that doesn't have the ts column modified (you can use an earlier version of my dataset, call memory_usage(deep=True) on the dataframe, do df.ts = (df.ts / 1000).astype(np.int32) and call memory usage as well. You will notice you don't lose information (unless you care about milliseconds) and the data occupies much less memory!
(btw I typed the commands out from memory, so maybe you need to change something here and there for it to work π)
", "votes": 1, "replies": [] }, { "id": 2034317, "author_name": "danielliao", "author_url": "", "post_date": "2022-11-18T03:30:40.877000", "content": "Thanks a lot @radek1!
\nI now noticed that there are two versions of otto full, and the second version must have used ts/1000. I tried to add the first version in a notebook but I can only find the version 2 not the version 1. How do I add the version 1 instead of version 2 (the default version)? Thank you
Well, with a patient and careful look, the way to switch different versions is right there.
\n
I just tested with the test set, and you are absolutely right that after df.ts = (df.ts / 1000).astype(np.int32), the RAM usage of ts is halved.
\n
Hi, @radek1, thank you for sharing. I am also trying to replicate your steps from scratch but get into trouble in the very first step. May I ask how did you create the train/valid sets using the host's repo? I am trying to do it with Kaggle notebook, what I did is:
\n!git clone https://github.com/otto-de/recsys-dataset.git sys\nsys.path.insert(-, )\n!pipenv run python -m src.testset --train-set ../input/otto-recommender-system/train.jsonl --days 7 --output-path 'out/' --seed 42No module named 'src'. I think I somehow put the host's repo int the wrong directory, but I don't know how to fix it. (Question from a Linux noob).Thank you.
", "votes": 0, "replies": [ { "id": 2023564, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-09T21:18:40.517000", "content": "Hey @shinomoriaoshi!
\nI ran the script like this: python -m src.testset --train-set data/train.jsonl --days 7 --output-path 'out/' --seed 42
Hope this helps! π
", "votes": 2, "replies": [] }, { "id": 2057618, "author_name": "Davide Stenner", "author_url": "", "post_date": "2022-12-07T08:59:23.913000", "content": "@shinomoriaoshi have you run successfully the module from kaggle? If so hwo?
", "votes": 0, "replies": [] } ] }, { "id": 2095668, "author_name": "", "author_url": "", "post_date": "2023-01-11T14:52:50.863000", "content": "", "votes": 0, "replies": [] }, { "id": 2029213, "author_name": "", "author_url": "", "post_date": "2022-11-14T14:07:46.810000", "content": "", "votes": 1, "replies": [ { "id": 2029811, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-15T00:28:33.123000", "content": "I outlined the method here, please see this comment
", "votes": 1, "replies": [] } ] }, { "id": 2109732, "author_name": "carolineoo", "author_url": "", "post_date": "2023-01-21T16:01:39.520000", "content": "Thanks for your sharing
", "votes": 0, "replies": [] }, { "id": 2098515, "author_name": "Jason", "author_url": "", "post_date": "2023-01-13T16:20:44.553000", "content": "So cool.Thanks for sharing
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2022844": "Hey!\n\nI ran a couple of experiments and local validation tracks public LB perfectly! Here is what the situation looks like:\n\n\n\nThe results are slightly poorer in local validation, but that is not surprising -- we are running on one week less of train data!\n\nHere's the setup:\n* I generated the split of the train set using the code in [organizer's repo](https://github.com/otto-de/recsys-dataset)\n* I am using the last 7 days of train set for validation (similarly to the test set)\n* If you don't want to run the code on the data yourself, I [uploaded the prepared data to Kaggle](https://www.kaggle.com/datasets/radek1/otto-train-and-test-data-for-local-validation)\n\nThere has been a lot of conversation on the forums on the metric, I think we now have a good understanding of how it works π (one can also always confirm one's understanding by checking out the organizer's repo!). Interesting discussions on the forums [here](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364064) and [here](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364530).\n\nHere is the code I am using for calculating the metric in pandas, starting from a valid submission DataFrame, in case this might be of help:\n\n```\n submission['session'] = submission.session_type.apply(lambda x: int(x.split('_')[0]))\n submission['type'] = submission.session_type.apply(lambda x: x.split('_')[1])\n submission.labels = submission.labels.apply(lambda x: [int(i) for i in x.split(' ')[:20]])\n\n test_labels = pd.read_parquet('out_processed/test_labels.parquet')\n test_labels = test_labels.merge(submission, how='left', on=['session', 'type'])\n test_labels['hits'] = test_labels.apply(lambda df: len(set(df.ground_truth).intersection(set(df.labels))), axis=1)\n test_labels['gt_count'] = test_labels.ground_truth.str.len().clip(0,20)\n\n recall_per_type = test_labels.groupby(['type'])['hits'].sum() / test_labels.groupby(['type'])['gt_count'].sum() \n\n score = (recall_per_type * pd.Series({'clicks': 0.10, 'carts': 0.30, 'orders': 0.60})).sum()\n```\n\nCompetitions where local validation tracks public LB are much more fun, IMHO π\n\nWishing you all the best in the challenge!\nRadek\n\n**IMPORTANT CAVEAT:** Please note that in my dataset, in order to minimize its size without losing information (unless you care about milliseconds, but I assume seconds should be enough π), I divide the `ts` column by 1000. Notebooks on Kaggle by other people use unmodified data, so if you would want to run this validation with that code, you need to either make changes to the code or multiply the `ts` column by 1000.\n\n**A couple of related resources you might find useful:**\n\n* [π‘ [2 methods] How-to ensemble predictions π π π ](https://www.kaggle.com/code/radek1/2-methods-how-to-ensemble-predictions)\n* [local validation tracks public LB perfecty -- here is the setup](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364991)\n* [π‘ For my friends from Twitter and LinkedIn -- here is how to dive into this competition π³](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368560)\n* [Full dataset processed to CSV/parquet files with optimized memory footprint](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363843)\n* [co-visitation matrix - simplified, imprvd logic π₯](https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic)\n* [π‘ Word2Vec How-to [training and submission]πππ](https://www.kaggle.com/code/radek1/word2vec-how-to-training-and-submission)\n\n", "2024516": "FYI, I published a version of Radek's validation data as multiple parquets and restored columns `ts` and `type` back to their original. Radek's dataset is more memory and disk efficient, but I published this alternative dataset for anyone who has a pipeline using a Kaggle dataset with multiple parquets and original `ts` and `type` columns. \n\nNow you can run your exact pipeline with validation data and compute CV score without making any changes to your pipeline. The Kaggle dataset is [here][1]. I plan to publish a notebook which computes validation score for my notebook soon.\n\n[1]: https://www.kaggle.com/datasets/cdeotte/otto-validation", "2023756": "Your CV scheme (data and metric code) has perfect correlation for me too. Thank you for publishing your setup. The first two data points are my public notebook version 4 and 9 [here][2] (with LB 0.556 and LB 0.562 respectively). Then next data point is Pietro's notebook [here][1] (when it was/is LB 0.565 version 1). The next data point is Ingvaras' notebook [here][3] (when it was/is LB 0.570 version 3). We see that each notebook improves both CV and LB. The last data point is my current LB score using some unpublished tricks.\n\n\n\n[1]: https://www.kaggle.com/code/pietromaldini1/multiple-clicks-vs-latest-items\n[2]: https://www.kaggle.com/code/cdeotte/test-data-leak-lb-boost\n[3]: https://www.kaggle.com/code/ingvarasgalinskas/item-type-vs-multiple-clicks-vs-latest-items", "2053831": "Thanks for sharing, this saves a lot of tine!", "2023511": "This is very helpful thank you. I used your files and code to compute the CV score for my notebook version 4 with LB 0.556. The computed CV score is 0.474 so I think there is a bug in my code somewhere. It appears that you have a smaller gap (between LB and CV) when using your CV files. I will review my code and try again...", "2029433": "Thanks for the dataset. I've spotted a problem with validation dataset. There are \"new\" AIDs (18785) in the validation test set comparing to validations train set). For the original train and test set this is not the case.\n\nI've added notebook with the calculation to the dataset: https://www.kaggle.com/code/piotrekga/new-aids-in-test-set\n\nI guess there was more logic in train/test split. @pnormann, could you please give us information on what to do with sessions and ground truth with new AIDs?", "2093576": "Sorry @radek1 what is the id2type and vice versa pkl files contain? What is the purpose of these 2 data files?", "2076910": "Hi @radek1, I have found something weird of the validation dataset you provided. \n\nAccording to the organizer's script for creating the validation dataset, when we split the 4 weeks train into 3-weeks-train (part-1) and 1-week-train (part-2), the part-2's aids should have all been seen in part-1. However, the `test.parquet` in the part-2 still have more than 2% aids which are not seen by part-1.\n\nI have shown this problem in this version of the notebook [here](https://www.kaggle.com/code/danielliao/reading-otto-recsys-organizer-scripts?scriptVersionId=114730145&cellId=9)", "2049291": "Thank you for sharing!!", "2025309": "Thank you for sharing.", "2024958": "Awesome Data to work with... Thanks for sharing this...", "2069469": "Thanks for sharing. I have a question, after you split the first 3 weeks into the train the last week into the test. How did you further split to get the `test.parquet` and `test_labels.parquet`?", "2059672": "Thank you for sharing validation strategy. I may be wrong because I haven't thought about this too much yet, but I would like to ask one question.\nYou adopted the last 7 days of train data for validation set. This means the users are same in train and validation while LB and train is different in users. From your result, domain shift by user doesn't occur, right?\n", "2057414": "Thank you for sharing :)", "2053420": "I was doing sanity check and I didn't understand how ground-truth is created. Can someone verify they are correct or not?\n\n \n\n\n\nWhy this one doesn't have a ground-truth for carts? Does the created ground-truth belongs to a timestamp after the cart happens?\n\n\n\nI think ground-truth labels belong to highlighted timestamp. Is this an intended behavior?\n\nEdit: I figured why it looks like that. Sessions are also randomly split while creating ground-truth. I'm not sure whether this is prone to leakage or not.", "2034529": "> IMPORTANT CAVEAT: Please note that in my dataset, in order to minimize its size without losing information (unless you care about milliseconds, but I assume seconds should be enough π), I divide the ts column by 1000.\n\n@radek1 is absolutely right about the loss of information. It is only millisecond accuracy is affected, not second level accuracy.\n\n\n\n\n", "2034277": "> IMPORTANT CAVEAT: Please note that in my dataset, in order to minimize its size without losing information (unless you care about milliseconds, but I assume seconds should be enough π), I divide the ts column by 1000. Notebooks on Kaggle by other people use unmodified data, so if you would want to run this validation with that code, you need to either make changes to the code or multiply the ts column by 1000.\n\nHi @radek1, many thanks for this great post with rich resources to dig into. Sorry to still bother you with the dividing 1000 issue.\n\nI am a little confused about which dataset you are referring to in the quotation at the top. In the dataset you said you \"divide the ts column by 1000\", is it the dataset named \"Otto Full Optimized Memory Footprint\" which is processed by \"process_data.ipynb\"? If so, why I can't find the `/1000` in the [source code](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363843#2015604) of \"process_data.ipynb\"?\n\nIf you refer to other dataset, could you paste me the link to the source code where you divide ts by 1000? Thanks", "2022985": "Hi, @radek1, thank you for sharing. I am also trying to replicate your steps from scratch but get into trouble in the very first step. May I ask how did you create the train/valid sets using the host's repo? I am trying to do it with Kaggle notebook, what I did is:\n1. `!git clone https://github.com/otto-de/recsys-dataset.git`\n2. Insert this into the system paths\n```python\nimport sys\nsys.path.insert(-1, './recsys-dataset')\n```\n3. Run the script\n`!pipenv run python -m src.testset --train-set ../input/otto-recommender-system/train.jsonl --days 7 --output-path 'out/' --seed 42`\nHowever, I constantly got the error saying `No module named 'src'`. I think I somehow put the host's repo int the wrong directory, but I don't know how to fix it. (Question from a Linux noob).\n\nThank you.", "2095668": "", "2029213": "", "2109732": "Thanks for your sharing", "2098515": "So cool.Thanks for sharing" } }