![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2Fd1438821834f8779624042ba02b4d132%2Fpreprocessing_script.png?generation=1667477805785949&alt=media\")
Hey friends!
\nI converted the dataset to live in csv & parquet files. You can find the dataset here.
I have also created an explainer notebook. In it, I run you through how to use this data.
\nI am also attaching the code I used to process this data as a notebook file to this post. I couldn't process the data on Kaggle due to RAM limitations so I ran it on my local machine and uploaded it to Kaggle.
\nHope this can speed you along in your work π Happy kaggling! π₯³
\nHey friends!
\nI converted the dataset to live in csv & parquet files. You can find the dataset here.
I have also created an explainer notebook. In it, I run you through how to use this data.
\nI am also attaching the code I used to process this data as a notebook file to this post. I couldn't process the data on Kaggle due to RAM limitations so I ran it on my local machine and uploaded it to Kaggle.
\nHope this can speed you along in your work π Happy kaggling! π₯³
\nThis is the script I used for preprocessing as a screenshot
\n
Hi @radek1 ! Glad to see you here.
\nI was taking a look at the data and I have seen that for example session 0 in train data lasts from July 31 (ts β 1659304800025) to August 28 (ts β 1661684983707)
\nIn my concept of a session, lasting a month seems too much, I don't know what you think.
\nAnyway, when I can I will investigate it further and I will tell you something, I don't know if it is something common or not
Please take a look at the answer here or alternatively at the section that discusses this in my EDA.
\nThis seemed suspect to me as well but turns out it is just about the session definition used by the organizer π
", "rawMarkdown": "Please take a look at the answer [here](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363554#2015486) or alternatively at the section that discusses this [in my EDA](https://www.kaggle.com/code/radek1/eda-an-overview-of-the-full-dataset).\n\nThis seemed suspect to me as well but turns out it is just about the session definition used by the organizer π", "votes": null }, { "id": "2016173", "postDate": "11/03/2022 20:28:01", "content": "As I wrote in yours \"last 20 aids\": well-done, great contribution to Otto competition.
", "rawMarkdown": "As I wrote in yours \"last 20 aids\": well-done, great contribution to Otto competition.", "votes": null }, { "id": "2016245", "postDate": "11/03/2022 21:28:28", "content": "Thank you, @mpwolke! π
", "rawMarkdown": "Thank you, @mpwolke! π", "votes": null }, { "id": "2018373", "postDate": "11/05/2022 16:19:33", "content": "Great work! upvoted!
", "rawMarkdown": "Great work! upvoted!", "votes": null }, { "id": "2018688", "postDate": "11/05/2022 22:32:22", "content": "Thank you, @nyamashina, very glad you are finding this useful! π
", "rawMarkdown": "Thank you, @nyamashina, very glad you are finding this useful! π", "votes": null }, { "id": "2018761", "postDate": "11/06/2022 01:18:27", "content": "thanks your notebook, i am a new kaggler, cay i ask you , why json to csv or parquet?
", "rawMarkdown": "thanks your notebook, i am a new kaggler, cay i ask you , why json to csv or parquet?", "votes": null }, { "id": "2018854", "postDate": "11/06/2022 04:44:45", "content": "hey @iocmeta! csv or parquet is easier to read and process with the most common library that most people use, which is pandas π Generally, csv and parquet files are easier to use for ML work in most circumstances, I believe.
Hi @radek1 Thank you so much for sharing!
\nAs you said above to Kaggle does not provide enough RAM (30G in CPU) to process the data, therefore I won't have luck here to run your process_data.ipynb in my apple M1 8G RAM laptop.
\nAlthough I can't process the entire OTTO dataset with your process_data.ipynb, I should be able to run all of the code for experiment as long as I have the entire OTTO dataset stored in my local disk, right?
", "rawMarkdown": "Hi @radek1 Thank you so much for sharing! \n\nAs you said above to Kaggle does not provide enough RAM (30G in CPU) to process the data, therefore I won't have luck here to run your process_data.ipynb in my apple M1 8G RAM laptop. \n\nAlthough I can't process the entire OTTO dataset with your process_data.ipynb, I should be able to run all of the code for experiment as long as I have the entire OTTO dataset stored in my local disk, right?", "votes": null }, { "id": "2023729", "postDate": "11/10/2022 01:41:00", "content": "Yeah that is a bit of a problem I am afraid. 8GB is very little, you would have to go to extremely lengths to conserve memory.
\nI am not completely sure that it is worth it to figure out how to do stuff with that small amount of RAM. I mean, you still can, but you might be better off just running things on Kaggle. 30GB of RAM plus unlimited runtime for CPU only kernels sounds like a sweet deal π
\nOf course, the situation doesn't look that hot when it comes to GPUs, but even 40 hours/week can get you very far I feel.
\nEssentially -- I think your time might be better spent figuring out how to gain access to a machine with more RAM (be that Kaggle kernels or cloud compute) then trying to figure out how to work with 8GB of RAM (unless you want to learn something really low level like Rust or C, etc, which is super cool as well but maybe a bit orthogonal to maximizing your learning as a data science practitioner building up their portfolio π)
", "rawMarkdown": "Yeah that is a bit of a problem I am afraid. 8GB is very little, you would have to go to extremely lengths to conserve memory.\n\nI am not completely sure that it is worth it to figure out how to do stuff with that small amount of RAM. I mean, you still can, but you might be better off just running things on Kaggle. 30GB of RAM plus unlimited runtime for CPU only kernels sounds like a sweet deal π\n\nOf course, the situation doesn't look that hot when it comes to GPUs, but even 40 hours/week can get you very far I feel. \n\nEssentially -- I think your time might be better spent figuring out how to gain access to a machine with more RAM (be that Kaggle kernels or cloud compute) then trying to figure out how to work with 8GB of RAM (unless you want to learn something really low level like Rust or C, etc, which is super cool as well but maybe a bit orthogonal to maximizing your learning as a data science practitioner building up their portfolio π)", "votes": null }, { "id": "2024279", "postDate": "11/10/2022 11:51:44", "content": "I tried to run process_data.ipynb to find out what the original json file look like. I love your codes, they are nice and simple to read and learn!
I also wanted to see how much ram does it take on Kaggle to convert test dataset to df. It seems 400MB jsonl takes nearly 4GB to do the processing.
\nHowever, when I reduced the chunksize to 100, it takes longer but the ram also reduced to 1.6 GB, which is 4 times larger than the test data size. It means 11 GB would require 44 GB ram on Kaggle to process.
So, I guess this is why the processing can not be done on kaggle on the training set.
", "rawMarkdown": "I tried to run `process_data.ipynb` to find out what the original json file look like. I love your codes, they are nice and simple to read and learn! \n\nI also wanted to see how much ram does it take on Kaggle to convert test dataset to df. It seems 400MB jsonl takes nearly 4GB to do the processing. \n\nHowever, when I reduced the `chunksize` to 100, it takes longer but the ram also reduced to 1.6 GB, which is 4 times larger than the test data size. It means 11 GB would require 44 GB ram on Kaggle to process. \n\nSo, I guess this is why the processing can not be done on kaggle on the training set.", "votes": null }, { "id": "2024286", "postDate": "11/10/2022 11:54:11", "content": "Thank you so much Radek for the advices! Very helpful indeed!
\nI only saw this reply just now. I should turn on the notification earlier.
", "rawMarkdown": "Thank you so much Radek for the advices! Very helpful indeed!\n\nI only saw this reply just now. I should turn on the notification earlier.", "votes": null }, { "id": "2031088", "postDate": "11/15/2022 20:32:32", "content": "Great work! Updated!
", "rawMarkdown": "Great work! Updated!", "votes": null }, { "id": "2031091", "postDate": "11/15/2022 20:33:57", "content": "Thanks @leiwong, appreciate it! π
", "rawMarkdown": "Thanks @leiwong, appreciate it! π", "votes": null }, { "id": "2031393", "postDate": "11/16/2022 03:47:39", "content": "Thanks so much for sharing this file Radek. Glad to see how much time I might expect to process the data on my machine!
", "rawMarkdown": "Thanks so much for sharing this file Radek. Glad to see how much time I might expect to process the data on my machine!", "votes": null }, { "id": "2031397", "postDate": "11/16/2022 03:50:11", "content": "Very glad you are finding this useful, @mattrosinski! π
", "rawMarkdown": "Very glad you are finding this useful, @mattrosinski! π", "votes": null }, { "id": "2034437", "postDate": "11/18/2022 06:12:11", "content": "Hi @radek1, since you have updated the otto fully optimized dataset in parquet by reducing the RAM further by dividing ts with 1000, I guess the original process_data.ipynb is not updated.
The updated process_data.ipynb will add this line into the last cell above right under the line test_df.type = test_df.type.astype(np.uint8), right?
test_df.ts = (test_df.ts/).astype(np.int32)\nIs this a Full data? Are you using full data running this notebook ?
", "rawMarkdown": "Is this a Full data? Are you using full data running this notebook ?", "votes": null }, { "id": "2065755", "postDate": "12/15/2022 03:39:57", "content": "Hey @luyi564! Yup, this is the full dataset
", "rawMarkdown": "Hey @luyi564! Yup, this is the full dataset", "votes": null }, { "id": "2066696", "postDate": "12/16/2022 00:50:26", "content": "Hi @radek1 , thanks for sharing. Do you still use the dataset you shared here for offline training & validation?
", "rawMarkdown": "Hi @radek1 , thanks for sharing. Do you still use the dataset you shared [here](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364991) for offline training & validation?", "votes": null }, { "id": "2066697", "postDate": "12/16/2022 00:54:36", "content": "In this competition, the session is actually user
Hey @wuwenmin! Yes, while working on the competition I was using the dataset that you mentioned.
", "rawMarkdown": "Hey @wuwenmin! Yes, while working on the competition I was using the dataset that you mentioned.", "votes": null }, { "id": "2069432", "postDate": "12/19/2022 01:28:59", "content": "Hi @radek1, I just recreated your optimized memory footprint dataset on Kaggle using polars within 30GB. Thank you for sharing and leading the way!
", "rawMarkdown": "Hi @radek1, I just [recreated](https://www.kaggle.com/code/danielliao/recreate-otto-full-optimized-memory-footprint/) your optimized memory footprint dataset on Kaggle using polars within 30GB. Thank you for sharing and leading the way!", "votes": null }, { "id": "2090270", "postDate": "01/07/2023 07:19:29", "content": "Thanks for your processed dataset!
\nYou can also use pd.Dataframe.from_records() to convert list of Dicts (session_data.events here) to a df, and then loop it for all sessions and concat them to get final events df.
\nAnyway I think it's more readable in your way to write the code.
Thank you for this work. It is very useful to the community.
", "rawMarkdown": "Thank you for this work. It is very useful to the community.", "votes": null }, { "id": "2090984", "postDate": "01/07/2023 22:06:03", "content": "Sorry for the delayed response, unfortunately it is not easy to track new comments using the Kaggle interfaceβ¦
\nBut yes, you are right @wuwenmin, one can think of the session being a user id. It is important to keep in mind though that there are no overlapping sessions (or users) between test and train, so using the session id as a feature will note be very useful!
@gpreda, thank you so much for your comments, much appreciated π It is a really nice feeling to have your work be appreciated and see that it is helping others.
\nThank you so much for taking the time to share this feedback with me! π€
", "rawMarkdown": "gpreda, thank you so much for your comments, much appreciated π It is a really nice feeling to have your work be appreciated and see that it is helping others.\n\nThank you so much for taking the time to share this feedback with me! π€", "votes": null } ], "comments": [ { "id": 2015604, "author_name": "radek1", "author_url": "", "post_date": "11/03/2022 12:16:51", "content": "This is the script I used for preprocessing as a screenshot
\n
Hi @radek1, since you have updated the otto fully optimized dataset in parquet by reducing the RAM further by dividing ts with 1000, I guess the original process_data.ipynb is not updated.
The updated process_data.ipynb will add this line into the last cell above right under the line test_df.type = test_df.type.astype(np.uint8), right?
test_df.ts = (test_df.ts/).astype(np.int32)\nThanks for your processed dataset!
\nYou can also use pd.Dataframe.from_records() to convert list of Dicts (session_data.events here) to a df, and then loop it for all sessions and concat them to get final events df.
\nAnyway I think it's more readable in your way to write the code.
Hi @radek1 ! Glad to see you here.
\nI was taking a look at the data and I have seen that for example session 0 in train data lasts from July 31 (ts β 1659304800025) to August 28 (ts β 1661684983707)
\nIn my concept of a session, lasting a month seems too much, I don't know what you think.
\nAnyway, when I can I will investigate it further and I will tell you something, I don't know if it is something common or not
Please take a look at the answer here or alternatively at the section that discusses this in my EDA.
\nThis seemed suspect to me as well but turns out it is just about the session definition used by the organizer π
", "votes": null, "replies": [] }, { "id": 2066697, "author_name": "wuwenmin", "author_url": "", "post_date": "12/16/2022 00:54:36", "content": "In this competition, the session is actually user
Sorry for the delayed response, unfortunately it is not easy to track new comments using the Kaggle interfaceβ¦
\nBut yes, you are right @wuwenmin, one can think of the session being a user id. It is important to keep in mind though that there are no overlapping sessions (or users) between test and train, so using the session id as a feature will note be very useful!
As I wrote in yours \"last 20 aids\": well-done, great contribution to Otto competition.
", "votes": null, "replies": [ { "id": 2016245, "author_name": "radek1", "author_url": "", "post_date": "11/03/2022 21:28:28", "content": "Thank you, @mpwolke! π
", "votes": null, "replies": [] } ] }, { "id": 2018373, "author_name": "nyamashina", "author_url": "", "post_date": "11/05/2022 16:19:33", "content": "Great work! upvoted!
", "votes": null, "replies": [ { "id": 2018688, "author_name": "radek1", "author_url": "", "post_date": "11/05/2022 22:32:22", "content": "Thank you, @nyamashina, very glad you are finding this useful! π
", "votes": null, "replies": [] } ] }, { "id": 2018761, "author_name": "iocmeta", "author_url": "", "post_date": "11/06/2022 01:18:27", "content": "thanks your notebook, i am a new kaggler, cay i ask you , why json to csv or parquet?
", "votes": null, "replies": [ { "id": 2018854, "author_name": "radek1", "author_url": "", "post_date": "11/06/2022 04:44:45", "content": "hey @iocmeta! csv or parquet is easier to read and process with the most common library that most people use, which is pandas π Generally, csv and parquet files are easier to use for ML work in most circumstances, I believe.
Hi @radek1 Thank you so much for sharing!
\nAs you said above to Kaggle does not provide enough RAM (30G in CPU) to process the data, therefore I won't have luck here to run your process_data.ipynb in my apple M1 8G RAM laptop.
\nAlthough I can't process the entire OTTO dataset with your process_data.ipynb, I should be able to run all of the code for experiment as long as I have the entire OTTO dataset stored in my local disk, right?
", "votes": null, "replies": [ { "id": 2023729, "author_name": "radek1", "author_url": "", "post_date": "11/10/2022 01:41:00", "content": "Yeah that is a bit of a problem I am afraid. 8GB is very little, you would have to go to extremely lengths to conserve memory.
\nI am not completely sure that it is worth it to figure out how to do stuff with that small amount of RAM. I mean, you still can, but you might be better off just running things on Kaggle. 30GB of RAM plus unlimited runtime for CPU only kernels sounds like a sweet deal π
\nOf course, the situation doesn't look that hot when it comes to GPUs, but even 40 hours/week can get you very far I feel.
\nEssentially -- I think your time might be better spent figuring out how to gain access to a machine with more RAM (be that Kaggle kernels or cloud compute) then trying to figure out how to work with 8GB of RAM (unless you want to learn something really low level like Rust or C, etc, which is super cool as well but maybe a bit orthogonal to maximizing your learning as a data science practitioner building up their portfolio π)
", "votes": null, "replies": [] }, { "id": 2024286, "author_name": "danielliao", "author_url": "", "post_date": "11/10/2022 11:54:11", "content": "Thank you so much Radek for the advices! Very helpful indeed!
\nI only saw this reply just now. I should turn on the notification earlier.
", "votes": null, "replies": [] } ] }, { "id": 2024279, "author_name": "danielliao", "author_url": "", "post_date": "11/10/2022 11:51:44", "content": "I tried to run process_data.ipynb to find out what the original json file look like. I love your codes, they are nice and simple to read and learn!
I also wanted to see how much ram does it take on Kaggle to convert test dataset to df. It seems 400MB jsonl takes nearly 4GB to do the processing.
\nHowever, when I reduced the chunksize to 100, it takes longer but the ram also reduced to 1.6 GB, which is 4 times larger than the test data size. It means 11 GB would require 44 GB ram on Kaggle to process.
So, I guess this is why the processing can not be done on kaggle on the training set.
", "votes": null, "replies": [] }, { "id": 2031088, "author_name": "leiwong", "author_url": "", "post_date": "11/15/2022 20:32:32", "content": "Great work! Updated!
", "votes": null, "replies": [ { "id": 2031091, "author_name": "radek1", "author_url": "", "post_date": "11/15/2022 20:33:57", "content": "Thanks @leiwong, appreciate it! π
", "votes": null, "replies": [] } ] }, { "id": 2031393, "author_name": "mattrosinski", "author_url": "", "post_date": "11/16/2022 03:47:39", "content": "Thanks so much for sharing this file Radek. Glad to see how much time I might expect to process the data on my machine!
", "votes": null, "replies": [ { "id": 2031397, "author_name": "radek1", "author_url": "", "post_date": "11/16/2022 03:50:11", "content": "Very glad you are finding this useful, @mattrosinski! π
", "votes": null, "replies": [] } ] }, { "id": 2065740, "author_name": "luyi564", "author_url": "", "post_date": "12/15/2022 03:11:01", "content": "Is this a Full data? Are you using full data running this notebook ?
", "votes": null, "replies": [ { "id": 2065755, "author_name": "radek1", "author_url": "", "post_date": "12/15/2022 03:39:57", "content": "Hey @luyi564! Yup, this is the full dataset
", "votes": null, "replies": [] } ] }, { "id": 2066696, "author_name": "wuwenmin", "author_url": "", "post_date": "12/16/2022 00:50:26", "content": "Hi @radek1 , thanks for sharing. Do you still use the dataset you shared here for offline training & validation?
", "votes": null, "replies": [ { "id": 2066851, "author_name": "radek1", "author_url": "", "post_date": "12/16/2022 05:57:25", "content": "Hey @wuwenmin! Yes, while working on the competition I was using the dataset that you mentioned.
", "votes": null, "replies": [] } ] }, { "id": 2069432, "author_name": "danielliao", "author_url": "", "post_date": "12/19/2022 01:28:59", "content": "Hi @radek1, I just recreated your optimized memory footprint dataset on Kaggle using polars within 30GB. Thank you for sharing and leading the way!
", "votes": null, "replies": [] }, { "id": 2090601, "author_name": "gpreda", "author_url": "", "post_date": "01/07/2023 14:01:19", "content": "Thank you for this work. It is very useful to the community.
", "votes": null, "replies": [ { "id": 2090985, "author_name": "radek1", "author_url": "", "post_date": "01/07/2023 22:07:12", "content": "@gpreda, thank you so much for your comments, much appreciated π It is a really nice feeling to have your work be appreciated and see that it is helping others.
\nThank you so much for taking the time to share this feedback with me! π€
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2015554": "Hey friends!\n\nI converted the dataset to live in `csv` & `parquet` files. You can find the dataset [here](https://www.kaggle.com/datasets/radek1/otto-full-optimized-memory-footprint).\n\nI have also created [an explainer notebook](https://www.kaggle.com/code/radek1/howto-full-dataset-as-parquet-csv-file). In it, I run you through how to use this data.\n\nI am also attaching the code I used to process this data as a notebook file to this post. I couldn't process the data on Kaggle due to RAM limitations so I ran it on my local machine and uploaded it to Kaggle.\n\nHope this can speed you along in your work π Happy kaggling! π₯³\n\n\n\n### Other resources you might find useful:\n\n* [π‘ [2 methods] How-to ensemble predictions π π π ](https://www.kaggle.com/code/radek1/2-methods-how-to-ensemble-predictions)\n* [local validation tracks public LB perfecty -- here is the setup](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364991)\n* [π‘ For my friends from Twitter and LinkedIn -- here is how to dive into this competition π³](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368560)\n* [Full dataset processed to CSV/parquet files with optimized memory footprint](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363843)\n* [co-visitation matrix - simplified, imprvd logic π₯](https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic)\n* [π‘ Word2Vec How-to [training and submission]πππ](https://www.kaggle.com/code/radek1/word2vec-how-to-training-and-submission)", "2015604": "This is the script I used for preprocessing as a screenshot\n\n", "2015683": "Hi @radek1 ! Glad to see you here.\n\nI was taking a look at the data and I have seen that for example **session 0** in train data lasts from **July 31 (ts β 1659304800025)** to **August 28 (ts β 1661684983707)**\n\nIn my concept of a session, lasting a month seems too much, I don't know what you think. \nAnyway, when I can I will investigate it further and I will tell you something, I don't know if it is something common or not", "2015687": "Please take a look at the answer [here](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363554#2015486) or alternatively at the section that discusses this [in my EDA](https://www.kaggle.com/code/radek1/eda-an-overview-of-the-full-dataset).\n\nThis seemed suspect to me as well but turns out it is just about the session definition used by the organizer π", "2016173": "As I wrote in yours \"last 20 aids\": well-done, great contribution to Otto competition.", "2016245": "Thank you, @mpwolke! π", "2018373": "Great work! upvoted!", "2018688": "Thank you, @nyamashina, very glad you are finding this useful! π", "2018761": "thanks your notebook, i am a new kaggler, cay i ask you , why json to csv or parquet?", "2018854": "hey @iocmeta! `csv` or `parquet` is easier to read and process with the most common library that most people use, which is `pandas` π Generally, `csv` and `parquet` files are easier to use for ML work in most circumstances, I believe.", "2023012": "Hi @radek1 Thank you so much for sharing! \n\nAs you said above to Kaggle does not provide enough RAM (30G in CPU) to process the data, therefore I won't have luck here to run your process_data.ipynb in my apple M1 8G RAM laptop. \n\nAlthough I can't process the entire OTTO dataset with your process_data.ipynb, I should be able to run all of the code for experiment as long as I have the entire OTTO dataset stored in my local disk, right?", "2023729": "Yeah that is a bit of a problem I am afraid. 8GB is very little, you would have to go to extremely lengths to conserve memory.\n\nI am not completely sure that it is worth it to figure out how to do stuff with that small amount of RAM. I mean, you still can, but you might be better off just running things on Kaggle. 30GB of RAM plus unlimited runtime for CPU only kernels sounds like a sweet deal π\n\nOf course, the situation doesn't look that hot when it comes to GPUs, but even 40 hours/week can get you very far I feel. \n\nEssentially -- I think your time might be better spent figuring out how to gain access to a machine with more RAM (be that Kaggle kernels or cloud compute) then trying to figure out how to work with 8GB of RAM (unless you want to learn something really low level like Rust or C, etc, which is super cool as well but maybe a bit orthogonal to maximizing your learning as a data science practitioner building up their portfolio π)", "2024279": "I tried to run `process_data.ipynb` to find out what the original json file look like. I love your codes, they are nice and simple to read and learn! \n\nI also wanted to see how much ram does it take on Kaggle to convert test dataset to df. It seems 400MB jsonl takes nearly 4GB to do the processing. \n\nHowever, when I reduced the `chunksize` to 100, it takes longer but the ram also reduced to 1.6 GB, which is 4 times larger than the test data size. It means 11 GB would require 44 GB ram on Kaggle to process. \n\nSo, I guess this is why the processing can not be done on kaggle on the training set.", "2024286": "Thank you so much Radek for the advices! Very helpful indeed!\n\nI only saw this reply just now. I should turn on the notification earlier.", "2031088": "Great work! Updated!", "2031091": "Thanks @leiwong, appreciate it! π", "2031393": "Thanks so much for sharing this file Radek. Glad to see how much time I might expect to process the data on my machine!", "2031397": "Very glad you are finding this useful, @mattrosinski! π", "2034437": "Hi @radek1, since you have updated the otto fully optimized dataset in parquet by reducing the RAM further by dividing `ts` with 1000, I guess the original process_data.ipynb is not updated.\n\nThe updated `process_data.ipynb` will add this line into the last cell above right under the line `test_df.type = test_df.type.astype(np.uint8)`, right? \n```python\ntest_df.ts = (test_df.ts/1000).astype(np.int32)\n```", "2065740": "Is this a Full data? Are you using full data running this notebook ?", "2065755": "Hey @luyi564! Yup, this is the full dataset", "2066696": "Hi @radek1 , thanks for sharing. Do you still use the dataset you shared [here](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364991) for offline training & validation?", "2066697": "In this competition, the `session` is actually `user`", "2066851": "Hey @wuwenmin! Yes, while working on the competition I was using the dataset that you mentioned.", "2069432": "Hi @radek1, I just [recreated](https://www.kaggle.com/code/danielliao/recreate-otto-full-optimized-memory-footprint/) your optimized memory footprint dataset on Kaggle using polars within 30GB. Thank you for sharing and leading the way!", "2090270": "Thanks for your processed dataset!\nYou can also use `pd.Dataframe.from_records()` to convert list of Dicts (`session_data.events` here) to a df, and then loop it for all sessions and concat them to get final events df.\nAnyway I think it's more readable in your way to write the code.", "2090601": "Thank you for this work. It is very useful to the community.", "2090984": "Sorry for the delayed response, unfortunately it is not easy to track new comments using the Kaggle interface...\n\nBut yes, you are right @wuwenmin, one can think of the `session` being a `user id`. It is important to keep in mind though that there are no overlapping `sessions` (or `users`) between test and train, so using the `session id` as a feature will note be very useful!", "2090985": "gpreda, thank you so much for your comments, much appreciated π It is a really nice feeling to have your work be appreciated and see that it is helping others.\n\nThank you so much for taking the time to share this feedback with me! π€" }, "source": "meta" }