![\"chunks_image\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2473952%2F1688baae242a52befb659da01ffbbb40%2Fchunk_image.png?generation=1679855369596237&alt=media\")
When using statistics such as the mean obtained by groupby session_id and level_group as features, it is not necessary to store the entire time series dataset in memory. Therefore, the policy is to split train.csv into smaller chunks and calculate statistics for each of these chunks. However, it must be handled in such a way that session_id is not split into multiple chunks.
The specific process is shown in my notebook. If you know of any good libraries, etc. that better optimize the process of splitting into chunks as it is done in my notebook, please let me know.
", "messages": [ { "id": "2198194", "postDate": "03/26/2023 18:40:39", "content": "When using statistics such as the mean obtained by groupby session_id and level_group as features, it is not necessary to store the entire time series dataset in memory. Therefore, the policy is to split train.csv into smaller chunks and calculate statistics for each of these chunks. However, it must be handled in such a way that session_id is not split into multiple chunks.
The specific process is shown in my notebook. If you know of any good libraries, etc. that better optimize the process of splitting into chunks as it is done in my notebook, please let me know.
", "rawMarkdown": "When using statistics such as the mean obtained by groupby `session_id` and `level_group` as features, it is not necessary to store the entire time series dataset in memory. Therefore, the policy is to split train.csv into smaller chunks and calculate statistics for each of these chunks. However, it must be handled in such a way that session_id is not split into multiple chunks.\n\n\n\nThe specific process is shown in [my notebook](https://www.kaggle.com/kentatanaka1data/the-training-dataset-into-small-chunks). If you know of any good libraries, etc. that better optimize the process of splitting into chunks as it is done in my notebook, please let me know.", "votes": null }, { "id": "2198232", "postDate": "03/26/2023 19:22:52", "content": "Thanks for sharing. I also read train data in chunks in my notebook here. The technique i use is to first read the entire session_id column and then determine where rows can safely be split (in code cell #2). Afterward (in code cell #7), i use nrows and skip_rows and read chunks manually.
Thank you for your comment. I have been referring to your notebook for a long time, but I did not know that you had added a new process to separate the chunks. I thought it is simple and looked good, but I didn't know where you defined reads and skips. I will read the code a little more closely.
My original notebook did not use chunks because the original competition was small enough. After Kaggle doubled the train data, i needed to add chunks to avoid memory error. The variables reads and skips are defined in the hidden code cell #2, so you'll need to click \"Show hidden code\" to see it.
I overlooked the hidden code in cell#2. Thanks for the additional explanation.
", "rawMarkdown": "I overlooked the hidden code in cell#2. Thanks for the additional explanation.", "votes": null }, { "id": "2198610", "postDate": "03/27/2023 07:20:38", "content": "I recommend using polars instead of pandas. It's much more memory efficient, and you don't even need to read the entire data from a CSV file.
Thank you for sharing the information, I will refer to the lazy load method to narrow down the target dataset using scan_csv, which is not possible with pandas.
Thanks for sharing your fabulous insights.
", "rawMarkdown": "Thanks for sharing your fabulous insights.", "votes": null }, { "id": "2199456", "postDate": "03/27/2023 18:06:41", "content": "The entire train can be downloaded in a 953 MB dataframe https://www.kaggle.com/code/vadimkamaev/reading-data-953-mb
", "rawMarkdown": "The entire train can be downloaded in a 953 MB dataframe https://www.kaggle.com/code/vadimkamaev/reading-data-953-mb", "votes": null }, { "id": "2199477", "postDate": "03/27/2023 18:21:50", "content": "Thanks for sharing the information. It is an important perspective to use memory efficiently by specifying the dtype of each column.
\nBy the way, the purpose of this issue is that I want to split up the training data. That is because I think it is useful when we have more training data or when we need more memory for feature engineering.
Thanks for sharing!! It is a very attractive code. I will refer to it.
", "rawMarkdown": "Thanks for sharing!! It is a very attractive code. I will refer to it.", "votes": null }, { "id": "2226976", "postDate": "04/19/2023 12:27:54", "content": "Thank you very much. However, please note that this code requires that the data in the resource file to be in the order of session_id, as in this competition's dataset. So it may be better to use other code that others have commented on.
", "rawMarkdown": "Thank you very much. However, please note that this code requires that the data in the resource file to be in the order of session_id, as in this competition's dataset. So it may be better to use other code that others have commented on.", "votes": null } ], "comments": [ { "id": 2198232, "author_name": "cdeotte", "author_url": "", "post_date": "03/26/2023 19:22:52", "content": "Thanks for sharing. I also read train data in chunks in my notebook here. The technique i use is to first read the entire session_id column and then determine where rows can safely be split (in code cell #2). Afterward (in code cell #7), i use nrows and skip_rows and read chunks manually.
Thank you for your comment. I have been referring to your notebook for a long time, but I did not know that you had added a new process to separate the chunks. I thought it is simple and looked good, but I didn't know where you defined reads and skips. I will read the code a little more closely.
My original notebook did not use chunks because the original competition was small enough. After Kaggle doubled the train data, i needed to add chunks to avoid memory error. The variables reads and skips are defined in the hidden code cell #2, so you'll need to click \"Show hidden code\" to see it.
I overlooked the hidden code in cell#2. Thanks for the additional explanation.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2198610, "author_name": "demche", "author_url": "", "post_date": "03/27/2023 07:20:38", "content": "I recommend using polars instead of pandas. It's much more memory efficient, and you don't even need to read the entire data from a CSV file.
Thank you for sharing the information, I will refer to the lazy load method to narrow down the target dataset using scan_csv, which is not possible with pandas.
Thanks for sharing your fabulous insights.
", "votes": null, "replies": [] }, { "id": 2199456, "author_name": "vadimkamaev", "author_url": "", "post_date": "03/27/2023 18:06:41", "content": "The entire train can be downloaded in a 953 MB dataframe https://www.kaggle.com/code/vadimkamaev/reading-data-953-mb
", "votes": null, "replies": [ { "id": 2199477, "author_name": "kentatanaka1data", "author_url": "", "post_date": "03/27/2023 18:21:50", "content": "Thanks for sharing the information. It is an important perspective to use memory efficiently by specifying the dtype of each column.
\nBy the way, the purpose of this issue is that I want to split up the training data. That is because I think it is useful when we have more training data or when we need more memory for feature engineering.
Thanks for sharing!! It is a very attractive code. I will refer to it.
", "votes": null, "replies": [ { "id": 2226976, "author_name": "kentatanaka1data", "author_url": "", "post_date": "04/19/2023 12:27:54", "content": "Thank you very much. However, please note that this code requires that the data in the resource file to be in the order of session_id, as in this competition's dataset. So it may be better to use other code that others have commented on.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2198194": "When using statistics such as the mean obtained by groupby `session_id` and `level_group` as features, it is not necessary to store the entire time series dataset in memory. Therefore, the policy is to split train.csv into smaller chunks and calculate statistics for each of these chunks. However, it must be handled in such a way that session_id is not split into multiple chunks.\n\n\n\nThe specific process is shown in [my notebook](https://www.kaggle.com/kentatanaka1data/the-training-dataset-into-small-chunks). If you know of any good libraries, etc. that better optimize the process of splitting into chunks as it is done in my notebook, please let me know.", "2198232": "Thanks for sharing. I also read train data in chunks in my notebook [here][1]. The technique i use is to first read the entire `session_id` column and then determine where rows can safely be split (in code cell #2). Afterward (in code cell #7), i use `nrows` and `skip_rows` and read chunks manually.\n\n[1]: https://www.kaggle.com/code/cdeotte/xgboost-baseline-0-680", "2198245": "Thank you for your comment. I have been referring to your notebook for a long time, but I did not know that you had added a new process to separate the chunks. I thought it is simple and looked good, but I didn't know where you defined `reads` and `skips`. I will read the code a little more closely.", "2198283": "My original notebook did not use chunks because the original competition was small enough. After Kaggle doubled the train data, i needed to add chunks to avoid memory error. The variables `reads` and `skips` are defined in the hidden `code cell #2`, so you'll need to click \"Show hidden code\" to see it.", "2198455": "I overlooked the hidden code in cell#2. Thanks for the additional explanation.", "2198610": "I recommend using `polars` instead of `pandas`. It's much more memory efficient, and you [don't even need](https://www.kaggle.com/code/demche/polars-memory-usage-optimization) to read the entire data from a CSV file.", "2198681": "Thank you for sharing the information, I will refer to the lazy load method to narrow down the target dataset using `scan_csv`, which is not possible with pandas.", "2198904": "Thanks for sharing your fabulous insights.", "2199456": "The entire train can be downloaded in a 953 MB dataframe https://www.kaggle.com/code/vadimkamaev/reading-data-953-mb", "2199477": "Thanks for sharing the information. It is an important perspective to use memory efficiently by specifying the dtype of each column.\nBy the way, the purpose of this issue is that I want to split up the training data. That is because I think it is useful when we have more training data or when we need more memory for feature engineering.", "2226955": "Thanks for sharing!! It is a very attractive code. I will refer to it.", "2226976": "Thank you very much. However, please note that this code requires that the data in the resource file to be in the order of session_id, as in this competition's dataset. So it may be better to use other code that others have commented on." }, "source": "meta" }