I'm working on this competition using Kaggle notebbok.
\nData saved in parquet format is converted DataFrame and use it.
\nHowever, when trying to add a new columns, the session is cut off due to lack of memory.
How do you use your data?
\nIs the only way to split the data into several pieces?
I only want it to run on the Kaggle notebook.
\nPlease give me some good way!
I'm working on this competition using Kaggle notebbok.
\nData saved in parquet format is converted DataFrame and use it.
\nHowever, when trying to add a new columns, the session is cut off due to lack of memory.
How do you use your data?
\nIs the only way to split the data into several pieces?
I only want it to run on the Kaggle notebook.
\nPlease give me some good way!
If you don't want to split you can try to use polars.
", "rawMarkdown": "If you don't want to split you can try to use polars.", "votes": null }, { "id": "2072579", "postDate": "12/22/2022 08:34:59", "content": "Thank you @simonveitner !
\nUpon investigation, I learned that polars is faster than pandas.
\nCan polars also use less memory??
Consider working in chunks. Break the dataframe into parts where all of one customer is contained within the same part. Then work on a part and write result to disk. Then clear memory and work on next part.
\nAlso reduce each column to least data usage. For example convert ts column into seconds with train['ts'] = (train['ts']//1000).astype('int32'). Make sure each of your columns has least dtype possible.
More tips are here in section titled Memory Management.
", "rawMarkdown": "Consider working in chunks. Break the dataframe into parts where all of one customer is contained within the same part. Then work on a part and write result to disk. Then clear memory and work on next part.\n\nAlso reduce each column to least data usage. For example convert `ts` column into seconds with `train['ts'] = (train['ts']//1000).astype('int32')`. Make sure each of your columns has least `dtype` possible.\n\nMore tips are [here][1] in section titled Memory Management.\n\n[1]: https://www.kaggle.com/competitions/otto-recommender-system/discussion/370210", "votes": null }, { "id": "2072909", "postDate": "12/22/2022 14:08:38", "content": "Thank you, @cdeotte !
\nI'll keep an eye on your notebook and try to work on splitting up the data.
I'll read your memory management article!
\nYou're very helpful, thank you so much!!!!!
", "rawMarkdown": "Thank you, @cdeotte !\nI'll keep an eye on your notebook and try to work on splitting up the data.\n\nI'll read your memory management article!\n\nYou're very helpful, thank you so much!!!!!", "votes": null }, { "id": "2072926", "postDate": "12/22/2022 14:25:09", "content": "Yes. But I suggest to use cudf as it's fester.
", "rawMarkdown": "Yes. But I suggest to use cudf as it's fester.", "votes": null }, { "id": "2072966", "postDate": "12/22/2022 14:52:00", "content": "Thank you!
\nI'll examine about cudf!
Memory optimization as explained by @cdeotte in a notebook from another competition can go a really long way!
\nIt is a trick every Kaggler should have under their toolbelt, IMO 🙂
", "rawMarkdown": "Memory optimization as explained by @cdeotte [in a notebook from another competition](https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/308635) can go a really long way!\n\nIt is a trick every Kaggler should have under their toolbelt, IMO 🙂", "votes": null } ], "comments": [ { "id": 2072565, "author_name": "simonveitner", "author_url": "", "post_date": "12/22/2022 08:21:01", "content": "If you don't want to split you can try to use polars.
", "votes": null, "replies": [ { "id": 2072579, "author_name": "takuma0306", "author_url": "", "post_date": "12/22/2022 08:34:59", "content": "Thank you @simonveitner !
\nUpon investigation, I learned that polars is faster than pandas.
\nCan polars also use less memory??
Yes. But I suggest to use cudf as it's fester.
", "votes": null, "replies": [ { "id": 2072966, "author_name": "takuma0306", "author_url": "", "post_date": "12/22/2022 14:52:00", "content": "Thank you!
\nI'll examine about cudf!
Consider working in chunks. Break the dataframe into parts where all of one customer is contained within the same part. Then work on a part and write result to disk. Then clear memory and work on next part.
\nAlso reduce each column to least data usage. For example convert ts column into seconds with train['ts'] = (train['ts']//1000).astype('int32'). Make sure each of your columns has least dtype possible.
More tips are here in section titled Memory Management.
", "votes": null, "replies": [ { "id": 2072909, "author_name": "takuma0306", "author_url": "", "post_date": "12/22/2022 14:08:38", "content": "Thank you, @cdeotte !
\nI'll keep an eye on your notebook and try to work on splitting up the data.
I'll read your memory management article!
\nYou're very helpful, thank you so much!!!!!
", "votes": null, "replies": [] } ] }, { "id": 2079064, "author_name": "radek1", "author_url": "", "post_date": "12/28/2022 23:35:28", "content": "Memory optimization as explained by @cdeotte in a notebook from another competition can go a really long way!
\nIt is a trick every Kaggler should have under their toolbelt, IMO 🙂
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2072563": "I'm working on this competition using Kaggle notebbok.\nData saved in parquet format is converted DataFrame and use it.\nHowever, when trying to add a new columns, the session is cut off due to lack of memory.\n\nHow do you use your data?\nIs the only way to split the data into several pieces?\n\nI only want it to run on the Kaggle notebook.\nPlease give me some good way!", "2072565": "If you don't want to split you can try to use polars.", "2072579": "Thank you @simonveitner !\nUpon investigation, I learned that polars is faster than pandas.\nCan polars also use less memory??", "2072898": "Consider working in chunks. Break the dataframe into parts where all of one customer is contained within the same part. Then work on a part and write result to disk. Then clear memory and work on next part.\n\nAlso reduce each column to least data usage. For example convert `ts` column into seconds with `train['ts'] = (train['ts']//1000).astype('int32')`. Make sure each of your columns has least `dtype` possible.\n\nMore tips are [here][1] in section titled Memory Management.\n\n[1]: https://www.kaggle.com/competitions/otto-recommender-system/discussion/370210", "2072909": "Thank you, @cdeotte !\nI'll keep an eye on your notebook and try to work on splitting up the data.\n\nI'll read your memory management article!\n\nYou're very helpful, thank you so much!!!!!", "2072926": "Yes. But I suggest to use cudf as it's fester.", "2072966": "Thank you!\nI'll examine about cudf!", "2079064": "Memory optimization as explained by @cdeotte [in a notebook from another competition](https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/308635) can go a really long way!\n\nIt is a trick every Kaggler should have under their toolbelt, IMO 🙂" }, "source": "meta" }