{ "id": 197686, "title": "How to reduce memory after using groupby", "url": "/competitions/riiid-test-answer-prediction/discussion/197686", "author_name": "Kaihua Zhang", "post_date": "2020-11-17T15:47:19.363000", "votes": 3, "comment_count": 6, "views": 0, "content": "

Hi, Guys.
\nI seem to face a problem of memory usage of the kaggle notebook when I try to reduce train data's memory:

\n
max_num = 258\ntrain = train.groupby(['user_id']).tail(258)\n
\n

Then the memory occupied by train data is smaller (from 3668M to 1621M).
\nHowever, the total memory become larger.(from 4493M to 6897M)

\n

So is this a matter of groupby?
\nand how can I free up this RAM ?

\n

\"\"

", "messages": [ { "id": 1082106, "postDate": "2020-11-17T15:47:19.363Z", "content": "

Hi, Guys.
\nI seem to face a problem of memory usage of the kaggle notebook when I try to reduce train data's memory:

\n
max_num = 258\ntrain = train.groupby(['user_id']).tail(258)\n
\n

Then the memory occupied by train data is smaller (from 3668M to 1621M).
\nHowever, the total memory become larger.(from 4493M to 6897M)

\n

So is this a matter of groupby?
\nand how can I free up this RAM ?

\n

\"\"

", "rawMarkdown": "Hi, Guys. \nI seem to face a problem of memory usage of the kaggle notebook when I try to reduce train data's memory:\n```python\nmax_num = 258\ntrain = train.groupby(['user_id']).tail(258)\n```\nThen the memory occupied by train data is smaller (from 3668M to 1621M).\nHowever, the total memory become larger.(from 4493M to 6897M)\n\nSo is this a matter of groupby?\nand how can I free up this RAM ?\n\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2266069%2F2dd68f2254add5da1e5842796a27448c%2F_20202317112306.png?generation=1605626798476773&alt=media)", "votes": 3 }, { "id": 1082179, "postDate": "2020-11-17T17:05:49.010Z", "content": "

Try clearing up memory manually. Here's the function that I use:

\n
import gc\ndef clear_mem():\n    %reset -f out\n    %reset -f in\n    gc.collect()\n
", "rawMarkdown": "Try clearing up memory manually. Here's the function that I use:\n```\nimport gc\ndef clear_mem():\n %reset -f out\n %reset -f in\n gc.collect()\n```", "votes": 4, "replies": [ { "id": 1082570, "postDate": "2020-11-18T03:02:03.780Z", "content": "

Thank you for your reply!
\nBut it's still the same.
\nI've found similar problems(RAM usage in a kaggle notebook from kaggle and How to Deallocate memory from an object in Jupyter Notebook from stackoverflow)
\nI'm looking for another method or give up😪

", "rawMarkdown": "Thank you for your reply!\nBut it's still the same.\nI've found similar problems([RAM usage in a kaggle notebook from kaggle](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191463) and [How to Deallocate memory from an object in Jupyter Notebook from stackoverflow](https://stackoverflow.com/questions/55221451/how-to-deallocate-memory-from-an-object-in-jupyter-notebook))\nI'm looking for another method or give up😪" } ] }, { "id": 1084394, "postDate": "2020-11-20T02:10:26.357Z", "content": "

There is two option to solve this problem.
\n1-) Use chunks to groupby objects.
\n2-) Calculate them and use them after as a constant value by merging to Original DataFrame.

", "rawMarkdown": "There is two option to solve this problem.\n1-) Use chunks to groupby objects.\n2-) Calculate them and use them after as a constant value by merging to Original DataFrame." }, { "id": 1083647, "postDate": "2020-11-19T07:45:05.920Z", "content": "

How do you check your used memory ?
\nWhen I run your code snippet and check with psutil, the memory consumption is reduced as expected.

", "rawMarkdown": "How do you check your used memory ?\nWhen I run your code snippet and check with psutil, the memory consumption is reduced as expected." }, { "id": 1082801, "postDate": "2020-11-18T08:34:34.760Z", "content": "

try creating a function, return your reduce train
\nmaybe useful

", "rawMarkdown": "try creating a function, return your reduce train \nmaybe useful" }, { "id": 1082178, "postDate": "2020-11-17T17:05:38.663Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1082179, "author_name": "Arsal", "author_url": "", "post_date": "2020-11-17T17:05:49.010000", "content": "

Try clearing up memory manually. Here's the function that I use:

\n
import gc\ndef clear_mem():\n    %reset -f out\n    %reset -f in\n    gc.collect()\n
", "votes": 4, "replies": [ { "id": 1082570, "author_name": "Kaihua Zhang", "author_url": "", "post_date": "2020-11-18T03:02:03.780000", "content": "

Thank you for your reply!
\nBut it's still the same.
\nI've found similar problems(RAM usage in a kaggle notebook from kaggle and How to Deallocate memory from an object in Jupyter Notebook from stackoverflow)
\nI'm looking for another method or give up😪

", "votes": 0, "replies": [] } ] }, { "id": 1084394, "author_name": "Kemal Emre Çolak", "author_url": "", "post_date": "2020-11-20T02:10:26.357000", "content": "

There is two option to solve this problem.
\n1-) Use chunks to groupby objects.
\n2-) Calculate them and use them after as a constant value by merging to Original DataFrame.

", "votes": 0, "replies": [] }, { "id": 1083647, "author_name": "Agrica", "author_url": "", "post_date": "2020-11-19T07:45:05.920000", "content": "

How do you check your used memory ?
\nWhen I run your code snippet and check with psutil, the memory consumption is reduced as expected.

", "votes": 0, "replies": [] }, { "id": 1082801, "author_name": "qiaqia", "author_url": "", "post_date": "2020-11-18T08:34:34.760000", "content": "

try creating a function, return your reduce train
\nmaybe useful

", "votes": 0, "replies": [] }, { "id": 1082178, "author_name": "", "author_url": "", "post_date": "2020-11-17T17:05:38.663000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1082106": "Hi, Guys. \nI seem to face a problem of memory usage of the kaggle notebook when I try to reduce train data's memory:\n```python\nmax_num = 258\ntrain = train.groupby(['user_id']).tail(258)\n```\nThen the memory occupied by train data is smaller (from 3668M to 1621M).\nHowever, the total memory become larger.(from 4493M to 6897M)\n\nSo is this a matter of groupby?\nand how can I free up this RAM ?\n\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2266069%2F2dd68f2254add5da1e5842796a27448c%2F_20202317112306.png?generation=1605626798476773&alt=media)", "1082179": "Try clearing up memory manually. Here's the function that I use:\n```\nimport gc\ndef clear_mem():\n %reset -f out\n %reset -f in\n gc.collect()\n```", "1084394": "There is two option to solve this problem.\n1-) Use chunks to groupby objects.\n2-) Calculate them and use them after as a constant value by merging to Original DataFrame.", "1083647": "How do you check your used memory ?\nWhen I run your code snippet and check with psutil, the memory consumption is reduced as expected.", "1082801": "try creating a function, return your reduce train \nmaybe useful", "1082178": "" } }