![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1443335%2Fe9a911587440fcddcfde467b1196e0b0%2Freset.png?generation=1607504897856996&alt=media\")
Hi there,
\nAre you facing Out of Memory issues while running Kernels? Are you able to debug them? Many people have pointed out that this competition is as much as about Model Building as of Software Engineering. Here are two things I use to solve memory issues for my Kernels.
\nSummary Tracker helps us to find out the objects with their memory usage. We can use sorting to find the top 10 objects using the code below.
\nfrom operator import itemgetter\nfrom pympler import tracker\n\nmem = tracker.SummaryTracker()\nprint(sorted(mem.create_summary(), reverse=True, key=itemgetter(2))[:10])\nThis way we know what is causing the issue. If you facing memory overload, try to do profiling step by step and understand the bottleneck.
\nIt's always about Time vs Memory Tradeoff. Since we have 9 hours (GPU usage also) to submit our predictions, it should be plenty of time to try techniques that reduce the memory overload unless you are doing model building and inference at the same time or using the test data to retrain your models periodically during the inference process. The following tips might help you:
\nget_referrers to find all the pointers to the object and try to reduce them.import gc\ngc.get_referrers(object)\nHave other ideas which help, please let me know.
", "messages": [ { "id": "1102568", "postDate": "12/05/2020 04:23:35", "content": "Hi there,
\nAre you facing Out of Memory issues while running Kernels? Are you able to debug them? Many people have pointed out that this competition is as much as about Model Building as of Software Engineering. Here are two things I use to solve memory issues for my Kernels.
\nSummary Tracker helps us to find out the objects with their memory usage. We can use sorting to find the top 10 objects using the code below.
\nfrom operator import itemgetter\nfrom pympler import tracker\n\nmem = tracker.SummaryTracker()\nprint(sorted(mem.create_summary(), reverse=True, key=itemgetter(2))[:10])\nThis way we know what is causing the issue. If you facing memory overload, try to do profiling step by step and understand the bottleneck.
\nIt's always about Time vs Memory Tradeoff. Since we have 9 hours (GPU usage also) to submit our predictions, it should be plenty of time to try techniques that reduce the memory overload unless you are doing model building and inference at the same time or using the test data to retrain your models periodically during the inference process. The following tips might help you:
\nget_referrers to find all the pointers to the object and try to reduce them.import gc\ngc.get_referrers(object)\nHave other ideas which help, please let me know.
", "rawMarkdown": "Hi there,\n\nAre you facing Out of Memory issues while running Kernels? Are you able to debug them? Many people have pointed out that this competition is as much as about Model Building as of Software Engineering. Here are two things I use to solve memory issues for my Kernels. \n\n## Summary Tracker\nSummary Tracker helps us to find out the objects with their memory usage. We can use sorting to find the top 10 objects using the code below.\n\n```python\nfrom operator import itemgetter\nfrom pympler import tracker\n\nmem = tracker.SummaryTracker()\nprint(sorted(mem.create_summary(), reverse=True, key=itemgetter(2))[:10])\n```\n\nThis way we know what is causing the issue. If you facing memory overload, try to do profiling step by step and understand the bottleneck.\n\n## Time vs Memory Tradeoff - Save Data to Save Memory\nIt's always about Time vs Memory Tradeoff. Since we have 9 hours (GPU usage also) to submit our predictions, it should be plenty of time to try techniques that reduce the memory overload unless you are doing model building and inference at the same time or using the test data to retrain your models periodically during the inference process. The following tips might help you:\n- Save RAM by saving that data that you don't need to complete the current step which is causing OOM issues. Ex: If merging Train and Questions DataFrames is causing the issue, then remove the other objects (Ex: You don't need to do feature engineering for the train and valid data at the same time, consider saving train and valid to memory and reloading it after the bottleneck is over). But remember that only 19.6GB of disk space is available at a time.\n- Use Garbage Collection. If it is not removing memory from your RAM, then you need to look at why some objects are not getting removed. Python garbage collection works by removing all the objects which don't have any references. Use `get_referrers` to find all the pointers to the object and try to reduce them.\n``` \nimport gc\ngc.get_referrers(object)\n```\n- Understand how Pandas works. Try to do in place operations and reduce creating copies of DataFrames when not required (Understand the concept of view vs copy in Pandas. It is confusing at times but look out for the memory usage, sometimes it helps).\n- Use %reset to reset ram in Jupyter notebooks. This frees up RAM (if -f is used). You can save the data to a hard disk (either to /kaggle/working/ or /kaggle/temp/) and reload the data again. Thanks, @erikbruin for suggesting this.\n\nHave other ideas which help, please let me know.", "votes": null }, { "id": "1102607", "postDate": "12/05/2020 05:37:45", "content": "Thank you for the insights!
", "rawMarkdown": "Thank you for the insights!", "votes": null }, { "id": "1103267", "postDate": "12/05/2020 19:17:14", "content": "Great tips, @manikanthr5! Many thanks 😃
", "rawMarkdown": "Great tips, @manikanthr5! Many thanks 😃", "votes": null }, { "id": "1106978", "postDate": "12/09/2020 09:11:03", "content": "There are also Jupyter notebook specific issues. I have had cases where the object is deleted from the namespace, but memory did not seem to be released back (even after specifically calling gc.collect). These things happen when you print things to your screen (such as a simple head()). According to the documentation, we can remove those by calling %reset out, but it seems as if this does not always clear all references to the object (RAM does not go down while monitoring the RAM usage while running the notebook).
\n
By the way, clearing everything in RAM with %reset -f always works (I have used that in my EDA).
", "rawMarkdown": "There are also Jupyter notebook specific issues. I have had cases where the object is deleted from the namespace, but memory did not seem to be released back (even after specifically calling gc.collect). These things happen when you print things to your screen (such as a simple head()). According to the documentation, we can remove those by calling %reset out, but it seems as if this does not always clear all references to the object (RAM does not go down while monitoring the RAM usage while running the notebook).\n\n\n\nBy the way, clearing everything in RAM with %reset -f always works (I have used that in my EDA).", "votes": null }, { "id": "1107019", "postDate": "12/09/2020 09:48:03", "content": "Yes. After looking at your EDA notebook, I started to regularly use %reset command in my notebooks. This is really useful to know.
", "rawMarkdown": "Yes. After looking at your EDA notebook, I started to regularly use %reset command in my notebooks. This is really useful to know.", "votes": null }, { "id": "1112240", "postDate": "12/14/2020 12:23:32", "content": "I had similar problems trying to read large parquet files for the Bengali AI competition. Here is my notebook showing techniques for memory profiling:
\nhttps://www.kaggle.com/jamesmcguigan/reading-parquet-files-ram-cpu-optimization
", "rawMarkdown": "I had similar problems trying to read large parquet files for the Bengali AI competition. Here is my notebook showing techniques for memory profiling:\n\nhttps://www.kaggle.com/jamesmcguigan/reading-parquet-files-ram-cpu-optimization", "votes": null } ], "comments": [ { "id": 1102607, "author_name": "yolandakaggle", "author_url": "", "post_date": "12/05/2020 05:37:45", "content": "Thank you for the insights!
", "votes": null, "replies": [] }, { "id": 1103267, "author_name": "pedrocouto39", "author_url": "", "post_date": "12/05/2020 19:17:14", "content": "Great tips, @manikanthr5! Many thanks 😃
", "votes": null, "replies": [] }, { "id": 1106978, "author_name": "erikbruin", "author_url": "", "post_date": "12/09/2020 09:11:03", "content": "There are also Jupyter notebook specific issues. I have had cases where the object is deleted from the namespace, but memory did not seem to be released back (even after specifically calling gc.collect). These things happen when you print things to your screen (such as a simple head()). According to the documentation, we can remove those by calling %reset out, but it seems as if this does not always clear all references to the object (RAM does not go down while monitoring the RAM usage while running the notebook).
\n
By the way, clearing everything in RAM with %reset -f always works (I have used that in my EDA).
", "votes": null, "replies": [ { "id": 1107019, "author_name": "manikanthr5", "author_url": "", "post_date": "12/09/2020 09:48:03", "content": "Yes. After looking at your EDA notebook, I started to regularly use %reset command in my notebooks. This is really useful to know.
", "votes": null, "replies": [] } ] }, { "id": 1112240, "author_name": "jamesmcguigan", "author_url": "", "post_date": "12/14/2020 12:23:32", "content": "I had similar problems trying to read large parquet files for the Bengali AI competition. Here is my notebook showing techniques for memory profiling:
\nhttps://www.kaggle.com/jamesmcguigan/reading-parquet-files-ram-cpu-optimization
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1102568": "Hi there,\n\nAre you facing Out of Memory issues while running Kernels? Are you able to debug them? Many people have pointed out that this competition is as much as about Model Building as of Software Engineering. Here are two things I use to solve memory issues for my Kernels. \n\n## Summary Tracker\nSummary Tracker helps us to find out the objects with their memory usage. We can use sorting to find the top 10 objects using the code below.\n\n```python\nfrom operator import itemgetter\nfrom pympler import tracker\n\nmem = tracker.SummaryTracker()\nprint(sorted(mem.create_summary(), reverse=True, key=itemgetter(2))[:10])\n```\n\nThis way we know what is causing the issue. If you facing memory overload, try to do profiling step by step and understand the bottleneck.\n\n## Time vs Memory Tradeoff - Save Data to Save Memory\nIt's always about Time vs Memory Tradeoff. Since we have 9 hours (GPU usage also) to submit our predictions, it should be plenty of time to try techniques that reduce the memory overload unless you are doing model building and inference at the same time or using the test data to retrain your models periodically during the inference process. The following tips might help you:\n- Save RAM by saving that data that you don't need to complete the current step which is causing OOM issues. Ex: If merging Train and Questions DataFrames is causing the issue, then remove the other objects (Ex: You don't need to do feature engineering for the train and valid data at the same time, consider saving train and valid to memory and reloading it after the bottleneck is over). But remember that only 19.6GB of disk space is available at a time.\n- Use Garbage Collection. If it is not removing memory from your RAM, then you need to look at why some objects are not getting removed. Python garbage collection works by removing all the objects which don't have any references. Use `get_referrers` to find all the pointers to the object and try to reduce them.\n``` \nimport gc\ngc.get_referrers(object)\n```\n- Understand how Pandas works. Try to do in place operations and reduce creating copies of DataFrames when not required (Understand the concept of view vs copy in Pandas. It is confusing at times but look out for the memory usage, sometimes it helps).\n- Use %reset to reset ram in Jupyter notebooks. This frees up RAM (if -f is used). You can save the data to a hard disk (either to /kaggle/working/ or /kaggle/temp/) and reload the data again. Thanks, @erikbruin for suggesting this.\n\nHave other ideas which help, please let me know.", "1102607": "Thank you for the insights!", "1103267": "Great tips, @manikanthr5! Many thanks 😃", "1106978": "There are also Jupyter notebook specific issues. I have had cases where the object is deleted from the namespace, but memory did not seem to be released back (even after specifically calling gc.collect). These things happen when you print things to your screen (such as a simple head()). According to the documentation, we can remove those by calling %reset out, but it seems as if this does not always clear all references to the object (RAM does not go down while monitoring the RAM usage while running the notebook).\n\n\n\nBy the way, clearing everything in RAM with %reset -f always works (I have used that in my EDA).", "1107019": "Yes. After looking at your EDA notebook, I started to regularly use %reset command in my notebooks. This is really useful to know.", "1112240": "I had similar problems trying to read large parquet files for the Bengali AI competition. Here is my notebook showing techniques for memory profiling:\n\nhttps://www.kaggle.com/jamesmcguigan/reading-parquet-files-ram-cpu-optimization" }, "source": "meta" }