{
  "id": 505491,
  "title": "[MEMORY MANAGEMENT] Function to clear max RAM and cache",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/505491",
  "author_name": "",
  "post_date": "2024-05-17T18:35:14.066758100Z",
  "votes": 22,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Most of you must be using the below two methods to clear RAM and cache:</p>\n<pre><code> ():\n    gc.collect()\n    libc = ctypes.CDLL()\n    libc.malloc_trim()\n</code></pre>\n<p>However, sometimes after processing multiple data chunks you still must see that the RAM used is not equivalent to dataframes and variables you have in memory. That is because there are some global variables stored in the process that are still in the memory. You can clear them using the code below.</p>\n<pre><code>all_vars = (().keys())\nvariables_to_keep = [, , ,,]\nvariables_to_delete = [var  var  all_vars  var   variables_to_keep]\n var  variables_to_delete:\n     ()[var]\n</code></pre>\n<p>keep all the dataframe names and variable names that you want to use and this will drop all other global variables. Also the variable '_oh' is used for some jupyter notebook functionalies like display etc so you should keep it. This helped me a lot to clear everything unnecessary before starting training process. Hope it helps someone.</p>",
  "messages": [
    {
      "id": "2820862",
      "postDate": "05/17/2024 18:35:14",
      "content": "<p>Most of you must be using the below two methods to clear RAM and cache:</p>\n<pre><code> ():\n    gc.collect()\n    libc = ctypes.CDLL()\n    libc.malloc_trim()\n</code></pre>\n<p>However, sometimes after processing multiple data chunks you still must see that the RAM used is not equivalent to dataframes and variables you have in memory. That is because there are some global variables stored in the process that are still in the memory. You can clear them using the code below.</p>\n<pre><code>all_vars = (().keys())\nvariables_to_keep = [, , ,,]\nvariables_to_delete = [var  var  all_vars  var   variables_to_keep]\n var  variables_to_delete:\n     ()[var]\n</code></pre>\n<p>keep all the dataframe names and variable names that you want to use and this will drop all other global variables. Also the variable '_oh' is used for some jupyter notebook functionalies like display etc so you should keep it. This helped me a lot to clear everything unnecessary before starting training process. Hope it helps someone.</p>",
      "rawMarkdown": "Most of you must be using the below two methods to clear RAM and cache:\n\n```python\ndef free_cache_and_ram():\n    gc.collect()\n    libc = ctypes.CDLL(\"libc.so.6\")\n    libc.malloc_trim(0)\n```\n\nHowever, sometimes after processing multiple data chunks you still must see that the RAM used is not equivalent to dataframes and variables you have in memory. That is because there are some global variables stored in the process that are still in the memory. You can clear them using the code below.\n\n```python\nall_vars = list(globals().keys())\nvariables_to_keep = ['train_x', 'test_x', 'features','str_cols','_oh']\nvariables_to_delete = [var for var in all_vars if var not in variables_to_keep]\nfor var in variables_to_delete:\n    del globals()[var]\n```\n\nkeep all the dataframe names and variable names that you want to use and this will drop all other global variables. Also the variable '_oh' is used for some jupyter notebook functionalies like display etc so you should keep it. This helped me a lot to clear everything unnecessary before starting training process. Hope it helps someone.",
      "votes": null
    },
    {
      "id": "2824992",
      "postDate": "05/20/2024 05:30:37",
      "content": "<p>interesting! Ill try your 2nd code snippet. Thank you</p>",
      "rawMarkdown": "interesting! Ill try your 2nd code snippet. Thank you",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2824992,
      "author_name": "bwandowando",
      "author_url": "",
      "post_date": "05/20/2024 05:30:37",
      "content": "<p>interesting! Ill try your 2nd code snippet. Thank you</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2820862": "Most of you must be using the below two methods to clear RAM and cache:\n\n```python\ndef free_cache_and_ram():\n    gc.collect()\n    libc = ctypes.CDLL(\"libc.so.6\")\n    libc.malloc_trim(0)\n```\n\nHowever, sometimes after processing multiple data chunks you still must see that the RAM used is not equivalent to dataframes and variables you have in memory. That is because there are some global variables stored in the process that are still in the memory. You can clear them using the code below.\n\n```python\nall_vars = list(globals().keys())\nvariables_to_keep = ['train_x', 'test_x', 'features','str_cols','_oh']\nvariables_to_delete = [var for var in all_vars if var not in variables_to_keep]\nfor var in variables_to_delete:\n    del globals()[var]\n```\n\nkeep all the dataframe names and variable names that you want to use and this will drop all other global variables. Also the variable '_oh' is used for some jupyter notebook functionalies like display etc so you should keep it. This helped me a lot to clear everything unnecessary before starting training process. Hope it helps someone.",
    "2824992": "interesting! Ill try your 2nd code snippet. Thank you"
  },
  "source": "meta"
}