{
  "id": 498780,
  "title": "Clearing loaded parquet file from CPU ram?",
  "url": "/competitions/uspto-explainable-ai/discussion/498780",
  "author_name": "",
  "post_date": "2024-04-29T15:10:10.834228300Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>How to clear loaded <strong><code>patent_data{year_month}</code></strong> file from CPU memory to reduce RAM usage? because of examining multiple pub_num, we have to load their respective year's parquet file, and after examining more than 4 patents, RAM  explodes. I tried <strong><code>del</code></strong> and <strong><code>gc.collect()</code></strong> but didn't work!</p>",
  "messages": [
    {
      "id": "2783087",
      "postDate": "04/29/2024 15:10:10",
      "content": "<p>How to clear loaded <strong><code>patent_data{year_month}</code></strong> file from CPU memory to reduce RAM usage? because of examining multiple pub_num, we have to load their respective year's parquet file, and after examining more than 4 patents, RAM  explodes. I tried <strong><code>del</code></strong> and <strong><code>gc.collect()</code></strong> but didn't work!</p>",
      "rawMarkdown": "How to clear loaded **`patent_data{year_month}`** file from CPU memory to reduce RAM usage? because of examining multiple pub_num, we have to load their respective year's parquet file, and after examining more than 4 patents, RAM  explodes. I tried **`del `** and **`gc.collect()`** but didn't work!",
      "votes": null
    },
    {
      "id": "2785189",
      "postDate": "04/30/2024 16:55:43",
      "content": "<p>for me, weirdly, gc.collect() only cleared memory after my for-loop was done (suddenly ram dropped from 25GB to 11GB).<br>\nEnclosing loading and definition into a function also didnt work. What helped a lot is only loading the needed columns (e.g., only title) - most of the memory foot print is in the abstract and mostly in the description.</p>",
      "rawMarkdown": "for me, weirdly, gc.collect() only cleared memory after my for-loop was done (suddenly ram dropped from 25GB to 11GB).\nEnclosing loading and definition into a function also didnt work. What helped a lot is only loading the needed columns (e.g., only title) - most of the memory foot print is in the abstract and mostly in the description.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2785189,
      "author_name": "valentinwerner",
      "author_url": "",
      "post_date": "04/30/2024 16:55:43",
      "content": "<p>for me, weirdly, gc.collect() only cleared memory after my for-loop was done (suddenly ram dropped from 25GB to 11GB).<br>\nEnclosing loading and definition into a function also didnt work. What helped a lot is only loading the needed columns (e.g., only title) - most of the memory foot print is in the abstract and mostly in the description.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2783087": "How to clear loaded **`patent_data{year_month}`** file from CPU memory to reduce RAM usage? because of examining multiple pub_num, we have to load their respective year's parquet file, and after examining more than 4 patents, RAM  explodes. I tried **`del `** and **`gc.collect()`** but didn't work!",
    "2785189": "for me, weirdly, gc.collect() only cleared memory after my for-loop was done (suddenly ram dropped from 25GB to 11GB).\nEnclosing loading and definition into a function also didnt work. What helped a lot is only loading the needed columns (e.g., only title) - most of the memory foot print is in the abstract and mostly in the description."
  },
  "source": "meta"
}