{
  "id": 586989,
  "title": "Memory management - how to effectively manage RAM memory",
  "url": "/competitions/drw-crypto-market-prediction/discussion/586989",
  "author_name": "",
  "post_date": "2025-06-29T10:38:30.369574800Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>In this competition memory management could be a issue and you can easily get out of memory.<br>\nWhat is the best way to manage RAM memory? Is it better to delete a dataframe no longer used and use garbage collector to free RAM memory or the best way is to save the dataframe to disk? Thank you</p>",
  "messages": [
    {
      "id": "3235515",
      "postDate": "06/29/2025 10:38:30",
      "content": "<p>In this competition memory management could be a issue and you can easily get out of memory.<br>\nWhat is the best way to manage RAM memory? Is it better to delete a dataframe no longer used and use garbage collector to free RAM memory or the best way is to save the dataframe to disk? Thank you</p>",
      "rawMarkdown": "In this competition memory management could be a issue and you can easily get out of memory.\nWhat is the best way to manage RAM memory? Is it better to delete a dataframe no longer used and use garbage collector to free RAM memory or the best way is to save the dataframe to disk? Thank you",
      "votes": null
    },
    {
      "id": "3235531",
      "postDate": "06/29/2025 10:58:18",
      "content": "<p>Use a bigger PC with good GPU and RAM. 128 GB RAM is more than enough for this assignment <a href=\"https://www.kaggle.com/obiaf88\" target=\"_blank\">@obiaf88</a> </p>",
      "rawMarkdown": "Use a bigger PC with good GPU and RAM. 128 GB RAM is more than enough for this assignment @obiaf88",
      "votes": null
    },
    {
      "id": "3235548",
      "postDate": "06/29/2025 11:23:02",
      "content": "<p>Thank you <a href=\"https://www.kaggle.com/ravi20076\" target=\"_blank\">@ravi20076</a> ; I'm using kaggle notebook with CPU and 30 GB RAM; I will read the data in chunks; with this kind of competitions it's better to use a PC with GPU and much RAM than kaggle notebooks? Thank you again.</p>",
      "rawMarkdown": "Thank you @ravi20076 ; I'm using kaggle notebook with CPU and 30 GB RAM; I will read the data in chunks; with this kind of competitions it's better to use a PC with GPU and much RAM than kaggle notebooks? Thank you again.",
      "votes": null
    },
    {
      "id": "3237401",
      "postDate": "07/01/2025 04:39:50",
      "content": "<p>It is a balancing act. I've found using EWC and fisher information matrices allows for more complex neural networks without running out of ram. I even have a notebook that can use RAM from 2 T4 GPUs and offload to system RAM is necessary.</p>",
      "rawMarkdown": "It is a balancing act. I've found using EWC and fisher information matrices allows for more complex neural networks without running out of ram. I even have a notebook that can use RAM from 2 T4 GPUs and offload to system RAM is necessary.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3235531,
      "author_name": "ravi20076",
      "author_url": "",
      "post_date": "06/29/2025 10:58:18",
      "content": "<p>Use a bigger PC with good GPU and RAM. 128 GB RAM is more than enough for this assignment <a href=\"https://www.kaggle.com/obiaf88\" target=\"_blank\">@obiaf88</a> </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3235548,
      "author_name": "obiaf88",
      "author_url": "",
      "post_date": "06/29/2025 11:23:02",
      "content": "<p>Thank you <a href=\"https://www.kaggle.com/ravi20076\" target=\"_blank\">@ravi20076</a> ; I'm using kaggle notebook with CPU and 30 GB RAM; I will read the data in chunks; with this kind of competitions it's better to use a PC with GPU and much RAM than kaggle notebooks? Thank you again.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3237401,
          "author_name": "taylorsamarel",
          "author_url": "",
          "post_date": "07/01/2025 04:39:50",
          "content": "<p>It is a balancing act. I've found using EWC and fisher information matrices allows for more complex neural networks without running out of ram. I even have a notebook that can use RAM from 2 T4 GPUs and offload to system RAM is necessary.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3235515": "In this competition memory management could be a issue and you can easily get out of memory.\nWhat is the best way to manage RAM memory? Is it better to delete a dataframe no longer used and use garbage collector to free RAM memory or the best way is to save the dataframe to disk? Thank you",
    "3235531": "Use a bigger PC with good GPU and RAM. 128 GB RAM is more than enough for this assignment @obiaf88",
    "3235548": "Thank you @ravi20076 ; I'm using kaggle notebook with CPU and 30 GB RAM; I will read the data in chunks; with this kind of competitions it's better to use a PC with GPU and much RAM than kaggle notebooks? Thank you again.",
    "3237401": "It is a balancing act. I've found using EWC and fisher information matrices allows for more complex neural networks without running out of ram. I even have a notebook that can use RAM from 2 T4 GPUs and offload to system RAM is necessary."
  },
  "source": "meta"
}