{
  "id": 479344,
  "title": "Using pandas to merge the datasets gives me the memory exceeded error.",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/479344",
  "author_name": "",
  "post_date": "2024-02-24T07:13:54.978906900Z",
  "votes": 2,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Any suggestions or recommendations my goal is to merge all the training files.The 20gb memory allocated is not enough and upgrading the notebook is not an option for me unfortunately.</p>",
  "messages": [
    {
      "id": "2666151",
      "postDate": "02/24/2024 07:13:54",
      "content": "<p>Any suggestions or recommendations my goal is to merge all the training files.The 20gb memory allocated is not enough and upgrading the notebook is not an option for me unfortunately.</p>",
      "rawMarkdown": "Any suggestions or recommendations my goal is to merge all the training files.The 20gb memory allocated is not enough and upgrading the notebook is not an option for me unfortunately.",
      "votes": null
    },
    {
      "id": "2666380",
      "postDate": "02/24/2024 11:07:22",
      "content": "<p>two ideas that might help:</p>\n<ul>\n<li>Try down-casting  the data types (e.g converting float32 to float16)</li>\n<li>Try to process the train data and train models, then delete train data before loading test data</li>\n</ul>",
      "rawMarkdown": "two ideas that might help:\n- Try down-casting  the data types (e.g converting float32 to float16)\n- Try to process the train data and train models, then delete train data before loading test data",
      "votes": null
    },
    {
      "id": "2666386",
      "postDate": "02/24/2024 11:11:41",
      "content": "<p>Yes, kindly use polars lazy frame to merge, this will work <a href=\"https://www.kaggle.com/saahirkhan\" target=\"_blank\">@saahirkhan</a> </p>",
      "rawMarkdown": "Yes, kindly use polars lazy frame to merge, this will work @saahirkhan",
      "votes": null
    },
    {
      "id": "2666853",
      "postDate": "02/24/2024 17:15:01",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/saahirkhan\" target=\"_blank\">@saahirkhan</a>  , adding to <a href=\"https://www.kaggle.com/fadylabib\" target=\"_blank\">@fadylabib</a> don't forget to use gc.collect() after deleting or  transforming since values accumulate. Have a look at <a href=\"https://www.kaggle.com/code/arjanso/reducing-dataframe-memory-size-by-65/notebook\" target=\"_blank\">this kernel</a> for reducing memory used, hope it helps!</p>",
      "rawMarkdown": "Hi @saahirkhan  , adding to @fadylabib don't forget to use gc.collect() after deleting or  transforming since values accumulate. Have a look at [this kernel](https://www.kaggle.com/code/arjanso/reducing-dataframe-memory-size-by-65/notebook) for reducing memory used, hope it helps!",
      "votes": null
    },
    {
      "id": "2667320",
      "postDate": "02/25/2024 04:49:26",
      "content": "<h1><a href=\"https://www.kaggle.com/code/darynarr/home-credit-drop-date-features/notebook\" target=\"_blank\">https://www.kaggle.com/code/darynarr/home-credit-drop-date-features/notebook</a></h1>\n<p>hopes this help</p>",
      "rawMarkdown": "#https://www.kaggle.com/code/darynarr/home-credit-drop-date-features/notebook\nhopes this help",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2666380,
      "author_name": "fadylabib",
      "author_url": "",
      "post_date": "02/24/2024 11:07:22",
      "content": "<p>two ideas that might help:</p>\n<ul>\n<li>Try down-casting  the data types (e.g converting float32 to float16)</li>\n<li>Try to process the train data and train models, then delete train data before loading test data</li>\n</ul>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2666386,
      "author_name": "ravi20076",
      "author_url": "",
      "post_date": "02/24/2024 11:11:41",
      "content": "<p>Yes, kindly use polars lazy frame to merge, this will work <a href=\"https://www.kaggle.com/saahirkhan\" target=\"_blank\">@saahirkhan</a> </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2666853,
      "author_name": "giorgosfoukarakis",
      "author_url": "",
      "post_date": "02/24/2024 17:15:01",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/saahirkhan\" target=\"_blank\">@saahirkhan</a>  , adding to <a href=\"https://www.kaggle.com/fadylabib\" target=\"_blank\">@fadylabib</a> don't forget to use gc.collect() after deleting or  transforming since values accumulate. Have a look at <a href=\"https://www.kaggle.com/code/arjanso/reducing-dataframe-memory-size-by-65/notebook\" target=\"_blank\">this kernel</a> for reducing memory used, hope it helps!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2667320,
      "author_name": "imtommi",
      "author_url": "",
      "post_date": "02/25/2024 04:49:26",
      "content": "<h1><a href=\"https://www.kaggle.com/code/darynarr/home-credit-drop-date-features/notebook\" target=\"_blank\">https://www.kaggle.com/code/darynarr/home-credit-drop-date-features/notebook</a></h1>\n<p>hopes this help</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2666151": "Any suggestions or recommendations my goal is to merge all the training files.The 20gb memory allocated is not enough and upgrading the notebook is not an option for me unfortunately.",
    "2666380": "two ideas that might help:\n- Try down-casting  the data types (e.g converting float32 to float16)\n- Try to process the train data and train models, then delete train data before loading test data",
    "2666386": "Yes, kindly use polars lazy frame to merge, this will work @saahirkhan",
    "2666853": "Hi @saahirkhan  , adding to @fadylabib don't forget to use gc.collect() after deleting or  transforming since values accumulate. Have a look at [this kernel](https://www.kaggle.com/code/arjanso/reducing-dataframe-memory-size-by-65/notebook) for reducing memory used, hope it helps!",
    "2667320": "#https://www.kaggle.com/code/darynarr/home-credit-drop-date-features/notebook\nhopes this help"
  },
  "source": "meta"
}