{
  "id": 468335,
  "title": "Memory Issue When Merging DataFrames on Kaggle",
  "url": "/competitions/hms-harmful-brain-activity-classification/discussion/468335",
  "author_name": "",
  "post_date": "2024-01-16T09:24:39.464624700Z",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hello, Kaggle community. I'm experiencing a memory issue when trying to merge multiple DataFrames in my notebook. Attached is an image of the error. The notebook restarts because it exceeds the available memory. Do you have any suggestions on how to handle large data volumes or optimize memory usage on Kaggle?</p>\n<p>Note: In my jupyter notebook i do not have this problem.</p>\n<p>You can see my code here: <br>\n<a href=\"https://www.kaggle.com/code/isabelocastillo/convert-to-dataframe\" target=\"_blank\">https://www.kaggle.com/code/isabelocastillo/convert-to-dataframe</a></p>",
  "messages": [
    {
      "id": "2604156",
      "postDate": "01/16/2024 09:24:39",
      "content": "<p>Hello, Kaggle community. I'm experiencing a memory issue when trying to merge multiple DataFrames in my notebook. Attached is an image of the error. The notebook restarts because it exceeds the available memory. Do you have any suggestions on how to handle large data volumes or optimize memory usage on Kaggle?</p>\n<p>Note: In my jupyter notebook i do not have this problem.</p>\n<p>You can see my code here: <br>\n<a href=\"https://www.kaggle.com/code/isabelocastillo/convert-to-dataframe\" target=\"_blank\">https://www.kaggle.com/code/isabelocastillo/convert-to-dataframe</a></p>",
      "rawMarkdown": "Hello, Kaggle community. I'm experiencing a memory issue when trying to merge multiple DataFrames in my notebook. Attached is an image of the error. The notebook restarts because it exceeds the available memory. Do you have any suggestions on how to handle large data volumes or optimize memory usage on Kaggle?\n\nNote: In my jupyter notebook i do not have this problem.\n\nYou can see my code here: \nhttps://www.kaggle.com/code/isabelocastillo/convert-to-dataframe",
      "votes": null
    },
    {
      "id": "2604368",
      "postDate": "01/16/2024 11:44:52",
      "content": "<p><a href=\"https://www.kaggle.com/isabelocastillo\" target=\"_blank\">@isabelocastillo</a> Instead dataframe, do with numpy with float16. need maximum 8.7GB for the eggs and for all spectrogram with float32 need 6GB<br>\n<a href=\"https://www.kaggle.com/datasets/cdeotte/brain-spectrograms\" target=\"_blank\">Dataset - Brain-Spectrograms</a> -- <strong>6GB</strong><br>\n<a href=\"https://www.kaggle.com/datasets/seshurajup/eegs-pairing-dataset\" target=\"_blank\">Dataset - eegs pairing dataset</a>  -- <strong>8.7GB</strong></p>",
      "rawMarkdown": "isabelocastillo Instead dataframe, do with numpy with float16. need maximum 8.7GB for the eggs and for all spectrogram with float32 need 6GB\n[Dataset - Brain-Spectrograms](https://www.kaggle.com/datasets/cdeotte/brain-spectrograms) -- **6GB**\n[Dataset - eegs pairing dataset](https://www.kaggle.com/datasets/seshurajup/eegs-pairing-dataset)  -- **8.7GB**",
      "votes": null
    },
    {
      "id": "2604401",
      "postDate": "01/16/2024 12:01:15",
      "content": "<p>Thanks <a href=\"https://www.kaggle.com/seshurajup\" target=\"_blank\">@seshurajup</a>, im will try with float 16, after i will comment you the results, have a nice day.</p>",
      "rawMarkdown": "Thanks @seshurajup, im will try with float 16, after i will comment you the results, have a nice day.",
      "votes": null
    },
    {
      "id": "2604403",
      "postDate": "01/16/2024 12:01:58",
      "content": "<p>Sure, have a nice day too</p>",
      "rawMarkdown": "Sure, have a nice day too",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2604368,
      "author_name": "seshurajup",
      "author_url": "",
      "post_date": "01/16/2024 11:44:52",
      "content": "<p><a href=\"https://www.kaggle.com/isabelocastillo\" target=\"_blank\">@isabelocastillo</a> Instead dataframe, do with numpy with float16. need maximum 8.7GB for the eggs and for all spectrogram with float32 need 6GB<br>\n<a href=\"https://www.kaggle.com/datasets/cdeotte/brain-spectrograms\" target=\"_blank\">Dataset - Brain-Spectrograms</a> -- <strong>6GB</strong><br>\n<a href=\"https://www.kaggle.com/datasets/seshurajup/eegs-pairing-dataset\" target=\"_blank\">Dataset - eegs pairing dataset</a>  -- <strong>8.7GB</strong></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2604401,
      "author_name": "isabelocastillo",
      "author_url": "",
      "post_date": "01/16/2024 12:01:15",
      "content": "<p>Thanks <a href=\"https://www.kaggle.com/seshurajup\" target=\"_blank\">@seshurajup</a>, im will try with float 16, after i will comment you the results, have a nice day.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2604403,
          "author_name": "seshurajup",
          "author_url": "",
          "post_date": "01/16/2024 12:01:58",
          "content": "<p>Sure, have a nice day too</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2604156": "Hello, Kaggle community. I'm experiencing a memory issue when trying to merge multiple DataFrames in my notebook. Attached is an image of the error. The notebook restarts because it exceeds the available memory. Do you have any suggestions on how to handle large data volumes or optimize memory usage on Kaggle?\n\nNote: In my jupyter notebook i do not have this problem.\n\nYou can see my code here: \nhttps://www.kaggle.com/code/isabelocastillo/convert-to-dataframe",
    "2604368": "isabelocastillo Instead dataframe, do with numpy with float16. need maximum 8.7GB for the eggs and for all spectrogram with float32 need 6GB\n[Dataset - Brain-Spectrograms](https://www.kaggle.com/datasets/cdeotte/brain-spectrograms) -- **6GB**\n[Dataset - eegs pairing dataset](https://www.kaggle.com/datasets/seshurajup/eegs-pairing-dataset)  -- **8.7GB**",
    "2604401": "Thanks @seshurajup, im will try with float 16, after i will comment you the results, have a nice day.",
    "2604403": "Sure, have a nice day too"
  },
  "source": "meta"
}