{
  "id": 516620,
  "title": "Your notebook tried to allocate more memory than is available. It has restarted....",
  "url": "/competitions/leash-BELKA/discussion/516620",
  "author_name": "",
  "post_date": "2024-07-03T04:56:38.289557900Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hello Kaggle Community,</p>\n<p>I'm currently working on a project involving a large dataset, and I've encountered an issue where my notebook tries to allocate more memory than is available, causing it to restart.</p>\n<p>Has anyone experienced a similar problem or could offer any advice on how to handle large data imports more efficiently in Kaggle notebooks? Any suggestions or best practices would be greatly appreciated.</p>",
  "messages": [
    {
      "id": "2902086",
      "postDate": "07/03/2024 04:56:38",
      "content": "<p>Hello Kaggle Community,</p>\n<p>I'm currently working on a project involving a large dataset, and I've encountered an issue where my notebook tries to allocate more memory than is available, causing it to restart.</p>\n<p>Has anyone experienced a similar problem or could offer any advice on how to handle large data imports more efficiently in Kaggle notebooks? Any suggestions or best practices would be greatly appreciated.</p>",
      "rawMarkdown": "Hello Kaggle Community,\n\nI'm currently working on a project involving a large dataset, and I've encountered an issue where my notebook tries to allocate more memory than is available, causing it to restart.\n\nHas anyone experienced a similar problem or could offer any advice on how to handle large data imports more efficiently in Kaggle notebooks? Any suggestions or best practices would be greatly appreciated.",
      "votes": null
    },
    {
      "id": "2902482",
      "postDate": "07/03/2024 09:28:53",
      "content": "<p>There is a <a href=\"https://www.kaggle.com/code/shlomoron/belka-shrunken-train-set-loading\" target=\"_blank\">shrunken</a> version of the competition dataset. But if you are usning external data… may be you can try repliclate his work.</p>\n<p>Also, do you really need all the memmory you are using? Perhaps you can load on memmory just batches and keep the rest on disk.</p>",
      "rawMarkdown": "There is a [shrunken](https://www.kaggle.com/code/shlomoron/belka-shrunken-train-set-loading) version of the competition dataset. But if you are usning external data... may be you can try repliclate his work.\n\nAlso, do you really need all the memmory you are using? Perhaps you can load on memmory just batches and keep the rest on disk.",
      "votes": null
    },
    {
      "id": "2902862",
      "postDate": "07/03/2024 13:48:46",
      "content": "<p>Thank you for your suggestions! , <a href=\"https://www.kaggle.com/sacuscreed\" target=\"_blank\">@sacuscreed</a>  …. </p>\n<p>I’ll start by trying the shrunken version of the competition dataset to see if that resolves the memory allocation issue. If needed, I'll look into batch loading for my external data as well.</p>",
      "rawMarkdown": "Thank you for your suggestions! , @sacuscreed  .... \n\nI’ll start by trying the shrunken version of the competition dataset to see if that resolves the memory allocation issue. If needed, I'll look into batch loading for my external data as well.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2902482,
      "author_name": "sacuscreed",
      "author_url": "",
      "post_date": "07/03/2024 09:28:53",
      "content": "<p>There is a <a href=\"https://www.kaggle.com/code/shlomoron/belka-shrunken-train-set-loading\" target=\"_blank\">shrunken</a> version of the competition dataset. But if you are usning external data… may be you can try repliclate his work.</p>\n<p>Also, do you really need all the memmory you are using? Perhaps you can load on memmory just batches and keep the rest on disk.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2902862,
          "author_name": "rajusaipainkra",
          "author_url": "",
          "post_date": "07/03/2024 13:48:46",
          "content": "<p>Thank you for your suggestions! , <a href=\"https://www.kaggle.com/sacuscreed\" target=\"_blank\">@sacuscreed</a>  …. </p>\n<p>I’ll start by trying the shrunken version of the competition dataset to see if that resolves the memory allocation issue. If needed, I'll look into batch loading for my external data as well.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2902086": "Hello Kaggle Community,\n\nI'm currently working on a project involving a large dataset, and I've encountered an issue where my notebook tries to allocate more memory than is available, causing it to restart.\n\nHas anyone experienced a similar problem or could offer any advice on how to handle large data imports more efficiently in Kaggle notebooks? Any suggestions or best practices would be greatly appreciated.",
    "2902482": "There is a [shrunken](https://www.kaggle.com/code/shlomoron/belka-shrunken-train-set-loading) version of the competition dataset. But if you are usning external data... may be you can try repliclate his work.\n\nAlso, do you really need all the memmory you are using? Perhaps you can load on memmory just batches and keep the rest on disk.",
    "2902862": "Thank you for your suggestions! , @sacuscreed  .... \n\nI’ll start by trying the shrunken version of the competition dataset to see if that resolves the memory allocation issue. If needed, I'll look into batch loading for my external data as well."
  },
  "source": "meta"
}