{
  "id": 309861,
  "title": "memory overflow in notebook",
  "url": "/competitions/happy-whale-and-dolphin/discussion/309861",
  "author_name": "Kiernan McGuigan",
  "post_date": "2022-02-26T04:34:02.746000",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hello,</p>\n<p>I was wondering if anyone could give me a hand, I can't seem to figure out this issue. I'm trying to iteratively load images for each batch when I need them (since theres too many images to hold in memory at once). I'm getting a running out of memory error but when I print out whats in locals and globals nothing is that big. I also tested this out with a batch size of 4 (so only 4 images should ever be loaded at once) and I still got a run out of memory issue. Could anyone help me out please?</p>\n<p><a href=\"https://www.kaggle.com/kiernanmcguigan/efficientnet-for-individual-determination\" target=\"_blank\">https://www.kaggle.com/kiernanmcguigan/efficientnet-for-individual-determination</a></p>",
  "messages": [
    {
      "id": 1705029,
      "postDate": "2022-02-26T04:34:02.747Z",
      "content": "<p>Hello,</p>\n<p>I was wondering if anyone could give me a hand, I can't seem to figure out this issue. I'm trying to iteratively load images for each batch when I need them (since theres too many images to hold in memory at once). I'm getting a running out of memory error but when I print out whats in locals and globals nothing is that big. I also tested this out with a batch size of 4 (so only 4 images should ever be loaded at once) and I still got a run out of memory issue. Could anyone help me out please?</p>\n<p><a href=\"https://www.kaggle.com/kiernanmcguigan/efficientnet-for-individual-determination\" target=\"_blank\">https://www.kaggle.com/kiernanmcguigan/efficientnet-for-individual-determination</a></p>",
      "rawMarkdown": "Hello,\n\nI was wondering if anyone could give me a hand, I can't seem to figure out this issue. I'm trying to iteratively load images for each batch when I need them (since theres too many images to hold in memory at once). I'm getting a running out of memory error but when I print out whats in locals and globals nothing is that big. I also tested this out with a batch size of 4 (so only 4 images should ever be loaded at once) and I still got a run out of memory issue. Could anyone help me out please?\n\nhttps://www.kaggle.com/kiernanmcguigan/efficientnet-for-individual-determination",
      "votes": 3
    },
    {
      "id": 1707598,
      "postDate": "2022-02-28T15:50:09.680Z",
      "content": "<p>Hey I ended up solving this. In my data generator I was initializing the dictionary with lists of images that correspond the the class (dict key). I thought I needed np arrays for no problem.random.choice (turns out you don’t) but when I then went back and reinitialized  the arrays in the dict as numpy arrays it blew up the memory. Not sure why this happens but thought I’d share.</p>",
      "rawMarkdown": "Hey I ended up solving this. In my data generator I was initializing the dictionary with lists of images that correspond the the class (dict key). I thought I needed np arrays for no problem.random.choice (turns out you don’t) but when I then went back and reinitialized  the arrays in the dict as numpy arrays it blew up the memory. Not sure why this happens but thought I’d share.\n\n"
    },
    {
      "id": 1707880,
      "postDate": "2022-02-28T23:00:21.080Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1707598,
      "author_name": "Kiernan McGuigan",
      "author_url": "",
      "post_date": "2022-02-28T15:50:09.680000",
      "content": "<p>Hey I ended up solving this. In my data generator I was initializing the dictionary with lists of images that correspond the the class (dict key). I thought I needed np arrays for no problem.random.choice (turns out you don’t) but when I then went back and reinitialized  the arrays in the dict as numpy arrays it blew up the memory. Not sure why this happens but thought I’d share.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1707880,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-02-28T23:00:21.080000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1705029": "Hello,\n\nI was wondering if anyone could give me a hand, I can't seem to figure out this issue. I'm trying to iteratively load images for each batch when I need them (since theres too many images to hold in memory at once). I'm getting a running out of memory error but when I print out whats in locals and globals nothing is that big. I also tested this out with a batch size of 4 (so only 4 images should ever be loaded at once) and I still got a run out of memory issue. Could anyone help me out please?\n\nhttps://www.kaggle.com/kiernanmcguigan/efficientnet-for-individual-determination",
    "1707598": "Hey I ended up solving this. In my data generator I was initializing the dictionary with lists of images that correspond the the class (dict key). I thought I needed np arrays for no problem.random.choice (turns out you don’t) but when I then went back and reinitialized  the arrays in the dict as numpy arrays it blew up the memory. Not sure why this happens but thought I’d share.\n\n",
    "1707880": ""
  }
}