{
  "id": 262978,
  "title": "Dataset Structure - RAM Memory Problem",
  "url": "/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/262978",
  "author_name": "",
  "post_date": "2021-08-08T04:24:29.879209800Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hello! I was trying to upload all the images to RAM (python). To do this I created a function that creates a dictionary with the same tree structure as the input train directory. I added the pixel arrrays (which are numpy arrays) from the dicoms to the dictionary, but doing this still uses a bunch of memory, to the point that it doesn´t fit in 16 GB of RAM. How should I handle the dataset?</p>\n<p>Thanks in advance! :D</p>",
  "messages": [
    {
      "id": "1458766",
      "postDate": "08/08/2021 04:24:29",
      "content": "<p>Hello! I was trying to upload all the images to RAM (python). To do this I created a function that creates a dictionary with the same tree structure as the input train directory. I added the pixel arrrays (which are numpy arrays) from the dicoms to the dictionary, but doing this still uses a bunch of memory, to the point that it doesn´t fit in 16 GB of RAM. How should I handle the dataset?</p>\n<p>Thanks in advance! :D</p>",
      "rawMarkdown": "Hello! I was trying to upload all the images to RAM (python). To do this I created a function that creates a dictionary with the same tree structure as the input train directory. I added the pixel arrrays (which are numpy arrays) from the dicoms to the dictionary, but doing this still uses a bunch of memory, to the point that it doesn´t fit in 16 GB of RAM. How should I handle the dataset?\n\nThanks in advance! :D",
      "votes": null
    },
    {
      "id": "1459273",
      "postDate": "08/08/2021 09:56:02",
      "content": "<p>Try creating a python generator with a batch size(say 32) to convert images to array rather than defining a function to convert images to array. The value of batch size depends upon the size of your RAM. If you have low memory try using batch size 18.</p>",
      "rawMarkdown": "Try creating a python generator with a batch size(say 32) to convert images to array rather than defining a function to convert images to array. The value of batch size depends upon the size of your RAM. If you have low memory try using batch size 18.",
      "votes": null
    },
    {
      "id": "1462942",
      "postDate": "08/10/2021 03:42:38",
      "content": "<p>The raw pixel data in these DICOMs is 10-16 bit. Normalize it down to 8 bit with something like this. Or you could apply a LUT as well.</p>\n<pre><code>pixels = pixels - np.min(pixels)\npixels = pixels / np.max(pixels)\npixels = (pixels * 255).astype(np.uint8)\n</code></pre>",
      "rawMarkdown": "The raw pixel data in these DICOMs is 10-16 bit. Normalize it down to 8 bit with something like this. Or you could apply a LUT as well.\n```\npixels = pixels - np.min(pixels)\npixels = pixels / np.max(pixels)\npixels = (pixels * 255).astype(np.uint8)\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1459273,
      "author_name": "mainak15071993",
      "author_url": "",
      "post_date": "08/08/2021 09:56:02",
      "content": "<p>Try creating a python generator with a batch size(say 32) to convert images to array rather than defining a function to convert images to array. The value of batch size depends upon the size of your RAM. If you have low memory try using batch size 18.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1462942,
      "author_name": "davidbroberts",
      "author_url": "",
      "post_date": "08/10/2021 03:42:38",
      "content": "<p>The raw pixel data in these DICOMs is 10-16 bit. Normalize it down to 8 bit with something like this. Or you could apply a LUT as well.</p>\n<pre><code>pixels = pixels - np.min(pixels)\npixels = pixels / np.max(pixels)\npixels = (pixels * 255).astype(np.uint8)\n</code></pre>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1458766": "Hello! I was trying to upload all the images to RAM (python). To do this I created a function that creates a dictionary with the same tree structure as the input train directory. I added the pixel arrrays (which are numpy arrays) from the dicoms to the dictionary, but doing this still uses a bunch of memory, to the point that it doesn´t fit in 16 GB of RAM. How should I handle the dataset?\n\nThanks in advance! :D",
    "1459273": "Try creating a python generator with a batch size(say 32) to convert images to array rather than defining a function to convert images to array. The value of batch size depends upon the size of your RAM. If you have low memory try using batch size 18.",
    "1462942": "The raw pixel data in these DICOMs is 10-16 bit. Normalize it down to 8 bit with something like this. Or you could apply a LUT as well.\n```\npixels = pixels - np.min(pixels)\npixels = pixels / np.max(pixels)\npixels = (pixels * 255).astype(np.uint8)\n```"
  },
  "source": "meta"
}