{
  "id": 166119,
  "title": "Does pydicom close the image files it opens?",
  "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/166119",
  "author_name": "",
  "post_date": "2020-07-11T20:06:12.276264300Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I noticed that when I loop through the image files in the dataset, I pretty much fill up the available RAM, and the RAM usage doesn't come back down if I overwrite the object references that were used when accessing said files.</p>\n\n<p>My suspicion is that <em>pydicom.dcmread()</em> doesn't close the files after accessing them (even if I use a <em>with</em> statement), leaving them to clog up the RAM, but a Google search about this topic didn't return anything useful (even the pydicom docs say very little about this).</p>\n\n<p>Can anyone shed some light on this? Is it necessary to clean up manually after pydicom?</p>",
  "messages": [
    {
      "id": "925107",
      "postDate": "07/11/2020 20:06:12",
      "content": "<p>I noticed that when I loop through the image files in the dataset, I pretty much fill up the available RAM, and the RAM usage doesn't come back down if I overwrite the object references that were used when accessing said files.</p>\n\n<p>My suspicion is that <em>pydicom.dcmread()</em> doesn't close the files after accessing them (even if I use a <em>with</em> statement), leaving them to clog up the RAM, but a Google search about this topic didn't return anything useful (even the pydicom docs say very little about this).</p>\n\n<p>Can anyone shed some light on this? Is it necessary to clean up manually after pydicom?</p>",
      "rawMarkdown": "I noticed that when I loop through the image files in the dataset, I pretty much fill up the available RAM, and the RAM usage doesn't come back down if I overwrite the object references that were used when accessing said files.\n\nMy suspicion is that *pydicom.dcmread()* doesn't close the files after accessing them (even if I use a *with* statement), leaving them to clog up the RAM, but a Google search about this topic didn't return anything useful (even the pydicom docs say very little about this).\n\nCan anyone shed some light on this? Is it necessary to clean up manually after pydicom?",
      "votes": null
    },
    {
      "id": "926109",
      "postDate": "07/12/2020 13:58:24",
      "content": "<p>PyDICOM does close the image in a call to dcmread(), see <a href=\"https://github.com/pydicom/pydicom/blob/master/pydicom/filereader.py\">https://github.com/pydicom/pydicom/blob/master/pydicom/filereader.py</a> at line 871.</p>",
      "rawMarkdown": "PyDICOM does close the image in a call to dcmread(), see https://github.com/pydicom/pydicom/blob/master/pydicom/filereader.py at line 871.",
      "votes": null
    },
    {
      "id": "934535",
      "postDate": "07/18/2020 14:10:53",
      "content": "<p>I have no idea how we are supposed to clean up after pydicom, but it certainly seems to leak memory. I closed and deleted everything, and if running enough DCM files through pydicom it will exhaust up to all RAM and never release it. Seems to be quite common with many ML libraries in my experience, I guess they are more used in short term scripts on bigger machines where this is not a big issue.</p>\n\n<p>I ended up making my own preprocessed dataset to import for this competition, and separate preprocessing code/kernels.. That way no need to play with pydicoms memory leaks or requirement for some libraries it needs (for this dataset) that are not installed on Kaggle.</p>",
      "rawMarkdown": "I have no idea how we are supposed to clean up after pydicom, but it certainly seems to leak memory. I closed and deleted everything, and if running enough DCM files through pydicom it will exhaust up to all RAM and never release it. Seems to be quite common with many ML libraries in my experience, I guess they are more used in short term scripts on bigger machines where this is not a big issue.\n\nI ended up making my own preprocessed dataset to import for this competition, and separate preprocessing code/kernels.. That way no need to play with pydicoms memory leaks or requirement for some libraries it needs (for this dataset) that are not installed on Kaggle.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 926109,
      "author_name": "amylizzle",
      "author_url": "",
      "post_date": "07/12/2020 13:58:24",
      "content": "<p>PyDICOM does close the image in a call to dcmread(), see <a href=\"https://github.com/pydicom/pydicom/blob/master/pydicom/filereader.py\">https://github.com/pydicom/pydicom/blob/master/pydicom/filereader.py</a> at line 871.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 934535,
      "author_name": "donkeys",
      "author_url": "",
      "post_date": "07/18/2020 14:10:53",
      "content": "<p>I have no idea how we are supposed to clean up after pydicom, but it certainly seems to leak memory. I closed and deleted everything, and if running enough DCM files through pydicom it will exhaust up to all RAM and never release it. Seems to be quite common with many ML libraries in my experience, I guess they are more used in short term scripts on bigger machines where this is not a big issue.</p>\n\n<p>I ended up making my own preprocessed dataset to import for this competition, and separate preprocessing code/kernels.. That way no need to play with pydicoms memory leaks or requirement for some libraries it needs (for this dataset) that are not installed on Kaggle.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "925107": "I noticed that when I loop through the image files in the dataset, I pretty much fill up the available RAM, and the RAM usage doesn't come back down if I overwrite the object references that were used when accessing said files.\n\nMy suspicion is that *pydicom.dcmread()* doesn't close the files after accessing them (even if I use a *with* statement), leaving them to clog up the RAM, but a Google search about this topic didn't return anything useful (even the pydicom docs say very little about this).\n\nCan anyone shed some light on this? Is it necessary to clean up manually after pydicom?",
    "926109": "PyDICOM does close the image in a call to dcmread(), see https://github.com/pydicom/pydicom/blob/master/pydicom/filereader.py at line 871.",
    "934535": "I have no idea how we are supposed to clean up after pydicom, but it certainly seems to leak memory. I closed and deleted everything, and if running enough DCM files through pydicom it will exhaust up to all RAM and never release it. Seems to be quite common with many ML libraries in my experience, I guess they are more used in short term scripts on bigger machines where this is not a big issue.\n\nI ended up making my own preprocessed dataset to import for this competition, and separate preprocessing code/kernels.. That way no need to play with pydicoms memory leaks or requirement for some libraries it needs (for this dataset) that are not installed on Kaggle."
  },
  "source": "meta"
}