{
  "id": 166439,
  "title": "preprocessed dataset with images scaled and metadata serapated",
  "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/166439",
  "author_name": "",
  "post_date": "2020-07-12T22:25:28.998039600Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I had some trouble running all preprocessing and getting the metadata in the Kaggle kernels (memory etc issues), so I made a pre-processed dataset: </p>\n\n<p><a href=\"https://www.kaggle.com/donkeys/osic-pulmonary-fibrosispreprocessed\">https://www.kaggle.com/donkeys/osic-pulmonary-fibrosispreprocessed</a></p>\n\n<p>And briefly updated my EDA kernel to show how to load the preprocssed data:</p>\n\n<p><a href=\"https://www.kaggle.com/donkeys/ct-slices-basic-eda\">https://www.kaggle.com/donkeys/ct-slices-basic-eda</a></p>\n\n<p>The preprocessing is done as:</p>\n\n<ul>\n<li>read that image arrays from the DCM files, </li>\n<li>convert the colors to a uniform format across patients, </li>\n<li>remove borders where they are present</li>\n<li>resize all to same size</li>\n<li>output the images as PNG</li>\n</ul>\n\n<p>all the metadata is collected to train and test dataframes.</p>\n\n<p>happy yo hear if you find issues or ideas for improvements.</p>\n\n<p>I will update the kernel more with additional details on the data and also upload the preprocessing code into another kernel in the next few days (having the time..)..</p>\n\n<p>Cheers, :)</p>",
  "messages": [
    {
      "id": "926695",
      "postDate": "07/12/2020 22:25:28",
      "content": "<p>I had some trouble running all preprocessing and getting the metadata in the Kaggle kernels (memory etc issues), so I made a pre-processed dataset: </p>\n\n<p><a href=\"https://www.kaggle.com/donkeys/osic-pulmonary-fibrosispreprocessed\">https://www.kaggle.com/donkeys/osic-pulmonary-fibrosispreprocessed</a></p>\n\n<p>And briefly updated my EDA kernel to show how to load the preprocssed data:</p>\n\n<p><a href=\"https://www.kaggle.com/donkeys/ct-slices-basic-eda\">https://www.kaggle.com/donkeys/ct-slices-basic-eda</a></p>\n\n<p>The preprocessing is done as:</p>\n\n<ul>\n<li>read that image arrays from the DCM files, </li>\n<li>convert the colors to a uniform format across patients, </li>\n<li>remove borders where they are present</li>\n<li>resize all to same size</li>\n<li>output the images as PNG</li>\n</ul>\n\n<p>all the metadata is collected to train and test dataframes.</p>\n\n<p>happy yo hear if you find issues or ideas for improvements.</p>\n\n<p>I will update the kernel more with additional details on the data and also upload the preprocessing code into another kernel in the next few days (having the time..)..</p>\n\n<p>Cheers, :)</p>",
      "rawMarkdown": "I had some trouble running all preprocessing and getting the metadata in the Kaggle kernels (memory etc issues), so I made a pre-processed dataset: \n\nhttps://www.kaggle.com/donkeys/osic-pulmonary-fibrosispreprocessed\n\nAnd briefly updated my EDA kernel to show how to load the preprocssed data:\n\nhttps://www.kaggle.com/donkeys/ct-slices-basic-eda\n\nThe preprocessing is done as:\n\n- read that image arrays from the DCM files, \n- convert the colors to a uniform format across patients, \n- remove borders where they are present\n- resize all to same size\n- output the images as PNG\n\nall the metadata is collected to train and test dataframes.\n\nhappy yo hear if you find issues or ideas for improvements.\n\nI will update the kernel more with additional details on the data and also upload the preprocessing code into another kernel in the next few days (having the time..)..\n\nCheers, :)",
      "votes": null
    },
    {
      "id": "927662",
      "postDate": "07/13/2020 14:23:27",
      "content": "<p>Thanks for the dataset! A novice question, why do the images seem unusually dark after preprocessing? I do understand the conversion of the colors to a uniform format, but the images seem a bit odd for me and seems prone to information loss.</p>\n\n<p>Apologies if my assumption is wrong, I've worked rarely with medical data :)</p>",
      "rawMarkdown": "Thanks for the dataset! A novice question, why do the images seem unusually dark after preprocessing? I do understand the conversion of the colors to a uniform format, but the images seem a bit odd for me and seems prone to information loss.\n\nApologies if my assumption is wrong, I've worked rarely with medical data :)",
      "votes": null
    },
    {
      "id": "928224",
      "postDate": "07/13/2020 19:53:33",
      "content": "<p>I have not worked much with medical image data myself, so I just tried to read some of the specs on the image format and use that as a basis. I will upload  the kernel with the preprocessing code maybe tomorrow after some cleaning up, and then you can also experiment with it if you like. Always an option to just up the brightness a bit more and see if it works better. Or maybe someone understands the format a bit better and can point what else needs to be done with it.. :)</p>",
      "rawMarkdown": "I have not worked much with medical image data myself, so I just tried to read some of the specs on the image format and use that as a basis. I will upload  the kernel with the preprocessing code maybe tomorrow after some cleaning up, and then you can also experiment with it if you like. Always an option to just up the brightness a bit more and see if it works better. Or maybe someone understands the format a bit better and can point what else needs to be done with it.. :)",
      "votes": null
    },
    {
      "id": "928302",
      "postDate": "07/13/2020 21:18:05",
      "content": "<p>Here is a link to the preprocessing kernel:\n<a href=\"https://www.kaggle.com/donkeys/preprocessing-images-to-normalize-colors-and-sizes\">https://www.kaggle.com/donkeys/preprocessing-images-to-normalize-colors-and-sizes</a></p>\n\n<p>If anyone has ideas on how to improve or what I missed, happy to hear!</p>",
      "rawMarkdown": "Here is a link to the preprocessing kernel:\nhttps://www.kaggle.com/donkeys/preprocessing-images-to-normalize-colors-and-sizes\n\nIf anyone has ideas on how to improve or what I missed, happy to hear!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 927662,
      "author_name": "aadhavvignesh",
      "author_url": "",
      "post_date": "07/13/2020 14:23:27",
      "content": "<p>Thanks for the dataset! A novice question, why do the images seem unusually dark after preprocessing? I do understand the conversion of the colors to a uniform format, but the images seem a bit odd for me and seems prone to information loss.</p>\n\n<p>Apologies if my assumption is wrong, I've worked rarely with medical data :)</p>",
      "votes": null,
      "replies": [
        {
          "id": 928224,
          "author_name": "donkeys",
          "author_url": "",
          "post_date": "07/13/2020 19:53:33",
          "content": "<p>I have not worked much with medical image data myself, so I just tried to read some of the specs on the image format and use that as a basis. I will upload  the kernel with the preprocessing code maybe tomorrow after some cleaning up, and then you can also experiment with it if you like. Always an option to just up the brightness a bit more and see if it works better. Or maybe someone understands the format a bit better and can point what else needs to be done with it.. :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 928302,
      "author_name": "donkeys",
      "author_url": "",
      "post_date": "07/13/2020 21:18:05",
      "content": "<p>Here is a link to the preprocessing kernel:\n<a href=\"https://www.kaggle.com/donkeys/preprocessing-images-to-normalize-colors-and-sizes\">https://www.kaggle.com/donkeys/preprocessing-images-to-normalize-colors-and-sizes</a></p>\n\n<p>If anyone has ideas on how to improve or what I missed, happy to hear!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "926695": "I had some trouble running all preprocessing and getting the metadata in the Kaggle kernels (memory etc issues), so I made a pre-processed dataset: \n\nhttps://www.kaggle.com/donkeys/osic-pulmonary-fibrosispreprocessed\n\nAnd briefly updated my EDA kernel to show how to load the preprocssed data:\n\nhttps://www.kaggle.com/donkeys/ct-slices-basic-eda\n\nThe preprocessing is done as:\n\n- read that image arrays from the DCM files, \n- convert the colors to a uniform format across patients, \n- remove borders where they are present\n- resize all to same size\n- output the images as PNG\n\nall the metadata is collected to train and test dataframes.\n\nhappy yo hear if you find issues or ideas for improvements.\n\nI will update the kernel more with additional details on the data and also upload the preprocessing code into another kernel in the next few days (having the time..)..\n\nCheers, :)",
    "927662": "Thanks for the dataset! A novice question, why do the images seem unusually dark after preprocessing? I do understand the conversion of the colors to a uniform format, but the images seem a bit odd for me and seems prone to information loss.\n\nApologies if my assumption is wrong, I've worked rarely with medical data :)",
    "928224": "I have not worked much with medical image data myself, so I just tried to read some of the specs on the image format and use that as a basis. I will upload  the kernel with the preprocessing code maybe tomorrow after some cleaning up, and then you can also experiment with it if you like. Always an option to just up the brightness a bit more and see if it works better. Or maybe someone understands the format a bit better and can point what else needs to be done with it.. :)",
    "928302": "Here is a link to the preprocessing kernel:\nhttps://www.kaggle.com/donkeys/preprocessing-images-to-normalize-colors-and-sizes\n\nIf anyone has ideas on how to improve or what I missed, happy to hear!"
  },
  "source": "meta"
}