Preprocessing DICOM files is generally quite computational-intensive and slow, especially considering the large dataset size (~128 GB). For this reason, I've fired up a GCP VM and transformed the data from DICOM to PNG for you. The dataset file size has been reduced from 128 GB to 5.2 GB.
\nNotes
\nImage-X.png). Note that, because of that, some entire folders (such as train/00109/FLAIR) are not included in this dataset (as they contain empty images only).Dataset link
\nScript
\nFor each image, run: (Adapted from @tanlikesmath work)
\ndicom = pydicom.read_file(path)\ndata = apply_voi_lut(dicom.pixel_array, dicom)\nif dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\ndata = data - np.min(data)\ndata = data / np.max(data)\ndata = (data * 255).astype(np.uint8)\nMore
\nIf you need other formats, you want a version with resized images or anything else, just ask ๐
\nThank you for reading, happy kaggling to all!
", "messages": [ { "id": 1388021, "postDate": "2021-07-14T15:44:09.440Z", "content": "Preprocessing DICOM files is generally quite computational-intensive and slow, especially considering the large dataset size (~128 GB). For this reason, I've fired up a GCP VM and transformed the data from DICOM to PNG for you. The dataset file size has been reduced from 128 GB to 5.2 GB.
\nNotes
\nImage-X.png). Note that, because of that, some entire folders (such as train/00109/FLAIR) are not included in this dataset (as they contain empty images only).Dataset link
\nScript
\nFor each image, run: (Adapted from @tanlikesmath work)
\ndicom = pydicom.read_file(path)\ndata = apply_voi_lut(dicom.pixel_array, dicom)\nif dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\ndata = data - np.min(data)\ndata = data / np.max(data)\ndata = (data * 255).astype(np.uint8)\nMore
\nIf you need other formats, you want a version with resized images or anything else, just ask ๐
\nThank you for reading, happy kaggling to all!
", "rawMarkdown": "Preprocessing DICOM files is generally quite computational-intensive and slow, especially considering the large dataset size (~128 GB). For this reason, I've fired up a GCP VM and transformed the data from DICOM to PNG for you. The dataset file size has been reduced from 128 GB to 5.2 GB. \n\n**Notes**\n\n - Images sizes have been kept as the original ones. \n - To further reduce file size, all empty DICOM images files have not been included in the dataset, you can easily spot them by looking at the images sequences (`Image-X.png`). Note that, because of that, some entire folders (such as `train/00109/FLAIR`) are not included in this dataset (as they contain empty images only).\n - The structure of the dataset is not changed. I.e all _.dcm_ files are now _.png_ files.\n\n**Dataset link**\n\n - [RSNA-MICCAI-PNG-Dataset](https://www.kaggle.com/jonathanbesomi/rsna-miccai-png) [5.2 GB]\n\n**Script**\n\nFor each image, run: (Adapted from @tanlikesmath [work](https://www.kaggle.com/tanlikesmath/brain-tumor-radiogenomic-classification-eda))\n\n```\ndicom = pydicom.read_file(path)\ndata = apply_voi_lut(dicom.pixel_array, dicom)\nif dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\ndata = data - np.min(data)\ndata = data / np.max(data)\ndata = (data * 255).astype(np.uint8)\n```\n\n**More**\n\nIf you need other formats, you want a version with resized images or anything else, just ask ๐\n\nThank you for reading, happy kaggling to all!", "votes": 369 }, { "id": 1395928, "postDate": "2021-07-21T16:14:41.843Z", "content": "Greate work.
\nCould you please elaborate on the decision to normalize an images the following way?
\ndata = data - np.min(data)\ndata = data / np.max(data)\ndata = (data * 255).astype(np.uint8)\nsaved images wont be a blank.
", "rawMarkdown": "saved images wont be a blank." } ] }, { "id": 1389541, "postDate": "2021-07-15T19:52:48.717Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing", "votes": 3, "replies": [ { "id": 1390192, "postDate": "2021-07-16T12:46:04.687Z", "content": "Good luck with the comp. @cinthiakleiner !!
", "rawMarkdown": "Good luck with the comp. @cinthiakleiner !!", "votes": -1 } ] }, { "id": 1392053, "postDate": "2021-07-18T10:33:41.790Z", "content": "Thank you for sharing this..
", "rawMarkdown": "Thank you for sharing this..", "votes": 4, "replies": [ { "id": 1393249, "postDate": "2021-07-19T13:48:42.510Z", "content": "You are welcome, hope it helps! :)
", "rawMarkdown": "You are welcome, hope it helps! :)", "votes": -1 } ] }, { "id": 1471361, "postDate": "2021-08-14T06:50:43.293Z", "content": "Why do you use this normalization formula?
\ndata = data - np.min(data)
\n data = data / np.max(data)
\nI checked the original Dicom file, some local minimal values are not equal to the global minimum value, so is the maximum value. The global minimal value is -32768.0 and the global maximum value is 65535.0 (I only checked the converted dicom files). But the minimum value of \"Image-118.dcm\" is -32754.13895939086, and the maximum value of \"Image-118.dcm\" is 16424.83312182741. Your formula threats the -32754 as 0 and 16424 as 255, I think it's wrong.
\nI am considering to use this formula:
\nglobal_min, global_max = -32768.0, 65535.0
\ndata = data - global_min
\n data = data / global_max
\nThe converted images look similar, but they have different pixel values.
I agree, your point is valid if dicom images have pixel range of [-32768, 65535] then it should map them to [0,255] not the one which is found in particular dcm file
", "rawMarkdown": "I agree, your point is valid if dicom images have pixel range of [-32768, 65535] then it should map them to [0,255] not the one which is found in particular dcm file" } ] }, { "id": 1389296, "postDate": "2021-07-15T15:32:41.653Z", "content": "Thanks a lot!
\nBut why using min-max normalize instead of normalize them using their full range (i.e. 0 to 2 ** bit_stored)??
Hey @terenceythsu, not sure I got your question โฆ why would you consider values between max and 2**stored bit?
", "rawMarkdown": "Hey @terenceythsu, not sure I got your question ... why would you consider values between max and 2**_stored bit_?", "votes": -2 }, { "id": 1391233, "postDate": "2021-07-17T12:11:16.277Z", "content": "Hi @jonathanbesomi , I think you got my point.
\nI just thought that the absolute value has its meaning, so maybe min-max normalize will destroy them.
Thanks for sharing!
", "rawMarkdown": "Thanks for sharing! ~~Just for those who don't know the size of the images. I just download them and check. The size of the png images is 512 x 512.~~", "votes": 1, "replies": [ { "id": 1388339, "postDate": "2021-07-14T20:37:10.277Z", "content": "Hey @yus002, actually, the dataset contains images of different sizes, exactly as the original DICOM files. I basically transformed the data from \".dcm\" to \".png\", without changing the dimensions of the images (as this can be done quickly during training time and it wasn't the goal of this step). The \"expensive\" operations are loading DICOM files and extract the images, and that's why this dataset. Hope things are more clear now!
", "rawMarkdown": "Hey @yus002, actually, the dataset **contains images of different sizes**, exactly as the original DICOM files. I basically transformed the data from \".dcm\" to \".png\", without changing the dimensions of the images (as this can be done quickly during training time and it wasn't the goal of this step). The \"expensive\" operations are loading DICOM files and extract the images, and that's why this dataset. Hope things are more clear now! ", "votes": 3 }, { "id": 1388342, "postDate": "2021-07-14T20:43:39.397Z", "content": "I see. Thanks!
", "rawMarkdown": "I see. Thanks!", "votes": 1 } ] }, { "id": 1388037, "postDate": "2021-07-14T15:56:11.977Z", "content": "Wow, this is amazing @jonathanbesomi! My computer has an SSD, so it's limited when it comes to space. Your solution will make this dataset way more accessible to people like me! Thanks a lot!
", "rawMarkdown": "Wow, this is amazing @jonathanbesomi! My computer has an SSD, so it's limited when it comes to space. Your solution will make this dataset way more accessible to people like me! Thanks a lot!", "votes": 1, "replies": [ { "id": 1388046, "postDate": "2021-07-14T16:01:02.010Z", "content": "Hey @jonaslneri, thank you for this comment, it means a lot to me to know that this work can be useful for others! Good luck with the competition! ;) ๐
", "rawMarkdown": "Hey @jonaslneri, thank you for this comment, it means a lot to me to know that this work can be useful for others! Good luck with the competition! ;) ๐", "votes": 1 } ] }, { "id": 1502064, "postDate": "2021-09-03T19:04:42.150Z", "content": "Thanks a lot for sharing, this is very kind of you.
", "rawMarkdown": "Thanks a lot for sharing, this is very kind of you." }, { "id": 1392161, "postDate": "2021-07-18T11:59:42.060Z", "content": "Great work Thanks for sharing !
", "rawMarkdown": "Great work Thanks for sharing !", "votes": 2, "replies": [ { "id": 1393248, "postDate": "2021-07-19T13:47:48.787Z", "content": "Thank you @mohamedhanyyy ! :)
", "rawMarkdown": "Thank you @mohamedhanyyy ! :)", "votes": -1 } ] }, { "id": 1391397, "postDate": "2021-07-17T14:50:51.857Z", "content": "Thanks for sharing ! Extremely useful
", "rawMarkdown": "Thanks for sharing ! Extremely useful", "votes": 2, "replies": [ { "id": 1393253, "postDate": "2021-07-19T13:51:32.413Z", "content": "Thank you Rithik! ๐
", "rawMarkdown": "Thank you Rithik! ๐" } ] }, { "id": 1390308, "postDate": "2021-07-16T14:51:45.033Z", "content": "Pretty amazing work, thank you for sharing with community.
", "rawMarkdown": "Pretty amazing work, thank you for sharing with community.", "votes": 2, "replies": [ { "id": 1393259, "postDate": "2021-07-19T13:52:27.640Z", "content": "Thank you @shangweichen !!
", "rawMarkdown": "Thank you @shangweichen !!" } ] }, { "id": 1389534, "postDate": "2021-07-15T19:41:14.060Z", "content": "Thank you very much for saving so much time and space.
", "rawMarkdown": "Thank you very much for saving so much time and space.", "votes": 2, "replies": [ { "id": 1390195, "postDate": "2021-07-16T12:46:57.323Z", "content": "Glad the dataset can be useful! All the best for the competition! ๐
", "rawMarkdown": "Glad the dataset can be useful! All the best for the competition! ๐" } ] }, { "id": 1388851, "postDate": "2021-07-15T09:05:29.050Z", "content": "Thanks for sharing โฆ. magnificent reduceโฆ
", "rawMarkdown": "Thanks for sharing .... magnificent reduce...", "votes": 2, "replies": [ { "id": 1390204, "postDate": "2021-07-16T12:55:20.143Z", "content": "Thanks @laxmankusuma !! All the best for this competition and the future ones!
", "rawMarkdown": "Thanks @laxmankusuma !! All the best for this competition and the future ones!" } ] }, { "id": 1388074, "postDate": "2021-07-14T16:22:29.750Z", "content": "Firstly, many thanks for your help.
\nI just had a beginner question, there are 4 subfolders with lots of images in each, for one data point or one patient, which one use for model training or which images to choose for model training
Hey @tharun2001!
\nShort answer: use all images you can for training.
\nLong answer: this is a binary classification task at the \"case\" level. For each \"case\", there are 4 folders and each folder contains multiple scans (of different types, you can read more in the data tab here). Note also that the same folder structure is also available for the test part. Therefore, it's suggested to model the problem so that we can exploit all data available, for instance, making 3D reconstructions of the scans.
\nHope it helps. Good luck!
", "rawMarkdown": "Hey @tharun2001!\n\n**Short answer**: use all images you can for training.\n\n**Long answer**: this is a binary classification task at the \"case\" level. For each \"case\", there are 4 folders and each folder contains multiple scans (of different types, you can [read more in the data tab here](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/data)). Note also that the same folder structure is also available for the test part. Therefore, it's suggested to model the problem so that we can exploit all data available, for instance, making 3D reconstructions of the scans.\n\nHope it helps. Good luck!", "votes": 3 } ] }, { "id": 1547031, "postDate": "2021-10-16T16:27:27.277Z", "content": "Thank you so much for doing this and sharing it with everyone! Really made my life easier!
", "rawMarkdown": "Thank you so much for doing this and sharing it with everyone! Really made my life easier!" }, { "id": 1540781, "postDate": "2021-10-10T22:37:27.200Z", "content": "\"data = (data * 255).astype(np.uint8)\" why using \"uint8\" dtype since decom image take value between 0 and 4096 ? dont we lose important information ?
", "rawMarkdown": "\"data = (data * 255).astype(np.uint8)\" why using \"uint8\" dtype since decom image take value between 0 and 4096 ? dont we lose important information ? " }, { "id": 1524936, "postDate": "2021-09-27T02:26:15.343Z", "content": "Thanks for the great data.
\nI'm trying to insert a directory structure data into kaggle's dataset instead of a file, but I can't do it.
\nHow did you manage to insert the data in a directory structure?
Thank you for sharing! It's a very useful to use by Colaboratory.
", "rawMarkdown": "Thank you for sharing! It's a very useful to use by Colaboratory." }, { "id": 1506049, "postDate": "2021-09-07T19:14:08.987Z", "content": "Thanks for the dataset! Do you have the full script for converting everything to PNG?
", "rawMarkdown": "Thanks for the dataset! Do you have the full script for converting everything to PNG?" }, { "id": 1503215, "postDate": "2021-09-05T06:36:18.467Z", "content": "Thanks for sharing, helps with the training time :)
", "rawMarkdown": "Thanks for sharing, helps with the training time :)" }, { "id": 1499113, "postDate": "2021-09-01T12:55:26.383Z", "content": "How did you filter out the empty DICOM images ? Could you share the script that you used to generate the dataset ?
", "rawMarkdown": "How did you filter out the empty DICOM images ? Could you share the script that you used to generate the dataset ?" }, { "id": 1485498, "postDate": "2021-08-22T06:41:55.600Z", "content": "How does this code actually work out deep down? I'm curious.
", "rawMarkdown": "How does this code actually work out deep down? I'm curious." }, { "id": 1464832, "postDate": "2021-08-10T18:18:42.247Z", "content": "How would I write the result data to a .png file?
def process_dicom(path, outpath):\n dicom = pydicom.read_file(path)\n data = apply_voi_lut(dicom.pixel_array, dicom)\n if dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\n data = data - np.min(data)\n data = data / np.max(data)\n data = (data * 255).astype(np.uint8)\n\n height = len(data)\n width = len(data[0])\n\n pixels_out = []\n for row in data:\n pixels_out.extend(row)\n\n image_out = Image.new('L', (width, height))\n image_out.save(outpath)\nTried something like this, I think the mode code is wrong! I am just getting a black image as output. 'L' is an 8-bit greyscale mode, see https://pillow.readthedocs.io/en/stable/handbook/concepts.html#concept-modes.
was following this example: https://stackoverflow.com/questions/30943966/how-can-i-create-a-png-image-file-from-a-list-of-pixel-values-in-python
", "rawMarkdown": "was following this example: https://stackoverflow.com/questions/30943966/how-can-i-create-a-png-image-file-from-a-list-of-pixel-values-in-python " }, { "id": 1465015, "postDate": "2021-08-10T20:01:26.530Z", "content": "Hah, forgot to do
\n image_out.putdata(pixels_out)\nAfter creating the PIL object! It works now!
", "rawMarkdown": "Hah, forgot to do\n```\n image_out.putdata(pixels_out)\n\n```\nAfter creating the PIL object! It works now!" } ] }, { "id": 1464692, "postDate": "2021-08-10T17:03:25.943Z", "content": "Apparently RE: this link https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/263612#1464675
\nWe can't actually use the PNG dataset when submitting we have to convert from DICOM to PNG ourselves.
\nRegardless this is very helpful, thank you!
", "rawMarkdown": "Apparently RE: this link https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/263612#1464675\n\nWe can't actually use the PNG dataset when submitting we have to convert from DICOM to PNG ourselves.\n\nRegardless this is very helpful, thank you!" }, { "id": 1461562, "postDate": "2021-08-09T13:09:33.177Z", "content": "Cool!
\nBut it is necessary to understand, that DICOM contains not only an image, but also different additional data. Here Peter showed how to extract it.
Awesome!!, Thank you so much for sharing this! :)
", "rawMarkdown": "Awesome!!, Thank you so much for sharing this! :)" }, { "id": 1407313, "postDate": "2021-08-01T18:08:22.900Z", "content": "Cool. Thats really helpful
", "rawMarkdown": "Cool. Thats really helpful" }, { "id": 1403601, "postDate": "2021-07-29T09:11:54.803Z", "content": "Hey,
\nThank you for the effort but I am curious, doesn't this conversion compromise the Meta-Data? Also, our inputs would be DICOM images, right?
", "rawMarkdown": "Hey,\n\nThank you for the effort but I am curious, doesn't this conversion compromise the Meta-Data? Also, our inputs would be DICOM images, right?" }, { "id": 1403028, "postDate": "2021-07-28T18:12:58.263Z", "content": "Hii, thank you for sharing your work with us. Can you also tell me how do I import the reduced dataset in google colab notebooks?
", "rawMarkdown": "Hii, thank you for sharing your work with us. Can you also tell me how do I import the reduced dataset in google colab notebooks?" }, { "id": 1402521, "postDate": "2021-07-28T08:57:15.930Z", "content": "Excellent!
", "rawMarkdown": "Excellent!" }, { "id": 1401825, "postDate": "2021-07-27T15:32:43.177Z", "content": "Thank you for sharing great & helpful information!
", "rawMarkdown": "Thank you for sharing great & helpful information!" }, { "id": 1398191, "postDate": "2021-07-23T20:15:10.140Z", "content": "Thank you to share your knowledge! Great work!!
", "rawMarkdown": "Thank you to share your knowledge! Great work!! " }, { "id": 1395919, "postDate": "2021-07-21T16:06:00.257Z", "content": "Is there any link to the data where just format changed from dicom to png/jpeg, keeping other things exactly same, like not removing those empty image files etc ?
\nIt would be a great help, thanks.
Thank for your help!!!
\nCan you share more sizes? More sizes will be very interesting!
Thanks for sharing I was looking for this and this will be really helpful for me.
", "rawMarkdown": "Thanks for sharing I was looking for this and this will be really helpful for me." }, { "id": 1394957, "postDate": "2021-07-20T18:49:56.077Z", "content": "Great work.
", "rawMarkdown": "Great work." }, { "id": 1394510, "postDate": "2021-07-20T12:21:57.700Z", "content": "Sincerely Thanks for sharing this it will really help us in training our model
", "rawMarkdown": "Sincerely Thanks for sharing this it will really help us in training our model" }, { "id": 1391993, "postDate": "2021-07-18T09:21:30.020Z", "content": "Glad to see something I was looking for quite a while
", "rawMarkdown": "Glad to see something I was looking for quite a while" }, { "id": 1391000, "postDate": "2021-07-17T08:22:26.017Z", "content": "That is brilliant. Thank you for time and energy-saving tip, John!
", "rawMarkdown": "That is brilliant. Thank you for time and energy-saving tip, John!", "replies": [ { "id": 1393258, "postDate": "2021-07-19T13:52:11.523Z", "content": "Thank you @samerrkhann! Good luck with this competition! ๐ฅ
", "rawMarkdown": "Thank you @samerrkhann! Good luck with this competition! ๐ฅ" } ] }, { "id": 1390999, "postDate": "2021-07-17T08:22:20.667Z", "content": "That is brilliant. Thank you for time and energy-saving tip, John!
", "rawMarkdown": "That is brilliant. Thank you for time and energy-saving tip, John!" }, { "id": 1389165, "postDate": "2021-07-15T13:44:39.600Z", "content": "Thanks for the transformation! any possibility of having the data in .nii.gz format? I am getting 3 different volumes for each image when applying dcm2nii
", "rawMarkdown": "Thanks for the transformation! any possibility of having the data in .nii.gz format? I am getting 3 different volumes for each image when applying dcm2nii", "replies": [ { "id": 1390199, "postDate": "2021-07-16T12:51:22.003Z", "content": "Hey @ranafago, what do you mean by this sentence: I am getting 3 different volumes for each image when applying dcm2nii?
\nI can try to look into what you are asking if there is a necessity .. why would you need data in the .nii.gz format?
I was working towards a model using handcrafted features since I expected a similar number of cases as last year's survival prediction task, and I just worked always with the data in .nii.gz. I understand the benefits of working with the data without the preprocessing, but I just want to make a small test to check if that approach is still viable against deep learning approaches. Nevertheless, the data for the segmentation task is already in that format, Thanks for your answer!
\nI am getting this output from dcm2nii I understand that the same image is reconstructed cropped and reoriented too. FLAIR image of case 00000:
\nChris Rorden's dcm2nii :: 4AUGUST2014 (Debian) 64bit BSD License
\nreading preferences file /home/rafa/.dcm2nii/dcm2nii.ini
\nData will be exported to FLAIR
\nValidating 400 potential DICOM images.
\nFound 400 DICOM images.
\nConverting 400/400 volumes: 1
\nImage-1.dcm->18991230_000000s004a1000.nii
\nGZipโฆ18991230_000000s004a1000.nii.gz
\nReorienting as FLAIR/o18991230_000000s004a1000.nii.gz
\nGZipโฆo18991230_000000s004a1000.nii.gz
\nCropping NIfTI/Analyze image FLAIR/o18991230_000000s004a1000.nii.gz
\nGZipโฆco18991230_000000s004a1000.nii.gz
Dear Jonathan
\nKindly provide us a (python) code to convert the dicom files to nii.gz format. This will be quite helpful as the codes we have developed so far, with the BraTS dataset, are all written for nii.gz format. Your help is much appreciated.
\nKind Regards
", "rawMarkdown": "Dear Jonathan\n\nKindly provide us a (python) code to convert the dicom files to nii.gz format. This will be quite helpful as the codes we have developed so far, with the BraTS dataset, are all written for nii.gz format. Your help is much appreciated.\n\nKind Regards" }, { "id": 1402463, "postDate": "2021-07-28T08:15:14.090Z", "content": "Hey HMD!
\nI did something similar here, check it out maybe it helps
\nhttps://www.kaggle.com/ranafago/preprocessing-dcm-to-nifti
", "rawMarkdown": "Hey HMD! \n\nI did something similar here, check it out maybe it helps \n\nhttps://www.kaggle.com/ranafago/preprocessing-dcm-to-nifti" }, { "id": 1407359, "postDate": "2021-08-01T19:08:29.263Z", "content": "Hi Ranafago,
\nThank you for sharing your code.
\nI was wondering if you had to disable any flags while using the function convert_directory from dicom2nifti. For example I had to disable validate_orthogonal to get T2w.nii.gz for two of the subjects.
", "rawMarkdown": "Hi Ranafago,\n\nThank you for sharing your code.\n\nI was wondering if you had to disable any flags while using the function convert_directory from dicom2nifti. For example I had to disable validate_orthogonal to get T2w.nii.gz for two of the subjects.\n\n\n" } ] }, { "id": 1389162, "postDate": "2021-07-15T13:42:21.547Z", "content": "You saved my internet. And my bill. Thanks!
", "rawMarkdown": "You saved my internet. And my bill. Thanks!", "replies": [ { "id": 1390202, "postDate": "2021-07-16T12:54:44.793Z", "content": "Yeah, I had to use a virtual machine to download and unzip the data, way too much data for my current machine too! Happy it helped and have fun in this competition. Will you attempt to reach a second gold medal?
", "rawMarkdown": "Yeah, I had to use a virtual machine to download and unzip the data, way too much data for my current machine too! Happy it helped and have fun in this competition. Will you attempt to reach a second gold medal?", "votes": 1 }, { "id": 1391999, "postDate": "2021-07-18T09:28:22.420Z", "content": "I am exploring the datasets and have my fun with personal projects. Such, compressed data is super helpful and experimental on my local machine ๐
", "rawMarkdown": "I am exploring the datasets and have my fun with personal projects. Such, compressed data is super helpful and experimental on my local machine ๐" } ] }, { "id": 1388350, "postDate": "2021-07-14T21:06:12.237Z", "content": "You have a place reserved in Heaven.
", "rawMarkdown": "You have a place reserved in Heaven.", "replies": [ { "id": 1390210, "postDate": "2021-07-16T13:00:52.087Z", "content": "That's a bit too much @josepc :D
", "rawMarkdown": "That's a bit too much @josepc :D " } ] }, { "id": 1485282, "postDate": "2021-08-22T00:15:39.883Z", "content": "Request to add train label file too๐. Thanks for your efforts you made to convert the file in .png file. Because of your efforts, it's now practical possible for me to work with.
", "rawMarkdown": "Request to add train label file too๐. Thanks for your efforts you made to convert the file in .png file. Because of your efforts, it's now practical possible for me to work with. ", "votes": 4, "isDeleted": true }, { "id": 1400510, "postDate": "2021-07-26T11:02:11.120Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1395003, "postDate": "2021-07-20T19:46:17.440Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1393109, "postDate": "2021-07-19T11:40:30.547Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1393244, "postDate": "2021-07-19T13:44:42.103Z", "content": "Hi @aryamansharma47, thank you for your comment, this is not a mistake and it has been already explained here.
\nBasically: as you can read from notes, \"to further reduce file size, all empty DICOM images files have not been included in the dataset\". All FLAIR folders for the folders you mentioned (train/00109, train/00123, etc) contain only empty images and therefore have been skipped.
\nHope it helps, good luck with the competition!
", "rawMarkdown": "Hi @aryamansharma47, thank you for your comment, this is not a mistake and it has been already explained [here](https://www.kaggle.com/jonathanbesomi/rsna-miccai-png/discussion/253022).\n\nBasically: as you can read from notes, \"**to further reduce file size, all empty DICOM images files have not been included in the dataset**\". All _FLAIR_ folders for the folders you mentioned (train/00109, train/00123, etc) contain only empty images and therefore have been skipped.\n\nHope it helps, good luck with the competition!", "votes": 1 }, { "id": 1393823, "postDate": "2021-07-20T00:09:33.667Z", "content": "my bad haha, thanks
", "rawMarkdown": "my bad haha, thanks " } ] }, { "id": 1392050, "postDate": "2021-07-18T10:28:23.657Z", "content": "Thanks for the sharing. As the target is radiomics, is precision lost (from 16 bits to 8bits) can damage subtitles textural patterns ? :s
", "rawMarkdown": "Thanks for the sharing. As the target is radiomics, is precision lost (from 16 bits to 8bits) can damage subtitles textural patterns ? :s ", "isDeleted": true, "replies": [ { "id": 1393252, "postDate": "2021-07-19T13:51:00.733Z", "content": "Hi Stephan, yes, as already discussed below, as we go from 128 GB to ~5GB it's expected to have some information loss. It's the tradeoff between preprocessing time/file size and information loss.
", "rawMarkdown": "Hi Stephan, yes, as already discussed below, as we go from 128 GB to ~5GB it's expected to have some information loss. It's the tradeoff between preprocessing time/file size and information loss. " } ] }, { "id": 1389521, "postDate": "2021-07-15T19:22:43.463Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1388414, "postDate": "2021-07-14T23:28:34.737Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1388747, "postDate": "2021-07-15T07:36:08.020Z", "content": "Not exactly, what he does is convert the values to the 0-255 range by minmax scaling
", "rawMarkdown": "Not exactly, what he does is convert the values to the 0-255 range by minmax scaling" }, { "id": 1389579, "postDate": "2021-07-15T20:51:14.420Z", "content": "In that case, since the images are converted to int8 arrays, we are losing some data precision due to the dynamic range compression and rounding, right?
", "rawMarkdown": "In that case, since the images are converted to int8 arrays, we are losing some data precision due to the dynamic range compression and rounding, right?" }, { "id": 1390205, "postDate": "2021-07-16T12:59:31.167Z", "content": "\n\nIn that case, since the images are converted to int8 arrays, we are losing some data precision due to the dynamic range compression and rounding, right?
\n
It's a tradeoff. Surely we lost some data, as we went from 128 GB to ~6GB but the advantage is that the dataset is much more malleable right now. I believe still, we can build quite strong models on this dataset too, especially in the initial phase for faster iterations.
", "rawMarkdown": "> In that case, since the images are converted to int8 arrays, we are losing some data precision due to the dynamic range compression and rounding, right?\n\nIt's a tradeoff. Surely we lost some data, as we went from 128 GB to ~6GB but the advantage is that the dataset is much more malleable right now. I believe still, we can build quite strong models on this dataset too, especially in the initial phase for faster iterations.", "votes": 2 } ] }, { "id": 1538355, "postDate": "2021-10-08T10:34:06.897Z", "content": "Thanks a lot!
", "rawMarkdown": "Thanks a lot!" }, { "id": 1535553, "postDate": "2021-10-06T00:37:46.337Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 1508350, "postDate": "2021-09-10T06:59:43.050Z", "content": "Thank you very much!
", "rawMarkdown": "Thank you very much!" }, { "id": 1507707, "postDate": "2021-09-09T13:03:12.023Z", "content": "Thank you for sharing :)
", "rawMarkdown": "Thank you for sharing :) " }, { "id": 1490981, "postDate": "2021-08-26T04:15:24.700Z", "content": "Thank you for sharing!
", "rawMarkdown": "Thank you for sharing!" }, { "id": 1485310, "postDate": "2021-08-22T01:42:25.983Z", "content": "Thank you for sharing!!
", "rawMarkdown": "Thank you for sharing!!" }, { "id": 1477503, "postDate": "2021-08-17T13:59:11.910Z", "content": "Awesome work! Thank you for sharing!
", "rawMarkdown": "Awesome work! Thank you for sharing!" }, { "id": 1473915, "postDate": "2021-08-15T19:17:01.260Z", "content": "thanks for sharing great work
", "rawMarkdown": "thanks for sharing great work" }, { "id": 1473423, "postDate": "2021-08-15T15:05:44.153Z", "content": "Thank you for sharing!
", "rawMarkdown": "Thank you for sharing!" }, { "id": 1473137, "postDate": "2021-08-15T11:41:05.720Z", "content": "Thanks for sharing! very helpful!
", "rawMarkdown": "Thanks for sharing! very helpful!" }, { "id": 1457486, "postDate": "2021-08-07T12:40:18.997Z", "content": "thank you so much
", "rawMarkdown": "thank you so much\n" }, { "id": 1443214, "postDate": "2021-08-04T02:12:53.270Z", "content": "Thank you you saved my diskโฆ. :)
", "rawMarkdown": "Thank you you saved my disk.... :)" }, { "id": 1400843, "postDate": "2021-07-26T16:25:57.900Z", "content": "Thanks a lot !
", "rawMarkdown": "Thanks a lot !" }, { "id": 1400319, "postDate": "2021-07-26T07:54:06.937Z", "content": "Thank you for sharing :)
", "rawMarkdown": "Thank you for sharing :)" }, { "id": 1400037, "postDate": "2021-07-26T00:54:52.077Z", "content": "Thank you Jonathan :D
", "rawMarkdown": "Thank you Jonathan :D" }, { "id": 1397491, "postDate": "2021-07-23T08:41:11.983Z", "content": "Great and Thanks for sharing this .
", "rawMarkdown": "Great and Thanks for sharing this .\n" }, { "id": 1396288, "postDate": "2021-07-22T01:43:03.203Z", "content": "Great! Thanks!
", "rawMarkdown": "Great! Thanks!" }, { "id": 1395033, "postDate": "2021-07-20T20:27:02.627Z", "content": "Thank you so much for sharing.
", "rawMarkdown": "Thank you so much for sharing." }, { "id": 1393896, "postDate": "2021-07-20T02:25:59.653Z", "content": "Thank you for such a nice work :)
", "rawMarkdown": "Thank you for such a nice work :)" }, { "id": 1391078, "postDate": "2021-07-17T09:12:21.437Z", "content": "Thank you for sharing.
", "rawMarkdown": "Thank you for sharing." } ], "comments": [ { "id": 1395928, "author_name": "Roman", "author_url": "", "post_date": "2021-07-21T16:14:41.843000", "content": "Greate work.
\nCould you please elaborate on the decision to normalize an images the following way?
\ndata = data - np.min(data)\ndata = data / np.max(data)\ndata = (data * 255).astype(np.uint8)\nsaved images wont be a blank.
", "votes": 0, "replies": [] } ] }, { "id": 1389541, "author_name": "Cinthia Cristina Calchi Kleiner", "author_url": "", "post_date": "2021-07-15T19:52:48.717000", "content": "Thanks for sharing
", "votes": 3, "replies": [ { "id": 1390192, "author_name": "Jonathan Besomi", "author_url": "", "post_date": "2021-07-16T12:46:04.687000", "content": "Good luck with the comp. @cinthiakleiner !!
", "votes": -1, "replies": [] } ] }, { "id": 1392053, "author_name": "Rahul Bhimani", "author_url": "", "post_date": "2021-07-18T10:33:41.790000", "content": "Thank you for sharing this..
", "votes": 4, "replies": [ { "id": 1393249, "author_name": "Jonathan Besomi", "author_url": "", "post_date": "2021-07-19T13:48:42.510000", "content": "You are welcome, hope it helps! :)
", "votes": -1, "replies": [] } ] }, { "id": 1471361, "author_name": "Shanshan Yu", "author_url": "", "post_date": "2021-08-14T06:50:43.293000", "content": "Why do you use this normalization formula?
\ndata = data - np.min(data)
\n data = data / np.max(data)
\nI checked the original Dicom file, some local minimal values are not equal to the global minimum value, so is the maximum value. The global minimal value is -32768.0 and the global maximum value is 65535.0 (I only checked the converted dicom files). But the minimum value of \"Image-118.dcm\" is -32754.13895939086, and the maximum value of \"Image-118.dcm\" is 16424.83312182741. Your formula threats the -32754 as 0 and 16424 as 255, I think it's wrong.
\nI am considering to use this formula:
\nglobal_min, global_max = -32768.0, 65535.0
\ndata = data - global_min
\n data = data / global_max
\nThe converted images look similar, but they have different pixel values.
I agree, your point is valid if dicom images have pixel range of [-32768, 65535] then it should map them to [0,255] not the one which is found in particular dcm file
", "votes": 0, "replies": [] } ] }, { "id": 1389296, "author_name": "lazyterence", "author_url": "", "post_date": "2021-07-15T15:32:41.653000", "content": "Thanks a lot!
\nBut why using min-max normalize instead of normalize them using their full range (i.e. 0 to 2 ** bit_stored)??
Hey @terenceythsu, not sure I got your question โฆ why would you consider values between max and 2**stored bit?
", "votes": -2, "replies": [] }, { "id": 1391233, "author_name": "lazyterence", "author_url": "", "post_date": "2021-07-17T12:11:16.277000", "content": "Hi @jonathanbesomi , I think you got my point.
\nI just thought that the absolute value has its meaning, so maybe min-max normalize will destroy them.
Thanks for sharing!
", "votes": 1, "replies": [ { "id": 1388339, "author_name": "Jonathan Besomi", "author_url": "", "post_date": "2021-07-14T20:37:10.277000", "content": "Hey @yus002, actually, the dataset contains images of different sizes, exactly as the original DICOM files. I basically transformed the data from \".dcm\" to \".png\", without changing the dimensions of the images (as this can be done quickly during training time and it wasn't the goal of this step). The \"expensive\" operations are loading DICOM files and extract the images, and that's why this dataset. Hope things are more clear now!
", "votes": 3, "replies": [] }, { "id": 1388342, "author_name": "Yue Sun", "author_url": "", "post_date": "2021-07-14T20:43:39.397000", "content": "I see. Thanks!
", "votes": 1, "replies": [] } ] }, { "id": 1388037, "author_name": "Jonas Neri", "author_url": "", "post_date": "2021-07-14T15:56:11.977000", "content": "Wow, this is amazing @jonathanbesomi! My computer has an SSD, so it's limited when it comes to space. Your solution will make this dataset way more accessible to people like me! Thanks a lot!
", "votes": 1, "replies": [ { "id": 1388046, "author_name": "Jonathan Besomi", "author_url": "", "post_date": "2021-07-14T16:01:02.010000", "content": "Hey @jonaslneri, thank you for this comment, it means a lot to me to know that this work can be useful for others! Good luck with the competition! ;) ๐
", "votes": 1, "replies": [] } ] }, { "id": 1502064, "author_name": "Mahmoud Limam", "author_url": "", "post_date": "2021-09-03T19:04:42.150000", "content": "Thanks a lot for sharing, this is very kind of you.
", "votes": 0, "replies": [] }, { "id": 1392161, "author_name": "Mohamed Hany", "author_url": "", "post_date": "2021-07-18T11:59:42.060000", "content": "Great work Thanks for sharing !
", "votes": 2, "replies": [ { "id": 1393248, "author_name": "Jonathan Besomi", "author_url": "", "post_date": "2021-07-19T13:47:48.787000", "content": "Thank you @mohamedhanyyy ! :)
", "votes": -1, "replies": [] } ] }, { "id": 1391397, "author_name": "Rithik Kotha", "author_url": "", "post_date": "2021-07-17T14:50:51.857000", "content": "Thanks for sharing ! Extremely useful
", "votes": 2, "replies": [ { "id": 1393253, "author_name": "Jonathan Besomi", "author_url": "", "post_date": "2021-07-19T13:51:32.413000", "content": "Thank you Rithik! ๐
", "votes": 0, "replies": [] } ] }, { "id": 1390308, "author_name": "README", "author_url": "", "post_date": "2021-07-16T14:51:45.033000", "content": "Pretty amazing work, thank you for sharing with community.
", "votes": 2, "replies": [ { "id": 1393259, "author_name": "Jonathan Besomi", "author_url": "", "post_date": "2021-07-19T13:52:27.640000", "content": "Thank you @shangweichen !!
", "votes": 0, "replies": [] } ] }, { "id": 1389534, "author_name": "Arnab Ray Chaudhuri", "author_url": "", "post_date": "2021-07-15T19:41:14.060000", "content": "Thank you very much for saving so much time and space.
", "votes": 2, "replies": [ { "id": 1390195, "author_name": "Jonathan Besomi", "author_url": "", "post_date": "2021-07-16T12:46:57.323000", "content": "Glad the dataset can be useful! All the best for the competition! ๐
", "votes": 0, "replies": [] } ] }, { "id": 1388851, "author_name": "laxman kusuma", "author_url": "", "post_date": "2021-07-15T09:05:29.050000", "content": "Thanks for sharing โฆ. magnificent reduceโฆ
", "votes": 2, "replies": [ { "id": 1390204, "author_name": "Jonathan Besomi", "author_url": "", "post_date": "2021-07-16T12:55:20.143000", "content": "Thanks @laxmankusuma !! All the best for this competition and the future ones!
", "votes": 0, "replies": [] } ] }, { "id": 1388074, "author_name": "tharun_01", "author_url": "", "post_date": "2021-07-14T16:22:29.750000", "content": "Firstly, many thanks for your help.
\nI just had a beginner question, there are 4 subfolders with lots of images in each, for one data point or one patient, which one use for model training or which images to choose for model training
Hey @tharun2001!
\nShort answer: use all images you can for training.
\nLong answer: this is a binary classification task at the \"case\" level. For each \"case\", there are 4 folders and each folder contains multiple scans (of different types, you can read more in the data tab here). Note also that the same folder structure is also available for the test part. Therefore, it's suggested to model the problem so that we can exploit all data available, for instance, making 3D reconstructions of the scans.
\nHope it helps. Good luck!
", "votes": 3, "replies": [] } ] }, { "id": 1547031, "author_name": "Tanmay Gupta", "author_url": "", "post_date": "2021-10-16T16:27:27.277000", "content": "Thank you so much for doing this and sharing it with everyone! Really made my life easier!
", "votes": 0, "replies": [] }, { "id": 1540781, "author_name": "Kwikik", "author_url": "", "post_date": "2021-10-10T22:37:27.200000", "content": "\"data = (data * 255).astype(np.uint8)\" why using \"uint8\" dtype since decom image take value between 0 and 4096 ? dont we lose important information ?
", "votes": 0, "replies": [] }, { "id": 1524936, "author_name": "Taiyo EG", "author_url": "", "post_date": "2021-09-27T02:26:15.343000", "content": "Thanks for the great data.
\nI'm trying to insert a directory structure data into kaggle's dataset instead of a file, but I can't do it.
\nHow did you manage to insert the data in a directory structure?
Thank you for sharing! It's a very useful to use by Colaboratory.
", "votes": 0, "replies": [] }, { "id": 1506049, "author_name": "alckasoc", "author_url": "", "post_date": "2021-09-07T19:14:08.987000", "content": "Thanks for the dataset! Do you have the full script for converting everything to PNG?
", "votes": 0, "replies": [] }, { "id": 1503215, "author_name": "Md Zarif Ul Alam", "author_url": "", "post_date": "2021-09-05T06:36:18.467000", "content": "Thanks for sharing, helps with the training time :)
", "votes": 0, "replies": [] }, { "id": 1499113, "author_name": "Ayushman Buragohain", "author_url": "", "post_date": "2021-09-01T12:55:26.383000", "content": "How did you filter out the empty DICOM images ? Could you share the script that you used to generate the dataset ?
", "votes": 0, "replies": [] }, { "id": 1485498, "author_name": "DeeplyAbstract", "author_url": "", "post_date": "2021-08-22T06:41:55.600000", "content": "How does this code actually work out deep down? I'm curious.
", "votes": 0, "replies": [] }, { "id": 1464832, "author_name": "Daniel Chen", "author_url": "", "post_date": "2021-08-10T18:18:42.247000", "content": "How would I write the result data to a .png file?
def process_dicom(path, outpath):\n dicom = pydicom.read_file(path)\n data = apply_voi_lut(dicom.pixel_array, dicom)\n if dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\n data = data - np.min(data)\n data = data / np.max(data)\n data = (data * 255).astype(np.uint8)\n\n height = len(data)\n width = len(data[0])\n\n pixels_out = []\n for row in data:\n pixels_out.extend(row)\n\n image_out = Image.new('L', (width, height))\n image_out.save(outpath)\nTried something like this, I think the mode code is wrong! I am just getting a black image as output. 'L' is an 8-bit greyscale mode, see https://pillow.readthedocs.io/en/stable/handbook/concepts.html#concept-modes.
was following this example: https://stackoverflow.com/questions/30943966/how-can-i-create-a-png-image-file-from-a-list-of-pixel-values-in-python
", "votes": 0, "replies": [] }, { "id": 1465015, "author_name": "Daniel Chen", "author_url": "", "post_date": "2021-08-10T20:01:26.530000", "content": "Hah, forgot to do
\n image_out.putdata(pixels_out)\nAfter creating the PIL object! It works now!
", "votes": 0, "replies": [] } ] }, { "id": 1464692, "author_name": "Daniel Chen", "author_url": "", "post_date": "2021-08-10T17:03:25.943000", "content": "Apparently RE: this link https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/263612#1464675
\nWe can't actually use the PNG dataset when submitting we have to convert from DICOM to PNG ourselves.
\nRegardless this is very helpful, thank you!
", "votes": 0, "replies": [] }, { "id": 1461562, "author_name": "Vadzim Tsitko", "author_url": "", "post_date": "2021-08-09T13:09:33.177000", "content": "Cool!
\nBut it is necessary to understand, that DICOM contains not only an image, but also different additional data. Here Peter showed how to extract it.
Awesome!!, Thank you so much for sharing this! :)
", "votes": 0, "replies": [] }, { "id": 1407313, "author_name": "Sougata", "author_url": "", "post_date": "2021-08-01T18:08:22.900000", "content": "Cool. Thats really helpful
", "votes": 0, "replies": [] }, { "id": 1403601, "author_name": "Rohaan Nadeem", "author_url": "", "post_date": "2021-07-29T09:11:54.803000", "content": "Hey,
\nThank you for the effort but I am curious, doesn't this conversion compromise the Meta-Data? Also, our inputs would be DICOM images, right?
", "votes": 0, "replies": [] }, { "id": 1403028, "author_name": "Mohd Faraz Shaikh", "author_url": "", "post_date": "2021-07-28T18:12:58.263000", "content": "Hii, thank you for sharing your work with us. Can you also tell me how do I import the reduced dataset in google colab notebooks?
", "votes": 0, "replies": [] }, { "id": 1402521, "author_name": "", "author_url": "", "post_date": "2021-07-28T08:57:15.930000", "content": "", "votes": 0, "replies": [] }, { "id": 1401825, "author_name": "", "author_url": "", "post_date": "2021-07-27T15:32:43.177000", "content": "", "votes": 0, "replies": [] }, { "id": 1398191, "author_name": "", "author_url": "", "post_date": "2021-07-23T20:15:10.140000", "content": "", "votes": 0, "replies": [] }, { "id": 1395919, "author_name": "", "author_url": "", "post_date": "2021-07-21T16:06:00.257000", "content": "", "votes": 0, "replies": [] }, { "id": 1395298, "author_name": "", "author_url": "", "post_date": "2021-07-21T05:46:41.627000", "content": "", "votes": 0, "replies": [] }, { "id": 1395281, "author_name": "", "author_url": "", "post_date": "2021-07-21T05:23:05.650000", "content": "", "votes": 0, "replies": [] }, { "id": 1394957, "author_name": "", "author_url": "", "post_date": "2021-07-20T18:49:56.077000", "content": "", "votes": 0, "replies": [] }, { "id": 1394510, "author_name": "", "author_url": "", "post_date": "2021-07-20T12:21:57.700000", "content": "", "votes": 0, "replies": [] }, { "id": 1391993, "author_name": "", "author_url": "", "post_date": "2021-07-18T09:21:30.020000", "content": "", "votes": 0, "replies": [] }, { "id": 1391000, "author_name": "", "author_url": "", "post_date": "2021-07-17T08:22:26.017000", "content": "", "votes": 0, "replies": [ { "id": 1393258, "author_name": "", "author_url": "", "post_date": "2021-07-19T13:52:11.523000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1390999, "author_name": "", "author_url": "", "post_date": "2021-07-17T08:22:20.667000", "content": "", "votes": 0, "replies": [] }, { "id": 1389165, "author_name": "", "author_url": "", "post_date": "2021-07-15T13:44:39.600000", "content": "", "votes": 0, "replies": [ { "id": 1390199, "author_name": "", "author_url": "", "post_date": "2021-07-16T12:51:22.003000", "content": "", "votes": 0, "replies": [] }, { "id": 1392583, "author_name": "", "author_url": "", "post_date": "2021-07-18T20:47:35.177000", "content": "", "votes": 0, "replies": [] }, { "id": 1402406, "author_name": "", "author_url": "", "post_date": "2021-07-28T07:04:35.300000", "content": "", "votes": 0, "replies": [] }, { "id": 1402463, "author_name": "", "author_url": "", "post_date": "2021-07-28T08:15:14.090000", "content": "", "votes": 0, "replies": [] }, { "id": 1407359, "author_name": "", "author_url": "", "post_date": "2021-08-01T19:08:29.263000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1389162, "author_name": "", "author_url": "", "post_date": "2021-07-15T13:42:21.547000", "content": "", "votes": 0, "replies": [ { "id": 1390202, "author_name": "", "author_url": "", "post_date": "2021-07-16T12:54:44.793000", "content": "", "votes": 1, "replies": [] }, { "id": 1391999, "author_name": "", "author_url": "", "post_date": "2021-07-18T09:28:22.420000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1388350, "author_name": "", "author_url": "", "post_date": "2021-07-14T21:06:12.237000", "content": "", "votes": 0, "replies": [ { "id": 1390210, "author_name": "", "author_url": "", "post_date": "2021-07-16T13:00:52.087000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1485282, "author_name": "", "author_url": "", "post_date": "2021-08-22T00:15:39.883000", "content": "", "votes": 4, "replies": [] }, { "id": 1400510, "author_name": "", "author_url": "", "post_date": "2021-07-26T11:02:11.120000", "content": "", "votes": 0, "replies": [] }, { "id": 1395003, "author_name": "", "author_url": "", "post_date": "2021-07-20T19:46:17.440000", "content": "", "votes": 0, "replies": [] }, { "id": 1393109, "author_name": "", "author_url": "", "post_date": "2021-07-19T11:40:30.547000", "content": "", "votes": 0, "replies": [ { "id": 1393244, "author_name": "", "author_url": "", "post_date": "2021-07-19T13:44:42.103000", "content": "", "votes": 1, "replies": [] }, { "id": 1393823, "author_name": "", "author_url": "", "post_date": "2021-07-20T00:09:33.667000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1392050, "author_name": "", "author_url": "", "post_date": "2021-07-18T10:28:23.657000", "content": "", "votes": 0, "replies": [ { "id": 1393252, "author_name": "", "author_url": "", "post_date": "2021-07-19T13:51:00.733000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1389521, "author_name": "", "author_url": "", "post_date": "2021-07-15T19:22:43.463000", "content": "", "votes": 0, "replies": [] }, { "id": 1388414, "author_name": "", "author_url": "", "post_date": "2021-07-14T23:28:34.737000", "content": "", "votes": 0, "replies": [ { "id": 1388747, "author_name": "", "author_url": "", "post_date": "2021-07-15T07:36:08.020000", "content": "", "votes": 0, "replies": [] }, { "id": 1389579, "author_name": "", "author_url": "", "post_date": "2021-07-15T20:51:14.420000", "content": "", "votes": 0, "replies": [] }, { "id": 1390205, "author_name": "", "author_url": "", "post_date": "2021-07-16T12:59:31.167000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 1538355, "author_name": "", "author_url": "", "post_date": "2021-10-08T10:34:06.897000", "content": "", "votes": 0, "replies": [] }, { "id": 1535553, "author_name": "", "author_url": "", "post_date": "2021-10-06T00:37:46.337000", "content": "", "votes": 0, "replies": [] }, { "id": 1508350, "author_name": "", "author_url": "", "post_date": "2021-09-10T06:59:43.050000", "content": "", "votes": 0, "replies": [] }, { "id": 1507707, "author_name": "", "author_url": "", "post_date": "2021-09-09T13:03:12.023000", "content": "", "votes": 0, "replies": [] }, { "id": 1490981, "author_name": "", "author_url": "", "post_date": "2021-08-26T04:15:24.700000", "content": "", "votes": 0, "replies": [] }, { "id": 1485310, "author_name": "", "author_url": "", "post_date": "2021-08-22T01:42:25.983000", "content": "", "votes": 0, "replies": [] }, { "id": 1477503, "author_name": "", "author_url": "", "post_date": "2021-08-17T13:59:11.910000", "content": "", "votes": 0, "replies": [] }, { "id": 1473915, "author_name": "", "author_url": "", "post_date": "2021-08-15T19:17:01.260000", "content": "", "votes": 0, "replies": [] }, { "id": 1473423, "author_name": "", "author_url": "", "post_date": "2021-08-15T15:05:44.153000", "content": "", "votes": 0, "replies": [] }, { "id": 1473137, "author_name": "", "author_url": "", "post_date": "2021-08-15T11:41:05.720000", "content": "", "votes": 0, "replies": [] }, { "id": 1457486, "author_name": "", "author_url": "", "post_date": "2021-08-07T12:40:18.997000", "content": "", "votes": 0, "replies": [] }, { "id": 1443214, "author_name": "", "author_url": "", "post_date": "2021-08-04T02:12:53.270000", "content": "", "votes": 0, "replies": [] }, { "id": 1400843, "author_name": "", "author_url": "", "post_date": "2021-07-26T16:25:57.900000", "content": "", "votes": 0, "replies": [] }, { "id": 1400319, "author_name": "", "author_url": "", "post_date": "2021-07-26T07:54:06.937000", "content": "", "votes": 0, "replies": [] }, { "id": 1400037, "author_name": "", "author_url": "", "post_date": "2021-07-26T00:54:52.077000", "content": "", "votes": 0, "replies": [] }, { "id": 1397491, "author_name": "", "author_url": "", "post_date": "2021-07-23T08:41:11.983000", "content": "", "votes": 0, "replies": [] }, { "id": 1396288, "author_name": "", "author_url": "", "post_date": "2021-07-22T01:43:03.203000", "content": "", "votes": 0, "replies": [] }, { "id": 1395033, "author_name": "", "author_url": "", "post_date": "2021-07-20T20:27:02.627000", "content": "", "votes": 0, "replies": [] }, { "id": 1393896, "author_name": "", "author_url": "", "post_date": "2021-07-20T02:25:59.653000", "content": "", "votes": 0, "replies": [] }, { "id": 1391078, "author_name": "", "author_url": "", "post_date": "2021-07-17T09:12:21.437000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1388021": "Preprocessing DICOM files is generally quite computational-intensive and slow, especially considering the large dataset size (~128 GB). For this reason, I've fired up a GCP VM and transformed the data from DICOM to PNG for you. The dataset file size has been reduced from 128 GB to 5.2 GB. \n\n**Notes**\n\n - Images sizes have been kept as the original ones. \n - To further reduce file size, all empty DICOM images files have not been included in the dataset, you can easily spot them by looking at the images sequences (`Image-X.png`). Note that, because of that, some entire folders (such as `train/00109/FLAIR`) are not included in this dataset (as they contain empty images only).\n - The structure of the dataset is not changed. I.e all _.dcm_ files are now _.png_ files.\n\n**Dataset link**\n\n - [RSNA-MICCAI-PNG-Dataset](https://www.kaggle.com/jonathanbesomi/rsna-miccai-png) [5.2 GB]\n\n**Script**\n\nFor each image, run: (Adapted from @tanlikesmath [work](https://www.kaggle.com/tanlikesmath/brain-tumor-radiogenomic-classification-eda))\n\n```\ndicom = pydicom.read_file(path)\ndata = apply_voi_lut(dicom.pixel_array, dicom)\nif dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\ndata = data - np.min(data)\ndata = data / np.max(data)\ndata = (data * 255).astype(np.uint8)\n```\n\n**More**\n\nIf you need other formats, you want a version with resized images or anything else, just ask ๐\n\nThank you for reading, happy kaggling to all!", "1395928": "Greate work.\n\nCould you please elaborate on the decision to normalize an images the following way?\n```\ndata = data - np.min(data)\ndata = data / np.max(data)\ndata = (data * 255).astype(np.uint8)\n```", "1389541": "Thanks for sharing", "1392053": "Thank you for sharing this..", "1471361": "Why do you use this normalization formula?\n`data = data - np.min(data)`\n `data = data / np.max(data)`\nI checked the original Dicom file, some local minimal values are not equal to the global minimum value, so is the maximum value. The global minimal value is -32768.0 and the global maximum value is 65535.0 (I only checked the converted dicom files). But the minimum value of \"Image-118.dcm\" is -32754.13895939086, and the maximum value of \"Image-118.dcm\" is 16424.83312182741. Your formula threats the -32754 as 0 and 16424 as 255, I think it's wrong.\nI am considering to use this formula:\n`global_min, global_max = -32768.0, 65535.0`\n`data = data - global_min`\n `data = data / global_max`\nThe converted images look similar, but they have different pixel values.", "1389296": "Thanks a lot!\nBut why using min-max normalize instead of normalize them using their full range (i.e. 0 to 2 ** bit_stored)??", "1388314": "Thanks for sharing! ~~Just for those who don't know the size of the images. I just download them and check. The size of the png images is 512 x 512.~~", "1388037": "Wow, this is amazing @jonathanbesomi! My computer has an SSD, so it's limited when it comes to space. Your solution will make this dataset way more accessible to people like me! Thanks a lot!", "1502064": "Thanks a lot for sharing, this is very kind of you.", "1392161": "Great work Thanks for sharing !", "1391397": "Thanks for sharing ! Extremely useful", "1390308": "Pretty amazing work, thank you for sharing with community.", "1389534": "Thank you very much for saving so much time and space.", "1388851": "Thanks for sharing .... magnificent reduce...", "1388074": "Firstly, many thanks for your help. \nI just had a beginner question, there are 4 subfolders with lots of images in each, for one data point or one patient, which one use for model training or which images to choose for model training", "1547031": "Thank you so much for doing this and sharing it with everyone! Really made my life easier!", "1540781": "\"data = (data * 255).astype(np.uint8)\" why using \"uint8\" dtype since decom image take value between 0 and 4096 ? dont we lose important information ? ", "1524936": "Thanks for the great data.\n\nI'm trying to insert a directory structure data into kaggle's dataset instead of a file, but I can't do it.\nHow did you manage to insert the data in a directory structure?\n\n\n", "1507500": "Thank you for sharing! It's a very useful to use by Colaboratory.", "1506049": "Thanks for the dataset! Do you have the full script for converting everything to PNG?", "1503215": "Thanks for sharing, helps with the training time :)", "1499113": "How did you filter out the empty DICOM images ? Could you share the script that you used to generate the dataset ?", "1485498": "How does this code actually work out deep down? I'm curious.", "1464832": "How would I write the result `data` to a `.png` file?", "1464692": "Apparently RE: this link https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/263612#1464675\n\nWe can't actually use the PNG dataset when submitting we have to convert from DICOM to PNG ourselves.\n\nRegardless this is very helpful, thank you!", "1461562": "Cool! \nBut it is necessary to understand, that DICOM contains not only an image, but also different additional data. [Here](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/252942) Peter showed how to extract it.", "1460162": "Awesome!!, Thank you so much for sharing this! :)", "1407313": "Cool. Thats really helpful", "1403601": "Hey,\n\nThank you for the effort but I am curious, doesn't this conversion compromise the Meta-Data? Also, our inputs would be DICOM images, right?", "1403028": "Hii, thank you for sharing your work with us. Can you also tell me how do I import the reduced dataset in google colab notebooks?", "1402521": "Excellent!", "1401825": "Thank you for sharing great & helpful information!", "1398191": "Thank you to share your knowledge! Great work!! ", "1395919": "Is there any link to the data where just format changed from dicom to png/jpeg, keeping other things exactly same, like not removing those empty image files etc ?\nIt would be a great help, thanks.", "1395298": "Thank for your help!!!\nCan you share more sizes? More sizes will be very interesting!", "1395281": "Thanks for sharing I was looking for this and this will be really helpful for me.", "1394957": "Great work.", "1394510": "Sincerely Thanks for sharing this it will really help us in training our model", "1391993": "Glad to see something I was looking for quite a while", "1391000": "That is brilliant. Thank you for time and energy-saving tip, John!", "1390999": "That is brilliant. Thank you for time and energy-saving tip, John!", "1389165": "Thanks for the transformation! any possibility of having the data in .nii.gz format? I am getting 3 different volumes for each image when applying dcm2nii", "1389162": "You saved my internet. And my bill. Thanks!", "1388350": "You have a place reserved in Heaven.", "1485282": "Request to add train label file too๐. Thanks for your efforts you made to convert the file in .png file. Because of your efforts, it's now practical possible for me to work with. ", "1400510": "", "1395003": "", "1393109": "", "1392050": "Thanks for the sharing. As the target is radiomics, is precision lost (from 16 bits to 8bits) can damage subtitles textural patterns ? :s ", "1389521": "", "1388414": "", "1538355": "Thanks a lot!", "1535553": "Thanks for sharing!", "1508350": "Thank you very much!", "1507707": "Thank you for sharing :) ", "1490981": "Thank you for sharing!", "1485310": "Thank you for sharing!!", "1477503": "Awesome work! Thank you for sharing!", "1473915": "thanks for sharing great work", "1473423": "Thank you for sharing!", "1473137": "Thanks for sharing! very helpful!", "1457486": "thank you so much\n", "1443214": "Thank you you saved my disk.... :)", "1400843": "Thanks a lot !", "1400319": "Thank you for sharing :)", "1400037": "Thank you Jonathan :D", "1397491": "Great and Thanks for sharing this .\n", "1396288": "Great! Thanks!", "1395033": "Thank you so much for sharing.", "1393896": "Thank you for such a nice work :)", "1391078": "Thank you for sharing." } }