{"cells":[{"metadata":{},"cell_type":"markdown","source":"# ☑️ Overview:\n\n### 🥅 Investigate and create a DataFrame for the metadata for the 33,026 CT images associated with the 176 patients in Training set.\n\n### 🔗 Link to my [DICOM Metadata DataFrame](https://www.kaggle.com/samuellongenbach/osic-pulmonary-fibrosis-dicom-metadata-pkl)\n\n### 👏 Thanks to the following notebooks for their ideas & code:  \n\n* [What should we consider when handling DICOM?](https://www.kaggle.com/jryoungw/what-should-we-consider-when-handling-dicom) by [jryoungw](https://www.kaggle.com/jryoungw)\n\n* [Pulmonary Fibrosis Competition: EDA & DICOM Prep](https://www.kaggle.com/andradaolteanu/pulmonary-fibrosis-competition-eda-dicom-prep) by [andradaolteanu](https://www.kaggle.com/andradaolteanu) \n\n* [Understanding DICOMS✔](https://www.kaggle.com/avirdee/understanding-dicoms) by [avirdee](https://www.kaggle.com/avirdee)\n\n\n\n### 👍 If you find the notebook or dataset helpful, suggestions & a upvote is appreciated!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 📚 Libraries:","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"##\nimport numpy as np\nimport pandas as pd\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n##\nimport pydicom ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 📊Load CSV Data:\n\n### Key Findings:\n* 176 Patients in the training set.\n* 5 Patients in the testing set which also show up in training set.\n\n* Each patient has a 3D baseline CT scan which is stored as a collection of 2D images.\n* We also refer to these images as slices.\n* The number of 2D images for each of the Patients varies from 12 to 1018 images.  ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/train.csv\")\ntest_df = pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/test.csv\")\n\ntrain_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"# of Patients in Train: \",len(np.unique(train_df[\"Patient\"])))\nprint(\"# of Patients in Test: \",len(np.unique(test_df[\"Patient\"])))\nprint(\"Train/Test overlap?: \",len(np.intersect1d(train_df[\"Patient\"],test_df[\"Patient\"])))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Base directory for Train .dcm files\nosic_dir = \"../input/osic-pulmonary-fibrosis-progression/train/\"\n\ntrain_df[\"Path\"] = osic_dir + train_df[\"Patient\"] \n\n# Calculate how many CT images each patient has\ntrain_df[\"CT_images\"] = 0\n\nfor k, path in enumerate(train_df[\"Path\"]):\n    train_df[\"CT_images\"][k] = len(os.listdir(path))\n\ntrain_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# CT Scans per Patient\ndata = train_df.groupby(by=\"Patient\")[\"CT_images\"].first().reset_index(drop=False)\n\n# Sort by number of CT Scans\ndata = data.sort_values(['CT_images']).reset_index(drop=True)\nprint(\"Minimum number of CT images: {}\".format(data[\"CT_images\"].min()), \"\\n\" +\n      \"Maximum number of CT images: {}\".format(data[\"CT_images\"].max()), \"\\n\" +\n      \"Median number of CT images: {}\".format(data[\"CT_images\"].median()))\n\n# Plot\nplt.figure(figsize = (16, 6))\np = sns.barplot(data[\"Patient\"], data[\"CT_images\"], color=\"darkgreen\")\nplt.axvline(x=85, color=\"lightgreen\", linestyle='--', lw=3)\n\nplt.title(\"Number of CT images in baseline for each Patient\", fontsize = 17)\nplt.xlabel('Patient', fontsize=14)\nplt.ylabel('Frequency', fontsize=14)\n\nplt.text(86, 850, \"Median=98\", fontsize=13)\n\np.axes.get_xaxis().set_visible(False);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ⚕️Collect DICOM Metadata:","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Example:\n* For Patient -> ID00122637202216437668965 we load all the 2D slices using **pydicom.dcmread**.\n* Below we display the metadata for one of the slices:\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"path = \"../input/osic-pulmonary-fibrosis-progression/train/ID00122637202216437668965\"\n\nslices = [pydicom.dcmread(path + '/' + s) for s in os.listdir(path)] \nslices.sort(key = lambda x: int(x.InstanceNumber)) \n\nprint(\"Patient: \",\"ID00122637202216437668965\")\nprint(\"Image/Slice: \",slices[36][\"InstanceNumber\"].value)\nslices[36]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Metadata for a Patient:\n* **Metadata_for_Patient** creates a DataFrame with all the metadata for a given Patient. \n* If the DICOM attribute doesn't exist, append np.nan.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dicom_atts = [\"SpecificCharacterSet\",\"ImageType\",\"SOPInstanceUID\",\"Modality\",\"Manufacturer\",\"ManufacturerModelName\",\"PatientName\",\"PatientID\",\n             \"PatientSex\",\"DeidentificationMethod\",\"BodyPartExamined\",\"SliceThickness\",\"KVP\",\"SpacingBetweenSlices\",\"DistanceSourceToDetector\",\"DistanceSourceToPatient\",\"GantryDetectorTilt\",\n             \"TableHeight\",\"RotationDirection\",\"XRayTubeCurrent\",\"GeneratorPower\",\"FocalSpots\",\"ConvolutionKernel\",\"PatientPosition\",\"RevolutionTime\",\"SingleCollimationWidth\",\"TotalCollimationWidth\",\"TableSpeed\",\"TableFeedPerRotation\",\"SpiralPitchFactor\",\n              \"StudyInstanceUID\",\"SeriesInstanceUID\",\"StudyID\",\"InstanceNumber\",\"PatientOrientation\",\"ImagePositionPatient\",\"ImageOrientationPatient\",\"FrameOfReferenceUID\",\"PositionReferenceIndicator\",\"SliceLocation\",\"SamplesPerPixel\",\"PhotometricInterpretation\",\n             \"Rows\",\"Columns\",\"PixelSpacing\",\"BitsAllocated\",\"BitsStored\",\"HighBit\",\"PixelRepresentation\",\"PixelPaddingValue\",\"WindowCenter\",\"WindowWidth\",\"RescaleIntercept\",\"RescaleSlope\",\"RescaleType\"]\n\nlist_attributes = [\"ImageType\",\"ImagePositionPatient\",\"ImageOrientationPatient\",\"PixelSpacing\"]\n\ndef Metadata_for_Patient(folder_path):\n    files = os.listdir(folder_path)\n    patient_id = folder_path.split('/')[-1]\n    \n    ## Each row is an image file:\n    base_data = {'Patient': [patient_id]*len(files), 'File': files}\n    patient_df = pd.DataFrame(data=base_data)\n    \n    ## Add Columns by looping through DICOM attributes for each image file:\n    slices = [pydicom.dcmread(folder_path + '/' + s) for s in files] \n    for d in dicom_atts:\n        attribute_i = []\n        for s in slices:\n            try:\n                attribute_i.append(s[d].value)\n            except:\n                attribute_i.append(np.nan)\n        patient_df[d] = attribute_i\n        \n    ## Store min pixel value for each image file \n    attribute_min_pixel = []\n    for s in slices:\n        try:\n            mp = np.min(s.pixel_array.astype(np.int16).flatten())\n        except:\n            mp = np.nan\n        attribute_min_pixel.append(mp)\n    patient_df[\"MinPixelValue\"] = attribute_min_pixel\n  \n    return patient_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Metadata_for_Patient(path).head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 💾 Create DataFrame for all Patients:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"DICOM_Meta_df = pd.DataFrame()\n\n## For all 176 Patient Folders: \nunique_patient_df = train_df.groupby(by=\"Patient\").first()\nfor pth in unique_patient_df[\"Path\"]:\n    temp_df = Metadata_for_Patient(pth)\n    DICOM_Meta_df = pd.concat([DICOM_Meta_df,temp_df],ignore_index=True)\n    \n## SAVE:\nDICOM_Meta_df.to_pickle(\"DICOM_Metadata.pkl\")\n\n## LOAD:\n#load_df = pd.read_pickle(\"DICOM_Metadata.pkl\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 💾 Metadata EDA:\n\n* To be done...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"## All Images:\nprint(\"Shape: \", DICOM_Meta_df.shape)\nDICOM_Meta_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## All Patients:\nunique_meta_df = DICOM_Meta_df.groupby(by=\"Patient\").first()\nunique_meta_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unique_meta_df.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### CT Image Issues:\n* Patient -> ID00011637202177653955184 can't load 31 images\n* Patient -> ID00052637202186188008618 can't load 1 images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"issues = DICOM_Meta_df[DICOM_Meta_df[\"MinPixelValue\"].isna()]\nissues","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"issues[\"Patient\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"issue_path = \"../input/osic-pulmonary-fibrosis-progression/train/ID00011637202177653955184/6.dcm\"\n\nfff = pydicom.dcmread(issue_path) \nfff.pixel_array","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}