{"cells":[{"metadata":{"trusted":true,"_uuid":"ae448b0ed29194053d40ebd29b2fa03982468552"},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nimport pylab\nimport numpy as np\nimport pydicom\nimport pandas as pd\nfrom glob import glob\nimport os\nfrom matplotlib.patches import Rectangle\n\ndatapath = '../input/'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a6d12a29f2498d51bc51c5621ec2ab2b60e99cbd"},"cell_type":"markdown","source":"# Lung Radiograph Images\nThe folder *stage_1_train_images* contains one image per patient, for a total of 25684 images."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"3b29d641de88c02088722c469a3b3831f4e335d1"},"cell_type":"code","source":"# counting the number of files in the image folder\n!ls ../input/stage_1_train_images/ | wc","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3c5877c08b0b5db659d1dd55113dd97d5b57f73c"},"cell_type":"markdown","source":"# Bounding Boxes and Target Label Data\nThe file *stage_1_train_labels.csv* contains the main training dataset, including the patiend ID, the bounding box coordinates, and the target label [0,1]. There can be multiple rows for the same patient ID, as each row corresponds to one observation (one box) per patient. \nThere are 28989 total boxes, and 25684 unique patient IDs. The negative/positive Target split is roughly 70-30%."},{"metadata":{"trusted":true,"_uuid":"fa5346f0e010257050379687a347a069a614f53a"},"cell_type":"code","source":"df_box = pd.read_csv(datapath+'stage_1_train_labels.csv')\nprint('Number of rows (unique boxes per patient) in main train dataset:', df_box.shape[0])\nprint('Number of unique patient IDs:', df_box['patientId'].nunique())\ndf_box.head(6)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c0f12a64d20fb44acff8fc3e52ed8630c5aab70"},"cell_type":"code","source":"df_box.groupby('Target').size().plot.bar()\nprint(df_box.groupby('Target').size() / df_box.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"13e42eb2bd668c47c91dccec70c818cf1b281631"},"cell_type":"markdown","source":"# Detailed Class Info Data\nThe file *stage_1_detailed_class_info.csv* contains detailed information about the positive and negative classes in the training set, and may be used to build more nuanced models. As in the main training dataset, this auxiliary dataset contains 28989 rows and 25684 unique patient IDs. \nThere's 3 classes: \n    1. Normal (29%)\n    2. No Lung Opacity / Not Normal (40%)\n    3. Lung Opacity (31%)\nThe first two classes correspond to Target = 0, whereas the third class correspond to Target = 1."},{"metadata":{"trusted":true,"_uuid":"f79065ff5ede4937b555d0dd3d3f9da486c83ab4"},"cell_type":"code","source":"df_aux = pd.read_csv(datapath+'stage_1_detailed_class_info.csv')\nprint('Number of rows in auxiliary dataset:', df_aux.shape[0])\nprint('Number of unique patient IDs:', df_aux['patientId'].nunique())\ndf_aux.head(6)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b3a937a2713093e270ddb3068cbb93db833e9bd1"},"cell_type":"code","source":"df_aux.groupby('class').size().plot.bar()\nprint(df_aux.groupby('class').size() / df_aux.shape[0])\nassert df_box.loc[df_box['Target']==0].shape[0] == df_aux.loc[df_aux['class'].isin(['Normal', \\\n    'No Lung Opacity / Not Normal'])].shape[0], 'Number of negative targets does not match between main and auxiliary dataset.'\nassert df_box.loc[df_box['Target']==1].shape[0] == df_aux.loc[df_aux['class'] == 'Lung Opacity'].shape[0], \\\n    'Number of positive targets does not match between main and auxiliary dataset.'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c5cc20e2eb8e9cbecf4494fe5264397438fb73a1"},"cell_type":"markdown","source":"# Merging Main (Boxes) and Auxiliary (Classes) Datasets\nThe main and auxiliary datasets do not share a joining keyword column, but it seems obvious that the rows are listed in the exact same order (check the patient Id columns to convince yourself), therefore we can combine the two dataframes by concatenating their columns."},{"metadata":{"trusted":true,"_uuid":"ca18783acf240faea13a23fd06fba7b41f9ea71d"},"cell_type":"code","source":"assert df_box['patientId'].values.tolist() == df_aux['patientId'].values.tolist(), 'PatientId columns are different.'\ndf_train = pd.concat([df_box, df_aux.drop(labels=['patientId'], axis=1)], axis=1)\ndf_train.head(6)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b1a1a6e3a7345d0f14e948147ac972f165fc450"},"cell_type":"markdown","source":"Just for peace of mind, we can check that there is a unique correspondence between Target and class labels."},{"metadata":{"trusted":true,"_uuid":"3eab5daaf119bbed48c0b1d88ae8b54121475849"},"cell_type":"code","source":"df_train.groupby(['class', 'Target']).size()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ef2c9148dec959b06313f09b8a1f29cff6113388"},"cell_type":"markdown","source":"NaN values are only present in the box coordinates columns, as expected."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"99659da131444b578b386314b1abf28621c867b2"},"cell_type":"code","source":"df_train.isnull().any()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e6555e378b682375c27e7f51f23820727b8c1694"},"cell_type":"markdown","source":"We can also make sure that positive targets are all associated with (non-NaN) box coordinates and viceversa."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"4b19fb1d29e6ea657a0fb4aa0320042e9f18cbfa"},"cell_type":"code","source":"# when target==1, are any of the box coordinates null? (should all be false)\ndf_train.loc[df_train['Target']==1, ['x', 'y', 'width', 'height']].isnull().any()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dc490e7944320cc9366927083e61f78e4de28cc1"},"cell_type":"code","source":"# when target==0, are all of the box coordinates null? (should all be true)\ndf_train.loc[df_train['Target']==0, ['x', 'y', 'width', 'height']].isnull().all()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8884679f0441b756e2237435ba788e846725d1a6"},"cell_type":"markdown","source":"# Radiograph Images\nThe radiograph images are stored in the folder *stage_1_train_images*. The images are saved in DICOM format (*.dcm*), which includes a header of meta-data and the raw pixel image itself. Images are named using their corrsponding patient ID. Images can be read in and modified using the library [pydicom](https://pydicom.github.io/). The headers of meta-data have been mostly anonymized for patient privacy, but they still contain a bunch of useful information that could be used to improve the classification model. The raw pixel images are stored in 1024x1024 8-bit encoded (=2^8=256 gray-scales) numpy arrays."},{"metadata":{"trusted":true,"_uuid":"217cd84d7db8c630e6ce69752853f3d1a659653e"},"cell_type":"code","source":"# sample of image filenames\n!ls -U ../input/stage_1_train_images/ | head -6","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dd1986328860f176e3fa01862b6ed6be87ca7c83"},"cell_type":"code","source":"# check that there is an image for each unique patient ID\nassert sorted(df_train['patientId'].unique().tolist()) == sorted([f[:-4] for f in os.listdir(datapath+'stage_1_train_images/')]), \\\n    'Discrepancy between patient IDs and radiograph images.'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4a7a42771e6c52d3ef4ac47f8cb9b2aeb48e03af"},"cell_type":"code","source":"# have a look at the header meta-data of an image \npId = df_train['patientId'].sample(1).values[0]    \ndcmdata = pydicom.read_file(datapath+'stage_1_train_images/'+pId+'.dcm')\nprint(dcmdata)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d28e276c47ad03ba1264c3ab84ebf242934a2718"},"cell_type":"code","source":"# extract the raw pixel image and look at its properties\ndcmimg = dcmdata.pixel_array\nprint(type(dcmimg))\nprint(dcmimg.dtype)\nprint(dcmimg.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"77436db5bde2957bac8bc02595f53a6c459cacc0"},"cell_type":"code","source":"# visualize the corresponding radiograph image\nplt.figure(figsize=(20,10))\nplt.imshow(dcmimg, cmap=pylab.cm.binary)\nplt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c0cec3b1313cdc96b564f84eb81892c0c7f5f1e2"},"cell_type":"markdown","source":"# Define utility functions for visualization\nBelow we define a bunch of useful functions to overlay images with boxes and labels."},{"metadata":{"trusted":true,"_uuid":"d832c4f159734b5e5c3f7b97ef550e913c7274d7"},"cell_type":"code","source":"def get_boxes_per_patient(df, pId):\n    '''\n    Given the dataset and one patient ID, \n    return an array of all the bounding boxes and their labels associated with that patient ID.\n    Example of return: \n    array([[x1, y1, width1, height1, class1, target1],\n           [x2, y2, width2, height2, class2, target2]])\n    '''\n    \n    boxes = df.loc[df['patientId']==pId][['x', 'y', 'width', 'height', 'class', 'Target']].values\n    return boxes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ae09da50fdcd33b96256cefb190171a79186adbc"},"cell_type":"code","source":"def get_dcm_data_per_patient(pId, sample='train'):\n    '''\n    Given one patient ID and the sample name (train/test), \n    return the corresponding dicom data.\n    '''\n    return pydicom.read_file(datapath+'stage_1_'+sample+'_images/'+pId+'.dcm')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4f6c6b6db74b7dc1f39bf38d4770d6045695ce9e"},"cell_type":"code","source":"def display_image_per_patient(df, pId, angle=0.0, sample='train'):\n    '''\n    Given one patient ID and the dataset,\n    display the corresponding dicom image with overlaying boxes and class annotation.\n    To be implemented: Optionally input the image rotation angle, in case of data augmentation.\n    '''\n    dcmdata = get_dcm_data_per_patient(pId, sample=sample)\n    dcmimg = dcmdata.pixel_array\n    boxes = get_boxes_per_patient(df, pId)\n    plt.figure(figsize=(20,10))\n    plt.imshow(dcmimg, cmap=pylab.cm.binary)\n    plt.axis('off')\n    \n    class_color_dict = {'Normal' : 'green',\n                        'No Lung Opacity / Not Normal' : 'orange',\n                        'Lung Opacity' : 'red'}\n\n    if len(boxes)>0:\n        for box in boxes:\n            # extracting individual coordinates and labels\n            x, y, w, h, c, t = box \n            # create a rectangle patch\n            patch = Rectangle((x,y), w, h, color='red', \n                              angle=angle, fill=False, lw=4, joinstyle='round', alpha=0.6)\n            # get current axis and draw rectangle\n            plt.gca().add_patch(patch)\n            \n    # add annotation text\n    plt.text(10, 50, c, color=class_color_dict[c], size=20, \n             bbox=dict(edgecolor=class_color_dict[c], facecolor='none', alpha=0.5, lw=2))\n            ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7cec38779658a4be5b53ad59f1005607ec5ae69f"},"cell_type":"code","source":"# run this cell multiple times to visualize different examples\npId = df_train['patientId'].sample(1).values[0]    \ndisplay_image_per_patient(df_train, pId, sample='train')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e8781223d16251bc021efb3cbe8024dcfb45720b"},"cell_type":"markdown","source":"# Extract useful meta-data from dicom headers\nWe can extract some information from the image headers and add it to the training dataset, so that we can check for possible correlations with the target."},{"metadata":{"trusted":true,"_uuid":"8b8bb3a9d3f0fbea4f7c53adfab009b9c13fd25f"},"cell_type":"code","source":"def get_metadata_per_patient(pId, attribute, sample='train'):\n    '''\n    Given a patient ID, return useful meta-data from the corresponding dicom image header.\n    Return: \n    attribute value\n    '''\n    # get dicom image\n    dcmdata = get_dcm_data_per_patient(pId, sample=sample)\n    # extract attribute values\n    attribute_value = getattr(dcmdata, attribute)\n    return attribute_value","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a6c89433fb606b2a5303fe13ca58d793b181fae9"},"cell_type":"code","source":"# create list of attributes that we want to extract (manually edited after checking which attributes contained valuable information)\nattributes = ['PatientSex', 'PatientAge', 'ViewPosition']\nfor a in attributes:\n    df_train[a] = df_train['patientId'].apply(lambda x: get_metadata_per_patient(x, a, sample='train'))\n# convert patient age from string to numeric\ndf_train['PatientAge'] = df_train['PatientAge'].apply(pd.to_numeric, errors='coerce')\n# remove a few outliers\ndf_train['PatientAge'] = df_train['PatientAge'].apply(lambda x: x if x<120 else np.nan)\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9e76c9353cd060a87039fc0cc1c1c9f65ba9ac10"},"cell_type":"code","source":"# look at age statistics between positive and negative target groups\ndf_train.drop_duplicates('patientId').groupby('Target')['PatientAge'].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3e34324b7a9ee1abf9e0b36ffc51f28f044def7c"},"cell_type":"code","source":"# look at gender statistics between positive and negative target groups\ndf_train.drop_duplicates('patientId').groupby(['PatientSex', 'Target']).size() / df_train.drop_duplicates('patientId').groupby(['PatientSex']).size()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"15bc5c4d8ce6a7eee5656508fe268e8af64af49c"},"cell_type":"code","source":"# look at patient position statistics between positive and negative target groups\ndf_train.drop_duplicates('patientId').groupby(['ViewPosition', 'Target']).size() / df_train.drop_duplicates('patientId').groupby(['ViewPosition']).size()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a0b303fec9db6ab5eac76fa8bf3feecac59340a2"},"cell_type":"code","source":"# absolute split of view position\ndf_train.groupby('ViewPosition').size()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a65869bfb7dbe41d1a29b6d270d34a71d4397b0f"},"cell_type":"markdown","source":"### **Age and gender - individually - do not seem to be correlated with the target. However, the view position of the radiograph image appears to be really important in terms of target split. AP means Anterior-Posterior, whereas PA means Posterior-Anterior. This [webpage](https://www.med-ed.virginia.edu/courses/rad/cxr/technique3chest.html) explains that \"Whenever possible the patient should be imaged in an upright PA position.  AP views are less useful and should be reserved for very ill patients who cannot stand erect\". One way to interpret this target unbalance in the ViewPosition variable is that patients that are imaged in an AP position are those that are more ill, and therefore more likely to have contracted pneumonia. Note that the absolute split between AP and PA images is about 50-50, so the above consideration is extremely significant. **"},{"metadata":{"_uuid":"55ab597e4a2c2b307a2a21dbfc26114d139eb77c"},"cell_type":"markdown","source":"# Extract Test Images Metadata"},{"metadata":{"trusted":true,"_uuid":"b1be28431d5eb9831d2d765fedd004c28f434aef"},"cell_type":"code","source":"patientIDs_test = [f[:-4] for f in os.listdir(datapath+'stage_1_test_images/')]\ndf_test = pd.DataFrame(data={'patientId' : patientIDs_test})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"38675421d5390a29e8d0c2f4c0cfa39f51443da9"},"cell_type":"code","source":"attributes = ['PatientSex', 'PatientAge', 'ViewPosition']\nfor a in attributes:\n    df_test[a] = df_test['patientId'].apply(lambda x: get_metadata_per_patient(x, a, sample='test'))\n# convert patient age from string to numeric\ndf_test['PatientAge'] = df_test['PatientAge'].apply(pd.to_numeric, errors='coerce')\n# remove a few outliers\ndf_test['PatientAge'] = df_test['PatientAge'].apply(lambda x: x if x<120 else np.nan)\ndf_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4f3a162f88d441e3ca7f60ae35f9f4badb3ef6ad"},"cell_type":"code","source":"# save dataframes\ndf_train.to_csv('train.csv', index=False)\ndf_test.to_csv('test.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"acb7a5cee577ec0f8d5bef561d0155065b940121"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}