{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nimport tqdm\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\n\nimport matplotlib.pyplot as plt\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nOpen Source Imaging Consortium (OSIC) is a not-for-profit, co-operative effort between academia, industry and philanthropy. In this competition we will predict a patients severity of lung decline based on a CT scan of their lungs. You’ll determine lung function based on output from a spirometer, which measures the volume of air inhaled and exhaled. The challenge is to use machine learning techniques to make a prediction with only the images as input. This high level description comes straight from the competition's overview page (https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/overview).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Data\n\nIn the dataset, you are provided with a baseline chest CT scan and associated clinical information for a set of patients. A patient has an image acquired at time Week = 0 and has numerous follow up visits over the course of approximately 1-2 years, at which time their FVC (forced vital capacity) is measured.\n\nIn the training set, you are provided with an anonymized, baseline CT scan and the entire history of FVC measurements.\nIn the test set, you are provided with a baseline CT scan and only the initial FVC measurement. You are asked to predict the final three FVC measurements for each patient, as well as a confidence value in your prediction.\n\nSince this is real medical data, you will notice the relative timing of FVC measurements varies widely. The timing of the initial measurement relative to the CT scan and the duration to the forecasted time points may be different for each patient. This is considered part of the challenge of the competition. To avoid potential leakage in the timing of follow up visits, you are asked to predict every patient's FVC measurement for every possible week. Those weeks which are not in the final three visits are ignored in scoring.\n\nThe data for the CT scan images are provided in DICOM format.\n\n## DICOM Format\nDigital Imaging and Communications in Medicine (DICOM) is the accepted standard for the communication and management of medical imaging information. DICOM is used for archiving and transmitting medical images. It enables the integration of medical imaging devices (radiological scanners), servers, network hardware and Picture Archiving and Communication Systems (PACS). The standard was widely adopted by hospitals and research centers and is steadly advancing as well toward small practice and cliniques.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Visualizing DICOM Images\n\nBefore we visualize the DICOM images we will import some packages and create a utility function for visualizing the images. The packages that we will import to load the DICOM images is `pydicom`. To get a list of the images in a directory we will use the `glob` package. To visualize the images we will use `greyscale` format.\n\nIf we look at the length of the image files returned we find there are 33,026 images in the `train` folder. For a quick visualization we will only look at the first 12 images returned by the `glob` function.","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pydicom\nimport glob\nimport os\nfrom typing import Dict, List","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_osic_images(image_files: List[str]) -> None:\n    # Take only the first 12 images in the list\n    image_files = image_files[:12]\n    \n    fig, axes = plt.subplots(4, 3, figsize=(20, 16))\n    axes = axes.flatten()\n    for image_index, image_file in enumerate(image_files):\n        # Load the DICOM image and convert to pixel array\n        image_data = pydicom.read_file(image_file).pixel_array\n        axes[image_index].imshow(image_data, cmap=plt.cm.bone)\n        \n        image_name = '-'.join(image_file.split('/')[-2:])\n        axes[image_index].set_title(f'{image_name}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_image_path = '/kaggle/input/osic-pulmonary-fibrosis-progression/train'\ntrain_image_files = glob.glob(os.path.join(train_image_path, '*', '*.dcm'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_osic_images(train_image_files)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# `pydicom` package\n\nThere is more than meets the eye with loading the dicom images using `pydicom`. `pydicom` also makes it easy to read meta-data from the dicom files. More information can be found here https://pydicom.github.io/pydicom/stable/old/getting_started.html. For example if we load in an image, instead of converting it to a pixel array straight away we can view more characteristics of the file. We can use this information to further inhance our knowledge of the images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"image_data = pydicom.read_file(train_image_files[0])\nimage_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Different calls in the image import\nimage_data.PatientName, image_data.Modality, image_data.BodyPartExamined","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training CSV\n\nWe also have meta-data contained in `/kaggle/input/osic-pulmonary-fibrosis-progression/train.csv` and `/kaggle/input/osic-pulmonary-fibrosis-progression/test.csv`. We will take a look at the contents. We are given 7 fields in `train.csv`.\n\n1. Patient - patient id\n2. Weeks - the relative number of weeks pre/post the baseline CT (may be negative)\n3. FVC (Forced Vital Capacity) - the recorded lung capacity in ml\n4. Percent - a computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics\n5. Age - age of the patient\n6. Sex - sex of the patient\n7. SmokingStatus - The status of the patient relative to smoking - the three categories are: `['Ex-smoker', 'Never smoked', 'Currently smokes']`\n\nMost of these definitions can also be found on the data page: https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/osic-pulmonary-fibrosis-progression/train.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Counts of the Weeks field. We will look at the top 20 weeks contained in the train.csv\n\nweeks_frequency = train_df['Weeks'].value_counts().head(20)\nweeks_frequency = weeks_frequency.reset_index()\nweeks_frequency = weeks_frequency.rename(columns={'index': 'Weeks', 'Weeks': 'Frequency'})\n\nplt.figure(figsize=(10, 7))\nax = sns.barplot(x='Weeks', y='Frequency', data=weeks_frequency, order=weeks_frequency['Weeks'])\nax.set_title('Top Weeks by Frequency')\nplt.grid()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Histogram of the Age field\n\nplt.figure(figsize=(10, 7))\nax = sns.distplot(train_df['Age'])\nax.set_title('Histogram for Age')\nplt.grid()\n\nprint(train_df['Age'].describe())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sex_frequency = train_df['Sex'].value_counts()\nsex_frequency = sex_frequency.reset_index()\nsex_frequency = sex_frequency.rename(columns={'index': 'Sex', 'Sex': 'Frequency'})\n\nplt.figure(figsize=(10, 7))\nax = sns.barplot(x='Sex', y='Frequency', data=sex_frequency, order=sex_frequency['Sex'])\nax.set_title('Sex Barplot')\nplt.grid()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"smoking_status_frequency = train_df['SmokingStatus'].value_counts()\nsmoking_status_frequency = smoking_status_frequency.reset_index()\nsmoking_status_frequency = smoking_status_frequency.rename(columns={'index': 'SmokingStatus', 'SmokingStatus': 'Frequency'})\n\nplt.figure(figsize=(10, 7))\nax = sns.barplot(x='SmokingStatus', y='Frequency', data=smoking_status_frequency, order=smoking_status_frequency['SmokingStatus'])\nax.set_title('Smoking Status Barplot')\nplt.grid()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Histogram of the FVC field\n\nplt.figure(figsize=(10, 7))\nax = sns.distplot(train_df['FVC'])\nax.set_title('Histogram for FVC')\nplt.grid()\n\nprint(train_df['FVC'].describe())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Histogram of the FVC field\n\nplt.figure(figsize=(10, 7))\nax = sns.distplot(train_df['Percent'])\nax.set_title('Histogram for Percent')\nplt.grid()\n\nprint(train_df['Percent'].describe())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Observations - Part 1\n\n* For the weeks field we can see that the highest frequency is Weeks=8 followed by Weeks=12, etc. I will investigate more as I progress towards modeling.\n* The mean age is ~67 years old with a standard deviation of ~7 years. The minimum age in the data set is ~49 years old while the maximum age is ~88 years old.\n* There are more male patients than female patients in the training data.\n* For the smoking status there are a lot of ex-smokers in the training data. Then the next largest group is non-smokers, followed by currently smoking. Interesting that the training data has a lot of ex-smokers.\n* The average FVC is ~2690 ml with a standard deviation of ~832 ml. The minimum FCV value we find in the training data is ~827 ml while the max is ~6399 ml.\n* The average percent is ~77 with a standard deviation of 19. The minimum is ~28 while the max is ~153.\n\nI generally try to just keep stats around later on when it comes to the modeling phase.\n\nLet's look as some more detailed analysis and see how our variables interact with one another.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Advanced EDA & Plots\n\nHere we will consider a multi-variate analysis and look at different features and their interations. The goal of our analysis is to see if we can find any useful relationships or anything that sticks out about the data through visualization and digging into the data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 2, figsize=(20, 10))\naxes = axes.flatten()\n\n# Distribution for Age\nage_male = train_df.loc[train_df['Sex'] == 'Male']['Age']\nage_female = train_df.loc[train_df['Sex'] == 'Female']['Age']\n\nsns.kdeplot(age_male, label='Male', shade=True, ax=axes[0])\nsns.kdeplot(age_female, label='Female', shade=True, ax=axes[0])\n\naxes[0].legend()\naxes[0].set_title('Age Distribution by Sex')\naxes[0].set_xlabel('Age')\naxes[0].grid()\n\n# Distribution for Smoking Status\nage_ex_smoker = train_df.loc[train_df['SmokingStatus'] == 'Ex-smoker']['Age']\nage_never_smoked = train_df.loc[train_df['SmokingStatus'] == 'Never smoked']['Age']\nage_currently_smoking = train_df.loc[train_df['SmokingStatus'] == 'Currently smokes']['Age']\n\nsns.kdeplot(age_ex_smoker, label='Ex-Smoker', shade=True, ax=axes[1])\nsns.kdeplot(age_never_smoked, label='Never Smoked', shade=True, ax=axes[1])\nsns.kdeplot(age_currently_smoking, label='Currently Smokes', shade=True, ax=axes[1])\n\naxes[1].legend()\naxes[1].set_title('Age Distribution by Smoking Status')\naxes[1].set_xlabel('Age')\naxes[1].grid()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* From the above age distributions we start to learn a little more about our data. For the most part the age distribution between male and female look pretty simlilar but we can see that there is a higher concentration around the 71 age mark. So there seem to be older males in the training data while females are a bit younger. \n\n* The ex-smokers look like their concentration is above 70 years old, while for the non smokers the main concentation is around the 65 year old mark. The currently smokes is interesting because from the kdeplot we can see four different modes with the highest concentration happening around 70 years old.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lmplot(x='Age', y='FVC', hue='Sex', col='Sex', data=train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* There seems to be a small decresing relationship between age and FVC for the male category.\n* There is also a small decreasing relationshop between the age and FVC for the female category although more subtle compared with the male category.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lmplot(x='Age', y='FVC', hue='SmokingStatus', col='SmokingStatus', data=train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Investigating the age vs. FVC we can see that there is also a decreasing relationship between age and GVC for ex-smokers as well as for currently smokes categories.\n* There is a positive relationship between age and FVC for people who have never smoked.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lmplot(x='Age', y='Percent', hue='Sex', col='Sex', data=train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* For age vs. percent for the male cateory we see that there is a slight positive relationship.\n* for the female category we see that there is a stronger positive relationship between age and percent.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lmplot(x='Age', y='Percent', hue='SmokingStatus', col='SmokingStatus', data=train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* When we look at age vs. percent by the patients smoking status we observe a slight positive relationship between age and percent for ex-smokers.\n* There is a strong positive relationship between age and percent for patients who have never smoked.\n* There is a negative relationship between age and percent for people who are currenlty smoking.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lmplot(x='Age', y='Weeks', hue='Sex', col='Sex', data=train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* For age vs weeks we see a slight negative relationship for both the male and female category.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lmplot(x='Age', y='Weeks', hue='SmokingStatus', col='SmokingStatus', data=train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* There seems to be a strong negative relationship between age and weeks for people who are currently smoking.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Extract Meta-Data from DICOM Images\n\nLet's see if we can also extract more information from the images themselves. We saw above that there is meta-data included with the dicom images. Let's look at an example again.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"image_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def extract_dicom_meta_data(filename: str) -> Dict:\n    # Load image\n    image_data = pydicom.read_file(train_image_files[0])\n    \n    row = {\n        'Patient': image_data.PatientID,\n        'body_part_examined': image_data.BodyPartExamined,\n        'image_position_patient': image_data.ImagePositionPatient,\n        'image_orientation_patient': image_data.ImageOrientationPatient,\n        'photometric_interpretation': image_data.PhotometricInterpretation,\n        'rows': image_data.Rows,\n        'columns': image_data.Columns,\n        'pixel_spacing': image_data.PixelSpacing,\n        'window_center': image_data.WindowCenter,\n        'window_width': image_data.WindowWidth\n    }\n    \n    return row","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"meta_data_df = []\nfor filename in tqdm.tqdm(train_image_files):\n    meta_data_df.append(extract_dicom_meta_data(filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert to a pd.DataFrame from dict\nmeta_data_df = pd.DataFrame.from_dict(meta_data_df)\nmeta_data_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Looks now like we have extended our dataset and added more meta-data! I am not quite sure yet if any of this data is useful but I will try to explore it when I get a chance.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# More EDA ...\n\nThanks for taking a peek at what I have so far. I will prepare more analysis and utility functions soon.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}