{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\n**Pulmonary fibrosis** occurs when scarred lung tissue complicates and diminishes the ability to breathe. Prognosis varies from patient to patient and the reason for these differences are unknown. Using CT scans provided by the **Open Source Imaging Consortium (OSIC)** in addition to patient metadata, the objective is to predict the decline in breathing capacity as measured by **Forced Vital Capacity**, or **FVC**.\n\nIn this notebook, I perform exploratory data analysis (EDA) on the OSIC Pulmonary Fibrosis Progression dataset. I tried to keep everything accessible to data science newcomers - upvotes and/or comments are greatly appreciated! 😄\n\n---","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Reading in Patient Metadata\n\nIn this section, I explore the tabular data pertaining to the patients' demographic and clinical information. First, I import the metadata from the provided csv files and take a look at its contents and overall structure. For reference, here are the columns contained in `train.csv` and `test.csv`:\n\n* `Patient`- a unique Id for each patient (also the name of the patient's DICOM folder)\n* `Weeks`- the relative number of weeks pre/post the baseline CT (may be negative)\n* `FVC` - the recorded lung capacity in ml\n* `Percent`- a computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics\n* `Age`\n* `Sex`\n* `SmokingStatus`\n\nBefore diving in to the data we need to import the following packages:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# import libraries\nimport os\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To list what is contained in a directory, we can use os.listdir()","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(\"../input/osic-pulmonary-fibrosis-progression\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this notebook, we will only concern ourselves with 'train.csv', which contains the patient epidemiological data and the 'train' subdirectory, which contains the patients' CT scans. Now let's read in 'train.csv' and begin to understand our data!","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# import metadata csv files as pandas DataFrames (DFs)\ntrain_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\n\n# size of the DF\nprint(train_df.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our training data consists of 1549 rows and 7 columns. Let's take a look at the first few rows:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can get additional information, such as the column data types and whether they contain null values using the .info() method:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So we have `Patient`, `Sex` and `SmokingStatus` being of type object (strings), `Weeks`, `FVC` and `Age` being integers and `Percent` as float (decimal) values.\n\nAnother useful way of getting information from a DF is by using .describe(). This method returns summary statistics such as the mean, minimum and maximum - note that returns output only for the numeric columns","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can get quite a bit of insight from above - taking `Age` for example, we can see that the youngest is 42, the oldest is 88 and the average patient is ~67 years old.\n\nNotice when we returned the first few rows of our DF, it showed the same `Patient` multiple times (for different follow-up visits). Let's find out how many unique patients there are:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# print the number of unique patient IDs\nn_patients = train_df['Patient'].nunique()\nprint(f'There are {n_patients} unique patients in the training dataset')\nprint('===============================')\n\n# print the number of rows for each unique patient ID\nprint(train_df['Patient'].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So there are 176 unique patients in the train DF. We can also note that not all patients have the same number of observations from the output of .value_counts(). From the output we might get the impression that many patients had 10 clinical visits and only a few had 7 or 6 visits, but to find out exactly the distribution of the number of visits, we can use .value_counts().value_counts() 😎","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# get the distribution of number of visits\nprint(train_df['Patient'].value_counts().value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This tells us explicitly that there are 9 rows (visits) for the majority of patients.\n\nSince the `Age`, `Sex` and `SmokingStatus` columns are repeated for rows of the same patient, we need to drop these duplicates before exploring these variables. However, the patients have numerous follow up visits over the course of ~1-2 years, so it's possible that `Age` for the same patient could have different values. Let's check whether `Age` contains different values for each patient: ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# for each patient, get the number of unique Age values\nprint(train_df.groupby('Patient')['Age'].nunique())\nprint('===============================')\n\n# get unique values of the output from above\nprint(train_df.groupby('Patient')['Age'].nunique().unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`Age` is constant for every patient, great! Now we create a new DF for individual patients and confirm the shape to have 176 rows:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df = train_df[['Patient', 'Age', 'Sex', 'SmokingStatus']].drop_duplicates()\npatient_df.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EDA: Patient Demographics\nNow we can start exploring the patient demographics! Let's look at the distributions of `Age`, `Sex` and `SmokingStatus`.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(patient_df['Age'])\nplt.title('Distribution of Age', size=20)\nplt.xlabel('');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`Age` has a roughly normal distribution with a large proportion of patients being 60 to 75 years old.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"f, axes = plt.subplots(1,2, figsize=(12, 4))\n\nsns.countplot(patient_df['Sex'], ax=axes[0], palette='muted')\naxes[0].set_title('Sex', size=20)\naxes[0].set_xlabel('')\n\nsns.countplot(patient_df['SmokingStatus'], ax=axes[1], palette=sns.color_palette(\"Set2\"))\naxes[1].set_title('Smoking Status', size=20)\naxes[1].set_xlabel('');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The patients are predominantly male and mainly consist of ex-smokers. Patients who never smoked are the second most common SmokingStatus group and current smokers make up the smallest group.\n\nWe now have a sense of the distributions of each categorical variable. We can also explore them with respect to each other! For example, let's look at how `Sex` is distributed with respect to each `SmokingStatus` group:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"g = sns.FacetGrid(patient_df, col=\"SmokingStatus\", hue='Sex', palette='muted', height=4)\ng.map(sns.countplot, 'Sex', order=['Male','Female'])\ng.axes[0, 0].set_ylabel('Number of Patients', size=14)\ng.set_titles(size=14, fontweight='bold', col_template=\"{col_name}\")\ng.fig.subplots_adjust(top=0.8)\ng.fig.suptitle('Distribution of Sex Across Smoking Groups', fontsize=20);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ex-smokers and current smokers are predominantly male while those who never smoked are split roughly evenly by gender. We can also look at how `Age` distributed across `SmokingStatus` groups: ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.violinplot(x='SmokingStatus', y='Age', data=patient_df, hue=\"Sex\", palette='muted', split=True)\nplt.title('Age Distributions Across Smoking Groups', size=16)\nplt.legend(loc='lower left', ncol=2);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Generally, patient `Age` distributions are similar across `SmokingStatus` groups with a mean of around 65-70 years. The exceptions are females who currently or used to smoke whose `Age` distributions tend to be more spread out, but this may be due to the few number of patients within these groups. \n\n# EDA: Patient Clinical Data\nWe now have a fairly good understanding of our patients' demographics. Let's investigate the numeric variables or the patients' clinical data, starting with the `Weeks` variable!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(train_df['Weeks'])\nplt.title('Distribution of Weeks', size=20);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The `Weeks` distribution is considerably skewed to the right. We can see that most data points fall within 75 weeks or roughly 1.5 years. Let's shift our attention to the Forced Vital Capicity or `FVC` and `Percent` variables. First, I compare these metrics between different `Sex` and also across `SmokingStatus` groups: ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.swarmplot(x='SmokingStatus', y='FVC', data=train_df, hue='Sex', dodge=True, size=3, palette='muted')\nplt.title('Males tend to Have Greater FVC than Females', size=18);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Males tend to have greater `FVC`s than females, as one might expect. However, rather surprisingly, FVC does not appear to be associated with `SmokingStatus`.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.swarmplot(x='SmokingStatus', y='Percent', data=train_df, hue='Sex', dodge=True, size=3, palette='muted')\nplt.title('Percent is Comparable Between\\nSex & Across Smoking Groups', size=18);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`Percent` is a calculated metric that quantifies one's FVC with respect to the typical FVC of their demographic. We can see the effect of this normalization above where the FVC distributions between males and females are now similar. There are also a few data points among the current smokers that have markedly higher Percent values than the rest of the cohort.\n\nLet's explore the relationships between our numeric variables using the quick and dirty sns.pairplot():","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"g = sns.pairplot(train_df, hue='SmokingStatus', palette=sns.color_palette('Set2'), corner=True, plot_kws={'alpha': 0.5})\ng._legend.set_bbox_to_anchor((0.7, 0.7));","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There does not seem to be any apparent associations between these variables aside from `FVC` and `Percent` which are positively correlated.\n\nTracking a patient's `FVC` over time is central to the objective of this competition. We can visualize individual FVC dynamics of individual patients like so:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.pointplot(x='Weeks', y='FVC', data=train_df[train_df['Patient']=='ID00007637202177411956430']);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"However, what if we wanted to compare changes in FVC between different patient groups? For example, we can plot the average `FVC` over time of each `SmokingStatus` group:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# subset data for weeks<100 since there are very few observations past this point\nsns.lineplot(x='Weeks', y='FVC', data=train_df[train_df['Weeks']<100], hue=\"SmokingStatus\", palette=sns.color_palette('Set2', n_colors=3))\nplt.legend(bbox_to_anchor=(1.05, 1))\nplt.title('Change in FVC Over Time', size=18);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"No differences in FVC progression between smoking groups are apparent. But note that FVC can vary between individuals - this noise could be masking any trend that may be present. We can look at FVC progression in relative terms by calculating the % change in FVC:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['FVC_%Change'] = train_df.groupby('Patient')['FVC'].pct_change().fillna(0)\nsns.lineplot(x='Weeks', y='FVC_%Change', data=train_df[train_df['Weeks']<=100], hue=\"SmokingStatus\", palette=sns.color_palette('Set2', n_colors=3))\nplt.title('% Change in FVC Over Time');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Although the % change in FVC are less variable compared to FVC, we still do not observe any notable differences in averaged FVC progression between smoking groups.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"---","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}