{"cells":[{"metadata":{},"cell_type":"markdown","source":"\n

OSIC Pulmonary Fibrosis Progression - EDA

\n \n# 1. Introduction 🃏 \n \n### 1.1 What is Pulmonary fibrosis?\n* [Pulmonary fibrosis is a lung disease that occurs when lung tissue becomes damaged and scarred.](https://www.mayoclinic.org/diseases-conditions/pulmonary-fibrosis/symptoms-causes/syc-20353690) This thickened, stiff tissue makes it more difficult for your lungs to work properly. If you want to know further about this type lung disease, I have linked below an informative video.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from IPython.display import HTML\nHTML('
')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1.2 What is OSIC Pulmonary Fibrosis Progression Competition?\n- In this competition, you’ll predict a patient’s severity of decline in lung function based on a CT scan of their lungs. You’ll determine lung function based on output from a spirometer, which measures the volume of air inhaled and exhaled. The challenge is to use machine learning techniques to make a prediction with the image, metadata, and baseline FVC as input.\n \n \n### 1.3 What we need to do? Observation\nWe will predict a patient’s severity of decline in lung function based on a CT scan of their lungs.\nIn other words, We will predict the final three FVC measurements for each patient, as well as a confidence value in your prediction.\n \n- The leaderboard of this competition is calculated with approximately 1%->15% of the test data. The final results will be based on the other 99%->85%, so the final standings may be different.\n \n### 1.4 Metric: Laplace Log Likelihood\n![](https://i.imgur.com/tEIZvli.png)\n- Image Credits: https://en.wikipedia.org/wiki/Laplace_distribution\n \n- The evaluation metric of this competition is a modified version of Laplace Log Likelihood. \nPredictions are evaluated with a modified version of the Laplace Log Likelihood. For each sample in test set, an `FVC` and a `Confidence` measure (standard deviation σ) has to be predicted.\n\n `Confidence` values smaller than 70 are clipped.\n\n $\\large \\sigma_{clipped} = max(\\sigma, 70),$\n\n Errors greater than 1000 are also clipped in order to avoid large errors.\n\n $\\large \\Delta = min ( |FVC_{true} - FVC_{predicted}|, 1000 ),$\n\n The metric is defined as:\n\n $\\Large metric = - \\frac{\\sqrt{2} \\Delta}{\\sigma_{clipped}} - \\ln ( \\sqrt{2} \\sigma_{clipped} ).$\n\n\nRead more about it on the [Evaluation Page](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/overview/evaluation).\n\nIf you feel this was something new and fresh, and it added some value to you, please consider upvoting, it motivates to keep writing good kernels. 😄","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Contents \n\n\n\n* [Basic Exploratory Data Analysis](#1) \n * [Getting started - Importing libraries]()\n * [Reading the train.csv]()\n \n \n* [Data Exploration](#2) \n * [Check Train & Test Info.]()\n * [Unique Patients(Ids)]()\n * [Exploring the 'SmokingStatus' column]()\n * [Weeks distribution]()\n * [FVC - The forced vital capacity]()\n * [Exploring the Percent column]()\n * [Gender Distribution]()\n * [Patient Overlap]()\n \n \n* [Visualising Images : DECOM](#3) \n * [Visualising One DECOM Image & Info.]()\n * [Visualising Multiple DECOM Images]()\n * [Visualization using gif]()\n \n \n* [Extracting DIOCOM files Info.](#4)\n\n\n* [Pandas Profiling](#5)\n * [Pandas Profiling Report for Train.csv]()\n * [Pandas Profiling Report for Test.csv]()\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 2. Importing the necessary libraries📗 ","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"import os\nfrom os import listdir\nimport pandas as pd\nimport numpy as np\nimport glob\nimport tqdm\nfrom typing import Dict\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n#plotly\n!pip install chart_studio\nimport plotly.express as px\nimport chart_studio.plotly as py\nimport plotly.graph_objs as go\nfrom plotly.offline import iplot\nimport cufflinks\ncufflinks.go_offline()\ncufflinks.set_config_file(world_readable=True, theme='pearl')\n\n#color\nfrom colorama import Fore, Back, Style\n\nimport seaborn as sns\nsns.set(style=\"whitegrid\")\n\n#pydicom\nimport pydicom\n\n# Suppress warnings \nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Settings for pretty nice plots\nplt.style.use('fivethirtyeight')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Reading the train.csv 📚","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# List files available\nlist(os.listdir(\"../input/osic-pulmonary-fibrosis-progression\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_PATH = \"../input/osic-pulmonary-fibrosis-progressiont/\"\n\ntrain_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\ntest_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\n\nprint(Fore.YELLOW + 'Training data shape: ',Style.RESET_ALL,train_df.shape)\ntrain_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_df.groupby(['SmokingStatus']).count()['Sex'].to_frame()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Basic Data Exploration 🏕️ ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## General Info","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Null values and Data types\nprint(Fore.YELLOW + 'Train Set !!',Style.RESET_ALL)\nprint(train_df.info())\nprint('-------------')\nprint(Fore.BLUE + 'Test Set !!',Style.RESET_ALL)\nprint(test_df.info())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The type of Percent column is float64.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Missing values","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is no missing values in train_df and test_df.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Total number of Patient in the dataset(train+test)\n\nprint(Fore.YELLOW +\"Total Patients in Train set: \",Style.RESET_ALL,train_df['Patient'].count())\nprint(Fore.BLUE +\"Total Patients in Test set: \",Style.RESET_ALL,test_df['Patient'].count())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" `5` : Patients in Test Set","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Unique Patients(Ids)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(Fore.YELLOW + \"The total patient ids are\",Style.RESET_ALL,f\"{train_df['Patient'].count()},\", Fore.BLUE + \"from those the unique ids are\", Style.RESET_ALL, f\"{train_df['Patient'].value_counts().shape[0]}.\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_patient_ids = set(train_df['Patient'].unique())\ntest_patient_ids = set(test_df['Patient'].unique())\n\ntrain_patient_ids.intersection(test_patient_ids)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We already see `5` patients in test set that can be found in train set as well.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"columns = train_df.keys()\ncolumns = list(columns)\nprint(columns)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Patient Counts","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['Patient'].value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In train set, there are multiple rows for one 'Patient'. Because Patient has different weeks, FVC, Percent.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['Patient'].value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In test set, we can see one Patient and it mean Patient id is unique.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"np.quantile(train_df['Patient'].value_counts(), 0.75) - np.quantile(test_df['Patient'].value_counts(), 0.25)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(np.quantile(train_df['Patient'].value_counts(), 0.95))\nprint(np.quantile(test_df['Patient'].value_counts(), 0.95))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Number of Patients and Images in Training Images Folder\n* https://www.kaggle.com/yeayates21/osic-simple-image-eda","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"files = folders = 0\n\npath = \"/kaggle/input/osic-pulmonary-fibrosis-progression/train\"\n\nfor _, dirnames, filenames in os.walk(path):\n # ^ this idiom means \"we won't be using this value\"\n files += len(filenames)\n folders += len(dirnames)\n#print(Fore.YELLOW +\"Total Patients in Train set: \",Style.RESET_ALL,train_df['Patient'].count())\nprint(Fore.YELLOW +f'{files:,}',Style.RESET_ALL,\"files/images, \" + Fore.BLUE + f'{folders:,}',Style.RESET_ALL ,'folders/patients')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"files = []\nfor _, dirnames, filenames in os.walk(path):\n # ^ this idiom means \"we won't be using this value\"\n files.append(len(filenames))\n\nprint(Fore.YELLOW +f'{round(np.mean(files)):,}',Style.RESET_ALL,'average files/images per patient')\nprint(Fore.BLUE +f'{round(np.max(files)):,}',Style.RESET_ALL, 'max files/images per patient')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Data Exploration in Details 🎠 ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Creating Individual Patient Dataframe\n\nfor `175` unique patients, we make new dataframe\n\nThanks [@wjdanalharthi](https://www.kaggle.com/wjdanalharthi)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df = train_df[['Patient', 'Age', 'Sex', 'SmokingStatus']].drop_duplicates()\npatient_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can use this:\n- https://www.kaggle.com/redwankarimsony/pulmonary-fibrosis-progression-interactive-eda","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating unique patient lists and their properties. \ntrain_dir = '../input/osic-pulmonary-fibrosis-progression/train/'\ntest_dir = '../input/osic-pulmonary-fibrosis-progression/test/'\n\npatient_ids = os.listdir(train_dir)\npatient_ids = sorted(patient_ids)\n\n#Creating new rows\nno_of_instances = []\nage = []\nsex = []\nsmoking_status = []\n\nfor patient_id in patient_ids:\n patient_info = train_df[train_df['Patient'] == patient_id].reset_index()\n no_of_instances.append(len(os.listdir(train_dir + patient_id)))\n age.append(patient_info['Age'][0])\n sex.append(patient_info['Sex'][0])\n smoking_status.append(patient_info['SmokingStatus'][0])\n\n#Creating the dataframe for the patient info \npatient_df = pd.DataFrame(list(zip(patient_ids, no_of_instances, age, sex, smoking_status)), \n columns =['Patient', 'no_of_instances', 'Age', 'Sex', 'SmokingStatus'])\nprint(patient_df.info())\npatient_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring the 'SmokingStatus' column","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df['SmokingStatus'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df['SmokingStatus'].value_counts().iplot(kind='bar',\n yTitle='Counts', \n linecolor='black', \n opacity=0.7,\n color='blue',\n theme='pearl',\n bargap=0.5,\n gridcolor='white',\n title='Distribution of the SmokingStatus column in the Unique Patient Set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`118` : Ex-smoker\n\n`49` : Never smoked\n\n`9` : Currently smokes","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Weeks distribution","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['Weeks'].value_counts().head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['Weeks'].value_counts().iplot(kind='barh',\n xTitle='Counts(Weeks)', \n linecolor='black', \n opacity=0.7,\n color='#FB8072',\n theme='pearl',\n bargap=0.2,\n gridcolor='white',\n title='Distribution of the Weeks in the training set')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['Weeks'].iplot(kind='hist',\n xTitle='Weeks', \n yTitle='Counts',\n linecolor='black', \n opacity=0.7,\n color='#FB8072',\n theme='pearl',\n bargap=0.2,\n gridcolor='white',\n title='Distribution of the Weeks in the training set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are some negative values for Weeks. \n\nBecause Weeks is the relative number of weeks pre/post the baseline CT.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Distribution Age over Week","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.scatter(train_df, x=\"Weeks\", y=\"Age\", color='Sex')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## FVC - The forced vital capacity","execution_count":null},{"metadata":{},"cell_type":"markdown","source":" The forced vital capacity (FVC), i.e. the volume of air exhaled\n - the recorded lung capacity in ml","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['FVC'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['FVC'].iplot(kind='hist',\n xTitle='Lung Capacity(ml)', \n linecolor='black', \n opacity=0.8,\n color='#FB8072',\n bargap=0.5,\n gridcolor='white',\n title='Distribution of the FVC in the training set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### FVC vs Percent","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.scatter(train_df, x=\"FVC\", y=\"Percent\", color='Age')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"FVC seems to related Percent linearly. Makes sense as both terms are proportional.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### FVC vs Age","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.scatter(train_df, x=\"FVC\", y=\"Age\", color='Sex')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Males have higher FVC than females irrespective of age","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### FVC vs Weeks","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.scatter(train_df, x=\"FVC\", y=\"Weeks\", color='SmokingStatus')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Pick one patient for FVC vs Weeks","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient = train_df[train_df.Patient == 'ID00228637202259965313869']\nfig = px.line(patient, x=\"Weeks\", y=\"FVC\", color='SmokingStatus')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Person never smoked has FVC lower than smoker. Some Ex-smoker have very high FVC.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Percent","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"A computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['Percent'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['Percent'].iplot(kind='hist',bins=30,color='blue',xTitle='Percent distribution',yTitle='Count')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Percent vs SmokingStatus In Patient Dataframe","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df = train_df\nfig = px.violin(df, y='Percent', x='SmokingStatus', box=True, color='Sex', points=\"all\",\n hover_data=train_df.columns)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nax = sns.violinplot(x = train_df['SmokingStatus'], y = train_df['Percent'], palette = 'Reds')\nax.set_xlabel(xlabel = 'Smoking Habit', fontsize = 15)\nax.set_ylabel(ylabel = 'Percent', fontsize = 15)\nax.set_title(label = 'Distribution of Smoking Status Over Percentage', fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.scatter(train_df, x=\"Age\", y=\"Percent\", color='SmokingStatus')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Pick one patient for FVC vs Weeks","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient = train_df[train_df.Patient == 'ID00228637202259965313869']\nfig = px.line(patient, x=\"Weeks\", y=\"Percent\", color='SmokingStatus')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Age Distribution of Unique Patients","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df['Age'].iplot(kind='hist',bins=30,color='red',xTitle='Ages of distribution',yTitle='Count')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Distribution of Age vs SmokingStatus In Patient Dataframe","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df['SmokingStatus'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsns.kdeplot(patient_df.loc[patient_df['SmokingStatus'] == 'Ex-smoker', 'Age'], label = 'Ex-smoker',shade=True)\nsns.kdeplot(patient_df.loc[patient_df['SmokingStatus'] == 'Never smoked', 'Age'], label = 'Never smoked',shade=True)\nsns.kdeplot(patient_df.loc[patient_df['SmokingStatus'] == 'Currently smokes', 'Age'], label = 'Currently smokes', shade=True)\n\n# Labeling of plot\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages');","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nax = sns.violinplot(x = patient_df['SmokingStatus'], y = patient_df['Age'], palette = 'Reds')\nax.set_xlabel(xlabel = 'Smoking habit', fontsize = 15)\nax.set_ylabel(ylabel = 'Age', fontsize = 15)\nax.set_title(label = 'Distribution of Smokers over Age', fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Distribution of Age vs Gender In Patient Dataframe","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsns.kdeplot(patient_df.loc[patient_df['Sex'] == 'Male', 'Age'], label = 'Male',shade=True)\nsns.kdeplot(patient_df.loc[patient_df['Sex'] == 'Female', 'Age'], label = 'Female',shade=True)\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Gender Distribution","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df['Sex'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df['Sex'].value_counts().iplot(kind='bar',\n yTitle='Count', \n linecolor='black', \n opacity=0.7,\n color='blue',\n theme='pearl',\n bargap=0.8,\n gridcolor='white',\n title='Distribution of the Sex column in Patient Dataframe')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`139` : Male\n\n`37` : Female","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Gender vs SmokingStatus In Patient Dataframe","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\na = sns.countplot(data=patient_df, x='SmokingStatus', hue='Sex')\n\nfor p in a.patches:\n a.annotate(format(p.get_height(), ','), \n (p.get_x() + p.get_width() / 2., \n p.get_height()), ha = 'center', va = 'center', \n xytext = (0, 4), textcoords = 'offset points')\n\nplt.title('Gender split by SmokingStatus', fontsize=16)\nsns.despine(left=True, bottom=True);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.box(patient_df, x=\"Sex\", y=\"Age\", points=\"all\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Patient Overlap","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Extract patient id's for the training set\nids_train = train_df.Patient.values\n# Extract patient id's for the validation set\nids_test = test_df.Patient.values\n#print(Fore.YELLOW +\"Total Patients in Train set: \",Style.RESET_ALL,train_df['Patient'].count())\n# Create a \"set\" datastructure of the training set id's to identify unique id's\nids_train_set = set(ids_train)\nprint(Fore.YELLOW + \"There are\",Style.RESET_ALL,f'{len(ids_train_set)}', Fore.BLUE + 'unique Patient IDs',Style.RESET_ALL,'in the training set')\n# Create a \"set\" datastructure of the validation set id's to identify unique id's\nids_test_set = set(ids_test)\nprint(Fore.YELLOW + \"There are\", Style.RESET_ALL, f'{len(ids_test_set)}', Fore.BLUE + 'unique Patient IDs',Style.RESET_ALL,'in the test set')\n\n# Identify patient overlap by looking at the intersection between the sets\npatient_overlap = list(ids_train_set.intersection(ids_test_set))\nn_overlap = len(patient_overlap)\nprint(Fore.YELLOW + \"There are\", Style.RESET_ALL, f'{n_overlap}', Fore.BLUE + 'Patient IDs',Style.RESET_ALL, 'in both the training and test sets')\nprint('')\nprint(Fore.CYAN + 'These patients are in both the training and test datasets:', Style.RESET_ALL)\nprint(f'{patient_overlap}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`5` patients are in both the training and test datasets.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Heatmap for train.csv","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"corrmat = train_df.corr() \nf, ax = plt.subplots(figsize =(9, 8)) \nsns.heatmap(corrmat, ax = ax, cmap = 'RdYlBu_r', linewidths = 0.5) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Please compare with the previous visualization information. And we may compare to Pandas Profiling below.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 6. Visualising Images : DECOM 🗺️ ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"`1` type of images containing the information:\n\n- `.dcm` files: [DICOM files](https://en.wikipedia.org/wiki/DICOM). It's saved in the \"Digital Imaging and Communications in Medicine\" format. It contains an image from a medical scan, such as an ultrasound or MRI + information about the patient.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(Fore.YELLOW + 'Train .dcm number of images:',Style.RESET_ALL, len(list(os.listdir('../input/osic-pulmonary-fibrosis-progression/train'))), '\\n' +\n Fore.BLUE + 'Test .dcm number of images:',Style.RESET_ALL, len(list(os.listdir('../input/osic-pulmonary-fibrosis-progression/test'))), '\\n' +\n '--------------------------------', '\\n' +\n 'There is the same number of images as in train/ test .csv datasets')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at the DICOM images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_pixel_array(dataset, figsize=(5,5)):\n plt.figure(figsize=figsize)\n plt.grid(False)\n plt.imshow(dataset.pixel_array, cmap='gray') # cmap=plt.cm.bone)\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# https://www.kaggle.com/schlerp/getting-to-know-dicom-and-the-data\ndef show_dcm_info(dataset):\n print(Fore.YELLOW + \"Filename.........:\",Style.RESET_ALL,file_path)\n print()\n\n pat_name = dataset.PatientName\n display_name = pat_name.family_name + \", \" + pat_name.given_name\n print(Fore.BLUE + \"Patient's name......:\",Style.RESET_ALL, display_name)\n print(Fore.BLUE + \"Patient id..........:\",Style.RESET_ALL, dataset.PatientID)\n print(Fore.BLUE + \"Patient's Sex.......:\",Style.RESET_ALL, dataset.PatientSex)\n print(Fore.YELLOW + \"Modality............:\",Style.RESET_ALL, dataset.Modality)\n print(Fore.GREEN + \"Body Part Examined..:\",Style.RESET_ALL, dataset.BodyPartExamined)\n \n if 'PixelData' in dataset:\n rows = int(dataset.Rows)\n cols = int(dataset.Columns)\n print(Fore.BLUE + \"Image size.......:\",Style.RESET_ALL,\" {rows:d} x {cols:d}, {size:d} bytes\".format(\n rows=rows, cols=cols, size=len(dataset.PixelData)))\n if 'PixelSpacing' in dataset:\n print(Fore.YELLOW + \"Pixel spacing....:\",Style.RESET_ALL,dataset.PixelSpacing)\n dataset.PixelSpacing = [1, 1]\n plt.figure(figsize=(10, 10))\n plt.imshow(dataset.pixel_array, cmap='gray')\n plt.show()\nfor file_path in glob.glob('../input/osic-pulmonary-fibrosis-progression/train/*/*.dcm'):\n dataset = pydicom.dcmread(file_path)\n show_dcm_info(dataset)\n break # Comment this out to see all","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/yeayates21/osic-simple-image-eda\n\nimdir = \"/kaggle/input/osic-pulmonary-fibrosis-progression/train/ID00123637202217151272140\"\nprint(\"total images for patient ID00123637202217151272140: \", len(os.listdir(imdir)))\n\n# view first (columns*rows) images in order\nfig=plt.figure(figsize=(12, 12))\ncolumns = 4\nrows = 5\nimglist = os.listdir(imdir)\nfor i in range(1, columns*rows +1):\n filename = imdir + \"/\" + str(i) + \".dcm\"\n ds = pydicom.dcmread(filename)\n fig.add_subplot(rows, columns, i)\n plt.imshow(ds.pixel_array, cmap='gray')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/yeayates21/osic-simple-image-eda\n\nimdir = \"/kaggle/input/osic-pulmonary-fibrosis-progression/train/ID00123637202217151272140\"\nprint(\"total images for patient ID00123637202217151272140: \", len(os.listdir(imdir)))\n\n# view first (columns*rows) images in order\nfig=plt.figure(figsize=(12, 12))\ncolumns = 4\nrows = 5\nimglist = os.listdir(imdir)\nfor i in range(1, columns*rows +1):\n filename = imdir + \"/\" + str(i) + \".dcm\"\n ds = pydicom.dcmread(filename)\n fig.add_subplot(rows, columns, i)\n plt.imshow(ds.pixel_array, cmap='jet')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualization using gif\n* https://www.kaggle.com/danpresil1/dicom-basic-preprocessing-and-visualization","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"apply_resample = False\n\ndef load_scan(path):\n slices = [pydicom.read_file(path + '/' + s) for s in os.listdir(path)]\n slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))\n try:\n slice_thickness = np.abs(slices[0].ImagePositionPatient[2] - slices[1].ImagePositionPatient[2])\n except:\n slice_thickness = np.abs(slices[0].SliceLocation - slices[1].SliceLocation)\n \n for s in slices:\n s.SliceThickness = slice_thickness\n \n return slices","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def load_scan(path):\n slices = [pydicom.read_file(path + '/' + s) for s in os.listdir(path)]\n slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))\n try:\n slice_thickness = np.abs(slices[0].ImagePositionPatient[2] - slices[1].ImagePositionPatient[2])\n except:\n slice_thickness = np.abs(slices[0].SliceLocation - slices[1].SliceLocation)\n \n for s in slices:\n s.SliceThickness = slice_thickness\n \n return slices","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def get_pixels_hu(slices):\n image = np.stack([s.pixel_array for s in slices])\n # Convert to int16 (from sometimes int16), \n # should be possible as values should always be low enough (<32k)\n image = image.astype(np.int16)\n\n # Set outside-of-scan pixels to 0\n # The intercept is usually -1024, so air is approximately 0\n image[image == -2000] = 0\n \n # Convert to Hounsfield units (HU)\n for slice_number in range(len(slices)):\n \n intercept = slices[slice_number].RescaleIntercept\n slope = slices[slice_number].RescaleSlope\n \n if slope != 1:\n image[slice_number] = slope * image[slice_number].astype(np.float64)\n image[slice_number] = image[slice_number].astype(np.int16)\n \n image[slice_number] += np.int16(intercept)\n \n return np.array(image, dtype=np.int16)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def set_lungwin(img, hu=[-1200., 600.]):\n lungwin = np.array(hu)\n newimg = (img-lungwin[0]) / (lungwin[1]-lungwin[0])\n newimg[newimg < 0] = 0\n newimg[newimg > 1] = 1\n newimg = (newimg * 255).astype('uint8')\n return newimg","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scans = load_scan('../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/')\nscan_array = set_lungwin(get_pixels_hu(scans))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Resample to 1mm (An optional step, it may not be relevant to this competition because of the large slice thickness on the z axis)\n\nfrom scipy.ndimage.interpolation import zoom\n\ndef resample(imgs, spacing, new_spacing):\n new_shape = np.round(imgs.shape * spacing / new_spacing)\n true_spacing = spacing * imgs.shape / new_shape\n resize_factor = new_shape / imgs.shape\n imgs = zoom(imgs, resize_factor, mode='nearest')\n return imgs, true_spacing, new_shape\n\nspacing_z = (scans[-1].ImagePositionPatient[2] - scans[0].ImagePositionPatient[2]) / len(scans)\n\nif apply_resample:\n scan_array_resample = resample(scan_array, np.array(np.array([spacing_z, *scans[0].PixelSpacing])), np.array([1.,1.,1.]))[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import imageio\nfrom IPython.display import Image\n\nimageio.mimsave(\"/tmp/gif.gif\", scan_array, duration=0.0001)\nImage(filename=\"/tmp/gif.gif\", format='png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualization using Animation\n* https://www.kaggle.com/pranavkasela/animating-the-lung-ct-scan","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"For animation, I used scan_array in 'visualization using gif' section.","execution_count":null},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import matplotlib.animation as animation\n\nfig = plt.figure()\n\nims = []\nfor image in scan_array:\n im = plt.imshow(image, animated=True, cmap=\"Greys\")\n plt.axis(\"off\")\n ims.append([im])\n\nani = animation.ArtistAnimation(fig, ims, interval=100, blit=False,\n repeat_delay=1000)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"HTML(ani.to_jshtml())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"HTML(ani.to_html5_video())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7. Extracting DIOCOM files information in a dataframe 🌊","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"* https://www.kaggle.com/trsekhar123/nb-to-extract-metadata-and-resize-images-train","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def extract_dicom_meta_data(filename: str) -> Dict:\n # Load image\n \n image_data = pydicom.read_file(filename)\n img=np.array(image_data.pixel_array).flatten()\n row = {\n 'Patient': image_data.PatientID,\n 'body_part_examined': image_data.BodyPartExamined,\n 'image_position_patient': image_data.ImagePositionPatient,\n 'image_orientation_patient': image_data.ImageOrientationPatient,\n 'photometric_interpretation': image_data.PhotometricInterpretation,\n 'rows': image_data.Rows,\n 'columns': image_data.Columns,\n 'pixel_spacing': image_data.PixelSpacing,\n 'window_center': image_data.WindowCenter,\n 'window_width': image_data.WindowWidth,\n 'modality': image_data.Modality,\n 'StudyInstanceUID': image_data.StudyInstanceUID,\n 'SeriesInstanceUID': image_data.StudyInstanceUID,\n 'StudyID': image_data.StudyInstanceUID, \n 'SamplesPerPixel': image_data.SamplesPerPixel,\n 'BitsAllocated': image_data.BitsAllocated,\n 'BitsStored': image_data.BitsStored,\n 'HighBit': image_data.HighBit,\n 'PixelRepresentation': image_data.PixelRepresentation,\n 'RescaleIntercept': image_data.RescaleIntercept,\n 'RescaleSlope': image_data.RescaleSlope,\n 'img_min': np.min(img),\n 'img_max': np.max(img),\n 'img_mean': np.mean(img),\n 'img_std': np.std(img)}\n\n return row","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_image_path = '/kaggle/input/osic-pulmonary-fibrosis-progression/train'\ntrain_image_files = glob.glob(os.path.join(train_image_path, '*', '*.dcm'))\n\nmeta_data_df = []\nfor filename in tqdm.tqdm(train_image_files):\n try:\n meta_data_df.append(extract_dicom_meta_data(filename))\n except Exception as e:\n continue","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The meta elements in dcom file seem to be slightly different for each file. So, I use Exception.\n- Be careful!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert to a pd.DataFrame from dict\nmeta_data_df = pd.DataFrame.from_dict(meta_data_df)\nmeta_data_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The code below still makes sense, so I leave it.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# source: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154658\nfolder='train'\nPATH='../input/osic-pulmonary-fibrosis-progression/'\n\nlast_index = 2\n\ncolumn_names = ['image_name', 'dcm_ImageOrientationPatient', \n 'dcm_ImagePositionPatient', 'dcm_PatientID',\n 'dcm_PatientName', 'dcm_PatientSex'\n 'dcm_rows', 'dcm_columns']\n\ndef extract_DICOM_attributes(folder):\n patients_folder = list(os.listdir(os.path.join(PATH, folder)))\n df = pd.DataFrame()\n \n i = 0\n \n for patient_id in patients_folder:\n \n img_path = os.path.join(PATH, folder, patient_id)\n \n print(img_path)\n \n images = list(os.listdir(img_path))\n \n #df = pd.DataFrame()\n\n for image in images:\n image_name = image.split(\".\")[0]\n\n dicom_file_path = os.path.join(img_path,image)\n dicom_file_dataset = pydicom.read_file(dicom_file_path)\n \n '''\n print(dicom_file_dataset.dir(\"pat\"))\n print(dicom_file_dataset.data_element(\"ImageOrientationPatient\"))\n print(dicom_file_dataset.data_element(\"ImagePositionPatient\"))\n print(dicom_file_dataset.data_element(\"PatientID\"))\n print(dicom_file_dataset.data_element(\"PatientName\"))\n print(dicom_file_dataset.data_element(\"PatientSex\"))\n '''\n \n imageOrientationPatient = dicom_file_dataset.ImageOrientationPatient\n #imagePositionPatient = dicom_file_dataset.ImagePositionPatient\n patientID = dicom_file_dataset.PatientID\n patientName = dicom_file_dataset.PatientName\n patientSex = dicom_file_dataset.PatientSex\n \n rows = dicom_file_dataset.Rows\n cols = dicom_file_dataset.Columns\n \n #print(rows)\n #print(columns)\n \n temp_dict = {'image_name': image_name, \n 'dcm_ImageOrientationPatient': imageOrientationPatient,\n #'dcm_ImagePositionPatient':imagePositionPatient,\n 'dcm_PatientID': patientID, \n 'dcm_PatientName': patientName,\n 'dcm_PatientSex': patientSex,\n 'dcm_rows': rows,\n 'dcm_columns': cols}\n\n\n df = df.append([temp_dict])\n \n i += 1\n \n if i == last_index:\n break\n \n return df\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"extract_DICOM_attributes('train')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Etc. - Pandas Profiling 🌤️","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The meta elements in dcom file seem to be slightly different for each file.\n- Be careful!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas_profiling as pdp","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\ntest_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"profile_train_df = pdp.ProfileReport(train_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"profile_train_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"profile_test_df = pdp.ProfileReport(test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"profile_test_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## If this kernel is useful, please upvote!\n- See you next time!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# References\n- https://www.kaggle.com/tarunpaparaju/siim-isic-melanoma-eda-pytorch-baseline\n- https://www.kaggle.com/andradaolteanu/siim-melanoma-competition-eda-augmentations\n- https://www.kaggle.com/parulpandey/melanoma-classification-eda-starter\n- https://www.kaggle.com/yeayates21/osic-simple-image-eda\n- https://www.kaggle.com/aviralpamecha/osic-in-depth-and-advanced-analysis\n- https://www.kaggle.com/danpresil1/dicom-basic-preprocessing-and-visualization","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}