\n \n# 1. Introduction 🃏 \n \n### 1.1 What is Pulmonary fibrosis?\n* [Pulmonary fibrosis is a lung disease that occurs when lung tissue becomes damaged and scarred.](https://www.mayoclinic.org/diseases-conditions/pulmonary-fibrosis/symptoms-causes/syc-20353690) This thickened, stiff tissue makes it more difficult for your lungs to work properly. If you want to know further about this type lung disease, I have linked below an informative video.\n* Pulmonary fibrosis is a condition that causes lung scarring and stiffness. This makes it difficult to breathe. It can prevent your body from getting enough oxygen and may eventually lead to respiratory failure, heart failure, or other complications.(https://www.healthline.com/health/pulmonary-fibrosis) "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from IPython.display import HTML\nHTML('
')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1.2 What is OSIC Pulmonary Fibrosis Progression Competition?\n- In this competition, you’ll predict a patient’s severity of decline in lung function based on a CT scan of their lungs. You’ll determine lung function based on output from a spirometer, which measures the volume of air inhaled and exhaled. The challenge is to use machine learning techniques to make a prediction with the image, metadata, and baseline FVC as input.\n \n \n### 1.3 What we need to do? Observation\nWe will predict a patient’s severity of decline in lung function based on a CT scan of their lungs.\nIn other words, We will predict the final three FVC measurements for each patient, as well as a confidence value in your prediction.\n \n- The leaderboard of this competition is calculated with approximately 1%->15% of the test data. The final results will be based on the other 99%->85%, so the final standings may be different.\n \n### 1.4 Metric: Laplace Log Likelihood\n\n- Image Credits: https://en.wikipedia.org/wiki/Laplace_distribution\n \n- The evaluation metric of this competition is a modified version of Laplace Log Likelihood. \nPredictions are evaluated with a modified version of the Laplace Log Likelihood. For each sample in test set, an `FVC` and a `Confidence` measure (standard deviation σ) has to be predicted.\n\n `Confidence` values smaller than 70 are clipped.\n\n $\\large \\sigma_{clipped} = max(\\sigma, 70),$\n\n Errors greater than 1000 are also clipped in order to avoid large errors.\n\n $\\large \\Delta = min ( |FVC_{true} - FVC_{predicted}|, 1000 ),$\n\n The metric is defined as:\n\n $\\Large metric = - \\frac{\\sqrt{2} \\Delta}{\\sigma_{clipped}} - \\ln ( \\sqrt{2} \\sigma_{clipped} ).$\n \n The metric will be maximum when:\n \n $ \\sigma_{clipped} = \\sqrt{2} \\Delta $\n\n\nRead more about it on the [Evaluation Page](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/overview/evaluation)."},{"metadata":{},"cell_type":"markdown","source":"## Contents \n\n\n\n* [Basic Exploratory Data Analysis](#1) \n * [Getting started - Importing libraries]()\n * [Reading the train.csv]()\n \n \n* [Data Exploration](#2) \n * [Check Train & Test Info.]()\n * [Unique Patients(Ids)]()\n * [Exploring the 'SmokingStatus' column]()\n * [Weeks distribution]()\n * [FVC - The forced vital capacity]()\n * [Exploring the Percent column]()\n * [Gender Distribution]()\n * [Patient Overlap]()\n \n \n* [Visualising Images : DECOM](#3) \n * [Visualising One DECOM Image & Info.]()\n * [Visualising Multiple DECOM Images]()\n * [Visualization using gif]()\n \n \n* [Extracting DIOCOM files Info.](#4)\n\n\n* [Pandas Profiling](#5)\n * [Pandas Profiling Report for Train.csv]()\n * [Pandas Profiling Report for Test.csv]()\n"},{"metadata":{},"cell_type":"markdown","source":"# 2. Importing the necessary libraries📗 "},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"import os\nfrom os import listdir\nimport pandas as pd\nimport numpy as np\nimport glob\nimport tqdm\nfrom typing import Dict\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n#plotly\n!pip install chart_studio\nimport plotly.express as px\nimport chart_studio.plotly as py\nimport plotly.graph_objs as go\nfrom plotly.offline import iplot\nimport cufflinks\ncufflinks.go_offline()\ncufflinks.set_config_file(world_readable=True, theme='pearl')\n\n#color\nfrom colorama import Fore, Back, Style\n\nimport seaborn as sns\nsns.set(style=\"whitegrid\")\n\n#pydicom\nimport pydicom\n\n# Suppress warnings \nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Settings for pretty nice plots\nplt.style.use('fivethirtyeight')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Reading the train.csv 📚"},{"metadata":{"trusted":true},"cell_type":"code","source":"# List files available\nlist(os.listdir(\"../input/osic-pulmonary-fibrosis-progression\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_PATH = \"../input/osic-pulmonary-fibrosis-progressiont/\"\n\ntrain_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\ntest_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\n\nprint(Fore.YELLOW + 'Training data shape: ',Style.RESET_ALL,train_df.shape)\ntrain_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_df.groupby(['SmokingStatus']).count()['Sex'].to_frame()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Basic Data Exploration 🏕️ "},{"metadata":{},"cell_type":"markdown","source":"## General Info"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Null values and Data types\nprint(Fore.YELLOW + 'Train Set !!',Style.RESET_ALL)\nprint(train_df.info())\nprint('-------------')\nprint(Fore.BLUE + 'Test Set !!',Style.RESET_ALL)\nprint(test_df.info())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The type of Percent column is float64."},{"metadata":{},"cell_type":"markdown","source":"### Missing values"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is no missing values in train_df and test_df."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Total number of Patient in the dataset(train+test)\n\nprint(Fore.YELLOW +\"Total Patients in Train set: \",Style.RESET_ALL,train_df['Patient'].count())\nprint(Fore.BLUE +\"Total Patients in Test set: \",Style.RESET_ALL,test_df['Patient'].count())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" `5` : Patients in Test Set"},{"metadata":{},"cell_type":"markdown","source":"## Unique Patients(Ids)"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(Fore.YELLOW + \"The total patient ids are\",Style.RESET_ALL,f\"{train_df['Patient'].count()},\", Fore.BLUE + \"from those the unique ids are\", Style.RESET_ALL, f\"{train_df['Patient'].value_counts().shape[0]}.\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_patient_ids = set(train_df['Patient'].unique())\ntest_patient_ids = set(test_df['Patient'].unique())\n\ntrain_patient_ids.intersection(test_patient_ids)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We already see `5` patients in test set that can be found in train set as well."},{"metadata":{"trusted":true},"cell_type":"code","source":"columns = train_df.keys()\ncolumns = list(columns)\nprint(columns)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Patient Counts"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['Patient'].value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In train set, there are multiple rows for one 'Patient'. Because Patient has different weeks, FVC, Percent."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['Patient'].value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In test set, we can see one Patient and it mean Patient id is unique."},{"metadata":{"trusted":true},"cell_type":"code","source":"np.quantile(train_df['Patient'].value_counts(), 0.75) - np.quantile(test_df['Patient'].value_counts(), 0.25)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(np.quantile(train_df['Patient'].value_counts(), 0.95))\nprint(np.quantile(test_df['Patient'].value_counts(), 0.95))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Number of Patients and Images in Training Images Folder\n* https://www.kaggle.com/yeayates21/osic-simple-image-eda"},{"metadata":{"trusted":true},"cell_type":"code","source":"files = folders = 0\n\npath = \"/kaggle/input/osic-pulmonary-fibrosis-progression/train\"\n\nfor _, dirnames, filenames in os.walk(path):\n # ^ this idiom means \"we won't be using this value\"\n files += len(filenames)\n folders += len(dirnames)\n#print(Fore.YELLOW +\"Total Patients in Train set: \",Style.RESET_ALL,train_df['Patient'].count())\nprint(Fore.YELLOW +f'{files:,}',Style.RESET_ALL,\"files/images, \" + Fore.BLUE + f'{folders:,}',Style.RESET_ALL ,'folders/patients')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"files = []\nfor _, dirnames, filenames in os.walk(path):\n # ^ this idiom means \"we won't be using this value\"\n files.append(len(filenames))\n\nprint(Fore.YELLOW +f'{round(np.mean(files)):,}',Style.RESET_ALL,'average files/images per patient')\nprint(Fore.BLUE +f'{round(np.max(files)):,}',Style.RESET_ALL, 'max files/images per patient')\nprint(Fore.GREEN +f'{round(np.min(files)):,}',Style.RESET_ALL,'min files/images per patient')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Data Exploration in Details 🎠 "},{"metadata":{},"cell_type":"markdown","source":"## Creating Individual Patient Dataframe\n\nfor `175` unique patients, we make new dataframe\n\nThanks [@wjdanalharthi](https://www.kaggle.com/wjdanalharthi)"},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df = train_df[['Patient', 'Age', 'Sex', 'SmokingStatus']].drop_duplicates()\npatient_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can use this:\n- https://www.kaggle.com/redwankarimsony/pulmonary-fibrosis-progression-interactive-eda"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating unique patient lists and their properties. \ntrain_dir = '../input/osic-pulmonary-fibrosis-progression/train/'\ntest_dir = '../input/osic-pulmonary-fibrosis-progression/test/'\n\npatient_ids = os.listdir(train_dir)\npatient_ids = sorted(patient_ids)\n\n#Creating new rows\nno_of_instances = []\nage = []\nsex = []\nsmoking_status = []\n\nfor patient_id in patient_ids:\n patient_info = train_df[train_df['Patient'] == patient_id].reset_index()\n no_of_instances.append(len(os.listdir(train_dir + patient_id)))\n age.append(patient_info['Age'][0])\n sex.append(patient_info['Sex'][0])\n smoking_status.append(patient_info['SmokingStatus'][0])\n\n#Creating the dataframe for the patient info \npatient_df = pd.DataFrame(list(zip(patient_ids, no_of_instances, age, sex, smoking_status)), \n columns =['Patient', 'no_of_instances', 'Age', 'Sex', 'SmokingStatus'])\nprint(patient_df.info())\npatient_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring the 'SmokingStatus' column"},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df['SmokingStatus'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df['SmokingStatus'].value_counts().iplot(kind='bar',\n yTitle='Counts', \n linecolor='black', \n opacity=0.7,\n color='blue',\n theme='pearl',\n bargap=0.5,\n gridcolor='white',\n title='Distribution of the SmokingStatus column in the Unique Patient Set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`118` : Ex-smoker\n\n`49` : Never smoked\n\n`9` : Currently smokes"},{"metadata":{},"cell_type":"markdown","source":"## Weeks distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['Weeks'].value_counts().head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['Weeks'].value_counts().iplot(kind='barh',\n xTitle='Counts(Weeks)', \n linecolor='black', \n opacity=0.7,\n color='#FB8072',\n theme='pearl',\n bargap=0.2,\n gridcolor='white',\n title='Distribution of the Weeks in the training set')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['Weeks'].iplot(kind='hist',\n xTitle='Weeks', \n yTitle='Counts',\n linecolor='black', \n opacity=0.7,\n color='#FB8072',\n theme='pearl',\n bargap=0.2,\n gridcolor='white',\n title='Distribution of the Weeks in the training set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are some negative values for Weeks. \n\nBecause Weeks is the relative number of weeks pre/post the baseline CT."},{"metadata":{},"cell_type":"markdown","source":"## Distribution Age over Week"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.scatter(train_df, x=\"Weeks\", y=\"Age\", color='Sex')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## FVC - The forced vital capacity"},{"metadata":{},"cell_type":"markdown","source":" The forced vital capacity (FVC), i.e. the volume of air exhaled\n - the recorded lung capacity in ml"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['FVC'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['FVC'].iplot(kind='hist',\n xTitle='Lung Capacity(ml)', \n linecolor='black', \n opacity=0.8,\n color='#FB8072',\n bargap=0.5,\n gridcolor='white',\n title='Distribution of the FVC in the training set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### FVC vs Percent"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.scatter(train_df, x=\"FVC\", y=\"Percent\", color='Age')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"FVC seems to related Percent linearly. Makes sense as both terms are proportional."},{"metadata":{},"cell_type":"markdown","source":"### FVC vs Age"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.scatter(train_df, x=\"FVC\", y=\"Age\", color='Sex')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Males have higher FVC than females irrespective of age"},{"metadata":{},"cell_type":"markdown","source":"### FVC vs Weeks"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.scatter(train_df, x=\"FVC\", y=\"Weeks\", color='SmokingStatus')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Pick one patient for FVC vs Weeks"},{"metadata":{"trusted":true},"cell_type":"code","source":"patient = train_df[train_df.Patient == 'ID00228637202259965313869']\nfig = px.line(patient, x=\"Weeks\", y=\"FVC\", color='SmokingStatus')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Person never smoked has FVC lower than smoker. Some Ex-smoker have very high FVC."},{"metadata":{},"cell_type":"markdown","source":"## Percent"},{"metadata":{},"cell_type":"markdown","source":"A computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['Percent'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['Percent'].iplot(kind='hist',bins=30,color='blue',xTitle='Percent distribution',yTitle='Count')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Percent vs SmokingStatus In Patient Dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = train_df\nfig = px.violin(df, y='Percent', x='SmokingStatus', box=True, color='Sex', points=\"all\",\n hover_data=train_df.columns)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nax = sns.violinplot(x = train_df['SmokingStatus'], y = train_df['Percent'], palette = 'Reds')\nax.set_xlabel(xlabel = 'Smoking Habit', fontsize = 15)\nax.set_ylabel(ylabel = 'Percent', fontsize = 15)\nax.set_title(label = 'Distribution of Smoking Status Over Percentage', fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.scatter(train_df, x=\"Age\", y=\"Percent\", color='SmokingStatus')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Pick one patient for FVC vs Weeks"},{"metadata":{"trusted":true},"cell_type":"code","source":"patient = train_df[train_df.Patient == 'ID00228637202259965313869']\nfig = px.line(patient, x=\"Weeks\", y=\"Percent\", color='SmokingStatus')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Age Distribution of Unique Patients"},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df['Age'].iplot(kind='hist',bins=30,color='red',xTitle='Ages of distribution',yTitle='Count')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Distribution of Age vs SmokingStatus In Patient Dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df['SmokingStatus'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsns.kdeplot(patient_df.loc[patient_df['SmokingStatus'] == 'Ex-smoker', 'Age'], label = 'Ex-smoker',shade=True)\nsns.kdeplot(patient_df.loc[patient_df['SmokingStatus'] == 'Never smoked', 'Age'], label = 'Never smoked',shade=True)\nsns.kdeplot(patient_df.loc[patient_df['SmokingStatus'] == 'Currently smokes', 'Age'], label = 'Currently smokes', shade=True)\n\n# Labeling of plot\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages');","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nax = sns.violinplot(x = patient_df['SmokingStatus'], y = patient_df['Age'], palette = 'Reds')\nax.set_xlabel(xlabel = 'Smoking habit', fontsize = 15)\nax.set_ylabel(ylabel = 'Age', fontsize = 15)\nax.set_title(label = 'Distribution of Smokers over Age', fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Distribution of Age vs Gender In Patient Dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsns.kdeplot(patient_df.loc[patient_df['Sex'] == 'Male', 'Age'], label = 'Male',shade=True)\nsns.kdeplot(patient_df.loc[patient_df['Sex'] == 'Female', 'Age'], label = 'Female',shade=True)\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Gender Distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df['Sex'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df['Sex'].value_counts().iplot(kind='bar',\n yTitle='Count', \n linecolor='black', \n opacity=0.7,\n color='blue',\n theme='pearl',\n bargap=0.8,\n gridcolor='white',\n title='Distribution of the Sex column in Patient Dataframe')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`139` : Male\n\n`37` : Female"},{"metadata":{},"cell_type":"markdown","source":"### Gender vs SmokingStatus In Patient Dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\na = sns.countplot(data=patient_df, x='SmokingStatus', hue='Sex')\n\nfor p in a.patches:\n a.annotate(format(p.get_height(), ','), \n (p.get_x() + p.get_width() / 2., \n p.get_height()), ha = 'center', va = 'center', \n xytext = (0, 4), textcoords = 'offset points')\n\nplt.title('Gender split by SmokingStatus', fontsize=16)\nsns.despine(left=True, bottom=True);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.box(patient_df, x=\"Sex\", y=\"Age\", points=\"all\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Patient Overlap"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Extract patient id's for the training set\nids_train = train_df.Patient.values\n# Extract patient id's for the validation set\nids_test = test_df.Patient.values\n#print(Fore.YELLOW +\"Total Patients in Train set: \",Style.RESET_ALL,train_df['Patient'].count())\n# Create a \"set\" datastructure of the training set id's to identify unique id's\nids_train_set = set(ids_train)\nprint(Fore.YELLOW + \"There are\",Style.RESET_ALL,f'{len(ids_train_set)}', Fore.BLUE + 'unique Patient IDs',Style.RESET_ALL,'in the training set')\n# Create a \"set\" datastructure of the validation set id's to identify unique id's\nids_test_set = set(ids_test)\nprint(Fore.YELLOW + \"There are\", Style.RESET_ALL, f'{len(ids_test_set)}', Fore.BLUE + 'unique Patient IDs',Style.RESET_ALL,'in the test set')\n\n# Identify patient overlap by looking at the intersection between the sets\npatient_overlap = list(ids_train_set.intersection(ids_test_set))\nn_overlap = len(patient_overlap)\nprint(Fore.YELLOW + \"There are\", Style.RESET_ALL, f'{n_overlap}', Fore.BLUE + 'Patient IDs',Style.RESET_ALL, 'in both the training and test sets')\nprint('')\nprint(Fore.CYAN + 'These patients are in both the training and test datasets:', Style.RESET_ALL)\nprint(f'{patient_overlap}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`5` patients are in both the training and test datasets."},{"metadata":{},"cell_type":"markdown","source":"## Heatmap for train.csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"corrmat = train_df.corr() \nf, ax = plt.subplots(figsize =(9, 8)) \nsns.heatmap(corrmat, ax = ax, cmap = 'RdYlBu_r', linewidths = 0.5) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Please compare with the previous visualization information. And we may compare to Pandas Profiling below."},{"metadata":{},"cell_type":"markdown","source":"# 6. Visualising Images : DECOM 🗺️ "},{"metadata":{},"cell_type":"markdown","source":"`1` type of images containing the information:\n\n- `.dcm` files: [DICOM files](https://en.wikipedia.org/wiki/DICOM). It's saved in the \"Digital Imaging and Communications in Medicine\" format. It contains an image from a medical scan, such as an ultrasound or MRI + information about the patient."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(Fore.YELLOW + 'Train .dcm number of images:',Style.RESET_ALL, len(list(os.listdir('../input/osic-pulmonary-fibrosis-progression/train'))), '\\n' +\n Fore.BLUE + 'Test .dcm number of images:',Style.RESET_ALL, len(list(os.listdir('../input/osic-pulmonary-fibrosis-progression/test'))), '\\n' +\n '--------------------------------', '\\n' +\n 'There is the same number of images as in train/ test .csv datasets')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at the DICOM images."},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_pixel_array(dataset, figsize=(5,5)):\n plt.figure(figsize=figsize)\n plt.grid(False)\n plt.imshow(dataset.pixel_array, cmap='gray') # cmap=plt.cm.bone)\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# https://www.kaggle.com/schlerp/getting-to-know-dicom-and-the-data\ndef show_dcm_info(dataset):\n print(Fore.YELLOW + \"Filename.........:\",Style.RESET_ALL,file_path)\n print()\n\n pat_name = dataset.PatientName\n display_name = pat_name.family_name + \", \" + pat_name.given_name\n print(Fore.BLUE + \"Patient's name......:\",Style.RESET_ALL, display_name)\n print(Fore.BLUE + \"Patient id..........:\",Style.RESET_ALL, dataset.PatientID)\n print(Fore.BLUE + \"Patient's Sex.......:\",Style.RESET_ALL, dataset.PatientSex)\n print(Fore.YELLOW + \"Modality............:\",Style.RESET_ALL, dataset.Modality)\n print(Fore.GREEN + \"Body Part Examined..:\",Style.RESET_ALL, dataset.BodyPartExamined)\n \n if 'PixelData' in dataset:\n rows = int(dataset.Rows)\n cols = int(dataset.Columns)\n print(Fore.BLUE + \"Image size.......:\",Style.RESET_ALL,\" {rows:d} x {cols:d}, {size:d} bytes\".format(\n rows=rows, cols=cols, size=len(dataset.PixelData)))\n if 'PixelSpacing' in dataset:\n print(Fore.YELLOW + \"Pixel spacing....:\",Style.RESET_ALL,dataset.PixelSpacing)\n dataset.PixelSpacing = [1, 1]\n plt.figure(figsize=(10, 10))\n plt.imshow(dataset.pixel_array, cmap='gray')\n plt.show()\nfor file_path in glob.glob('../input/osic-pulmonary-fibrosis-progression/train/*/*.dcm'):\n dataset = pydicom.dcmread(file_path)\n show_dcm_info(dataset)\n break # Comment this out to see all","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/yeayates21/osic-simple-image-eda\n\nimdir = \"/kaggle/input/osic-pulmonary-fibrosis-progression/train/ID00123637202217151272140\"\nprint(\"total images for patient ID00123637202217151272140: \", len(os.listdir(imdir)))\n\n# view first (columns*rows) images in order\nfig=plt.figure(figsize=(12, 12))\ncolumns = 4\nrows = 5\nimglist = os.listdir(imdir)\nfor i in range(1, columns*rows +1):\n filename = imdir + \"/\" + str(i) + \".dcm\"\n ds = pydicom.dcmread(filename)\n fig.add_subplot(rows, columns, i)\n plt.imshow(ds.pixel_array, cmap='gray')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/yeayates21/osic-simple-image-eda\n\nimdir = \"/kaggle/input/osic-pulmonary-fibrosis-progression/train/ID00123637202217151272140\"\nprint(\"total images for patient ID00123637202217151272140: \", len(os.listdir(imdir)))\n\n# view first (columns*rows) images in order\nfig=plt.figure(figsize=(12, 12))\ncolumns = 4\nrows = 5\nimglist = os.listdir(imdir)\nfor i in range(1, columns*rows +1):\n filename = imdir + \"/\" + str(i) + \".dcm\"\n ds = pydicom.dcmread(filename)\n fig.add_subplot(rows, columns, i)\n plt.imshow(ds.pixel_array, cmap='jet')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualization using gif\n* https://www.kaggle.com/danpresil1/dicom-basic-preprocessing-and-visualization"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"apply_resample = False\n\ndef load_scan(path):\n slices = [pydicom.read_file(path + '/' + s) for s in os.listdir(path)]\n slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))\n try:\n slice_thickness = np.abs(slices[0].ImagePositionPatient[2] - slices[1].ImagePositionPatient[2])\n except:\n slice_thickness = np.abs(slices[0].SliceLocation - slices[1].SliceLocation)\n \n for s in slices:\n s.SliceThickness = slice_thickness\n \n return slices","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def load_scan(path):\n slices = [pydicom.read_file(path + '/' + s) for s in os.listdir(path)]\n slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))\n try:\n slice_thickness = np.abs(slices[0].ImagePositionPatient[2] - slices[1].ImagePositionPatient[2])\n except:\n slice_thickness = np.abs(slices[0].SliceLocation - slices[1].SliceLocation)\n \n for s in slices:\n s.SliceThickness = slice_thickness\n \n return slices","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def get_pixels_hu(slices):\n image = np.stack([s.pixel_array for s in slices])\n # Convert to int16 (from sometimes int16), \n # should be possible as values should always be low enough (<32k)\n image = image.astype(np.int16)\n\n # Set outside-of-scan pixels to 0\n # The intercept is usually -1024, so air is approximately 0\n image[image == -2000] = 0\n \n # Convert to Hounsfield units (HU)\n for slice_number in range(len(slices)):\n \n intercept = slices[slice_number].RescaleIntercept\n slope = slices[slice_number].RescaleSlope\n \n if slope != 1:\n image[slice_number] = slope * image[slice_number].astype(np.float64)\n image[slice_number] = image[slice_number].astype(np.int16)\n \n image[slice_number] += np.int16(intercept)\n \n return np.array(image, dtype=np.int16)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def set_lungwin(img, hu=[-1200., 600.]):\n lungwin = np.array(hu)\n newimg = (img-lungwin[0]) / (lungwin[1]-lungwin[0])\n newimg[newimg < 0] = 0\n newimg[newimg > 1] = 1\n newimg = (newimg * 255).astype('uint8')\n return newimg","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scans = load_scan('../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/')\nscan_array = set_lungwin(get_pixels_hu(scans))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Resample to 1mm (An optional step, it may not be relevant to this competition because of the large slice thickness on the z axis)\n\nfrom scipy.ndimage.interpolation import zoom\n\ndef resample(imgs, spacing, new_spacing):\n new_shape = np.round(imgs.shape * spacing / new_spacing)\n true_spacing = spacing * imgs.shape / new_shape\n resize_factor = new_shape / imgs.shape\n imgs = zoom(imgs, resize_factor, mode='nearest')\n return imgs, true_spacing, new_shape\n\nspacing_z = (scans[-1].ImagePositionPatient[2] - scans[0].ImagePositionPatient[2]) / len(scans)\n\nif apply_resample:\n scan_array_resample = resample(scan_array, np.array(np.array([spacing_z, *scans[0].PixelSpacing])), np.array([1.,1.,1.]))[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import imageio\nfrom IPython.display import Image\n\nimageio.mimsave(\"/tmp/gif.gif\", scan_array, duration=0.0001)\nImage(filename=\"/tmp/gif.gif\", format='png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualization using Animation\n* https://www.kaggle.com/pranavkasela/animating-the-lung-ct-scan"},{"metadata":{},"cell_type":"markdown","source":"For animation, I used scan_array in 'visualization using gif' section."},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import matplotlib.animation as animation\n\nfig = plt.figure()\n\nims = []\nfor image in scan_array:\n im = plt.imshow(image, animated=True, cmap=\"Greys\")\n plt.axis(\"off\")\n ims.append([im])\n\nani = animation.ArtistAnimation(fig, ims, interval=100, blit=False,\n repeat_delay=1000)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"HTML(ani.to_jshtml())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"HTML(ani.to_html5_video())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7. Extracting DIOCOM files information in a dataframe 🌊"},{"metadata":{},"cell_type":"markdown","source":"* https://www.kaggle.com/trsekhar123/nb-to-extract-metadata-and-resize-images-train"},{"metadata":{"trusted":true},"cell_type":"code","source":"def extract_dicom_meta_data(filename: str) -> Dict:\n # Load image\n \n image_data = pydicom.read_file(filename)\n img=np.array(image_data.pixel_array).flatten()\n row = {\n 'Patient': image_data.PatientID,\n 'body_part_examined': image_data.BodyPartExamined,\n 'image_position_patient': image_data.ImagePositionPatient,\n 'image_orientation_patient': image_data.ImageOrientationPatient,\n 'photometric_interpretation': image_data.PhotometricInterpretation,\n 'rows': image_data.Rows,\n 'columns': image_data.Columns,\n 'pixel_spacing': image_data.PixelSpacing,\n 'window_center': image_data.WindowCenter,\n 'window_width': image_data.WindowWidth,\n 'modality': image_data.Modality,\n 'StudyInstanceUID': image_data.StudyInstanceUID,\n 'SeriesInstanceUID': image_data.StudyInstanceUID,\n 'StudyID': image_data.StudyInstanceUID, \n 'SamplesPerPixel': image_data.SamplesPerPixel,\n 'BitsAllocated': image_data.BitsAllocated,\n 'BitsStored': image_data.BitsStored,\n 'HighBit': image_data.HighBit,\n 'PixelRepresentation': image_data.PixelRepresentation,\n 'RescaleIntercept': image_data.RescaleIntercept,\n 'RescaleSlope': image_data.RescaleSlope,\n 'img_min': np.min(img),\n 'img_max': np.max(img),\n 'img_mean': np.mean(img),\n 'img_std': np.std(img)}\n\n return row","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_image_path = '/kaggle/input/osic-pulmonary-fibrosis-progression/train'\ntrain_image_files = glob.glob(os.path.join(train_image_path, '*', '*.dcm'))\n\nmeta_data_df = []\nfor filename in tqdm.tqdm(train_image_files):\n try:\n meta_data_df.append(extract_dicom_meta_data(filename))\n except Exception as e:\n continue","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The meta elements in dcom file seem to be slightly different for each file. So, I use Exception.\n- Be careful!"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert to a pd.DataFrame from dict\nmeta_data_df = pd.DataFrame.from_dict(meta_data_df)\nmeta_data_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The code below still makes sense, so I leave it."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# source: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154658\nfolder='train'\nPATH='../input/osic-pulmonary-fibrosis-progression/'\n\nlast_index = 2\n\ncolumn_names = ['image_name', 'dcm_ImageOrientationPatient', \n 'dcm_ImagePositionPatient', 'dcm_PatientID',\n 'dcm_PatientName', 'dcm_PatientSex'\n 'dcm_rows', 'dcm_columns']\n\ndef extract_DICOM_attributes(folder):\n patients_folder = list(os.listdir(os.path.join(PATH, folder)))\n df = pd.DataFrame()\n \n i = 0\n \n for patient_id in patients_folder:\n \n img_path = os.path.join(PATH, folder, patient_id)\n \n print(img_path)\n \n images = list(os.listdir(img_path))\n \n #df = pd.DataFrame()\n\n for image in images:\n image_name = image.split(\".\")[0]\n\n dicom_file_path = os.path.join(img_path,image)\n dicom_file_dataset = pydicom.read_file(dicom_file_path)\n \n '''\n print(dicom_file_dataset.dir(\"pat\"))\n print(dicom_file_dataset.data_element(\"ImageOrientationPatient\"))\n print(dicom_file_dataset.data_element(\"ImagePositionPatient\"))\n print(dicom_file_dataset.data_element(\"PatientID\"))\n print(dicom_file_dataset.data_element(\"PatientName\"))\n print(dicom_file_dataset.data_element(\"PatientSex\"))\n '''\n \n imageOrientationPatient = dicom_file_dataset.ImageOrientationPatient\n #imagePositionPatient = dicom_file_dataset.ImagePositionPatient\n patientID = dicom_file_dataset.PatientID\n patientName = dicom_file_dataset.PatientName\n patientSex = dicom_file_dataset.PatientSex\n \n rows = dicom_file_dataset.Rows\n cols = dicom_file_dataset.Columns\n \n #print(rows)\n #print(columns)\n \n temp_dict = {'image_name': image_name, \n 'dcm_ImageOrientationPatient': imageOrientationPatient,\n #'dcm_ImagePositionPatient':imagePositionPatient,\n 'dcm_PatientID': patientID, \n 'dcm_PatientName': patientName,\n 'dcm_PatientSex': patientSex,\n 'dcm_rows': rows,\n 'dcm_columns': cols}\n\n\n df = df.append([temp_dict])\n \n i += 1\n \n if i == last_index:\n break\n \n return df\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"extract_DICOM_attributes('train')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The meta elements in dcom file seem to be slightly different for each file.\n- Be careful!"},{"metadata":{},"cell_type":"markdown","source":"# Etc. - Pandas Profiling 🌤️"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas_profiling as pdp","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\ntest_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"profile_train_df = pdp.ProfileReport(train_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"profile_train_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"profile_test_df = pdp.ProfileReport(test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"profile_test_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# References\n- https://www.kaggle.com/piantic/osic-pulmonary-fibrosis-progression-basic-eda"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}
\n