{"cells":[{"metadata":{},"cell_type":"markdown","source":"![](https://www.osicild.org/uploads/1/2/2/7/122798879/editor/kaggle-v01-clipped_2.png?1569348761)\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# About Pulmonary Fibrosis","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> **What is Pulmonary Fibrosis??** \n\n> The word **“pulmonary”** means **lung** and the word **“fibrosis”** means **scar tissue**— similar to scars that you may have on your skin from an old injury or surgery. So, in its simplest sense, pulmonary fibrosis (PF) means **scarring in the lungs**. Over time, the scar tissue can destroy the normal lung and make it hard for oxygen to get into your blood.\n\n\n\n\n![](https://www.pulmonaryfibrosis.org/images/default-source/default-album/normal-and-impaired-gas-exchange.png?sfvrsn=c3b0918d_0)\n\n\n\n\n> **Symptoms and Causes**\n\n> There are five main categories of identifiable causes of pulmonary fibrosis: **Drug-induced, Radiation-induced, Environmental, Autoimmune, and Occupational**. Sometimes it can be challenging for doctors to figure out what causes PF.  PF of unknown cause is called **“idiopathic”**.\n\n\n> **Treatment Options**\n\n> As given in the description, current methods make fibrotic lung diseases **difficult to treat**, even with access to a chest CT scan. In addition, the wide range of varied prognoses create issues organizing clinical trials.\n\n\n> **So what do you mean by Prognosis?**\n\n>**Prognosis** is a medical term for predicting the likely or expected development of a disease, including whether the signs and symptoms will improve or worsen (and how quickly) or remain stable over time; expectations of quality of life, such as the ability to carry out daily activities; the potential for complications and associated health issues; and the likelihood of survival (including life expectancy). ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Reference:**[https://www.pulmonaryfibrosis.org/life-with-pf/about-pf](http://)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# About the Competition","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> **What do you need to do?**\n\n> In this competition, you’ll predict a patient’s severity of decline in lung function based on a CT scan of their lungs. You’ll determine lung function based on output from a spirometer, which measures the volume of air inhaled and exhaled. The challenge is to use machine learning techniques to make a prediction with the image, metadata, and baseline FVC as input.\n\n\n> **Submission**\n\n> The test set consists of Three_Patient_Week(s) per patient. You need to predict the **Forced vital capacity(FVC)** i.e. volume of air exhaled and the **confidence** value in your prediction.\n\n\n>**Your Evaluation Metric**\n\n> For each true FVC measurement, you will predict both an FVC and a confidence measure (standard deviation σ). The metric is computed as:\n\n\\begin{equation} \n\\sigma_{clipped} = max(\\sigma, 70)\\\\\n\\Delta = min ( |FVC_{true} - FVC_{predicted}|, 1000 )\\\\\nmetric = -   \\frac{\\sqrt{2} \\Delta}{\\sigma_{clipped}} - \\ln ( \\sqrt{2} \\sigma_{clipped} )\n\\end{equation}\n\n\n> A great notebook to understand the metric --> https://www.kaggle.com/rohanrao/osic-understanding-laplace-log-likelihood","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Let's get started!","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"import os\nfrom os import listdir\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n\nimport plotly.express as px\nimport plotly.graph_objs as go\n\n\nimport pydicom\nimport glob\nimport imageio\nfrom IPython.display import Image\n\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Reading the data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# List of files available\nlist(os.listdir(\"../input/osic-pulmonary-fibrosis-progression\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Data Description**\n\n* train.csv - the training set, contains full history of clinical information\n* test.csv - the test set, contains only the baseline measurement\n* train/ - contains the training patients' baseline CT scan in DICOM format\n* test/ - contains the test patients' baseline CT scan in DICOM format\n* sample_submission.csv - demonstrates the submission format","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\ntest_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\n\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**What does the columns represent?**\n\n* **Patient**- a unique Id for each patient (also the name of the patient's DICOM folder)\n* **Weeks**- the relative number of weeks pre/post the baseline CT (may be negative)\n* **FVC** - the recorded lung capacity in ml\n* **Percent**- a computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics\n* **Age**- Age of the patient\n* **Sex**\n* **SmokingStatus**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Data Exploration","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Shape of the Data**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Shape of Training data: ', train_df.shape)\nprint('Shape of Test data: ', test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interesting to note that the test set consists of only 5 images. Moreover, as given in the description, the provided test set is a small representative set of files (**copied from the training set**) to demonstrate the format of the private test set.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Null count and Datatype**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"No missing value in either train/test csv","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Unique IDs**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"The total patient ids are {train_df['Patient'].count()}\")\nprint(f\"Number of unique ids are {train_df['Patient'].value_counts().shape[0]} \")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are multiple records of the same patient as the number of unique ids are less than total patient ids record.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Visualizing the Data","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"new_df = train_df.groupby([train_df.Patient,train_df.Age,train_df.Sex, train_df.SmokingStatus])['Patient'].count()\nnew_df.index = new_df.index.set_names(['id','Age','Sex','SmokingStatus'])\nnew_df = new_df.reset_index()\nnew_df.rename(columns = {'Patient': 'freq'},inplace = True)\nnew_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The 'freq' column represents number of observations for that patient","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.bar(new_df, x='id',y ='freq',color='freq')\nfig.update_layout(xaxis={'categoryorder':'total ascending'},title='No. of observations for each patient')\nfig.update_xaxes(showticklabels=False)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The number of oberservations for every unique patient in the train csv ranges from 6 to 10 wherein most of them have 9 observations.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.histogram(new_df, x='Age',nbins = 42)\nfig.update_traces(marker_color='rgb(158,202,225)', marker_line_color='rgb(8,48,107)',\n                 marker_line_width=1.5, opacity=0.6)\nfig.update_layout(title = 'Distribution of Age for unique patients')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We notice the range of age to be between 48-88 where we have more records for patients in the age range 64-74.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.histogram(new_df, x='Sex')\nfig.update_traces(marker_color='rgb(202,158,225)', marker_line_color='rgb(48,8,107)',\n                 marker_line_width=2, opacity=0.8)\nfig.update_layout(title = 'Distribution of Sex for unique patients')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"More number of male patients than female patients.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.histogram(new_df, x='SmokingStatus')\nfig.update_traces(marker_color='rgb(202,225,158)', marker_line_color='rgb(48,107,8)',\n                 marker_line_width=2, opacity=0.8)\nfig.update_layout(title = 'Distribution of SmokingStatus for unique patients')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A big chunk of data is of patients who are Ex-smokers whereas very few patients who currently smoke.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.histogram(new_df, x='SmokingStatus',color = 'Sex')\nfig.update_traces(marker_line_color='black',marker_line_width=2, opacity=0.85)\nfig.update_layout(title = 'Distribution of SmokingStatus for unique patients')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Records with patient who have never smoked have equal distribution of male and female patients whereas a large majority of ex-smokers are males.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.histogram(new_df, x='Age',color = 'Sex',color_discrete_map={'Male':'#EB89B5','Female':'#330C73'},marginal = 'rug',hover_data = new_df.columns)\nfig.update_layout(title = 'Distribution of Age w.r.t Sex for unique patients')\nfig.update_traces(marker_line_color='black',marker_line_width=1.5, opacity=0.85)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Male and female records are almost distributed throughout the age range.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.histogram(new_df, x='Age',color = 'SmokingStatus',color_discrete_map={'Male':'#EB89B5','Female':'#330C73'},marginal = 'rug',hover_data = new_df.columns)\nfig.update_layout(title = 'Distribution of Age w.r.t SmokingStatus for unique patients')\nfig.update_traces(marker_line_color='black',marker_line_width=1.5, opacity=0.85)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> **NOTE: Double click on the side legend to isolate a category**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Patients who currently smoke show only a few occurrence along the age range.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import plotly.figure_factory as ff\ndf1 = train_df.groupby('SmokingStatus').get_group('Ex-smoker')\ndf2 = train_df.groupby('SmokingStatus').get_group('Never smoked')\ndf3 = train_df.groupby('SmokingStatus').get_group('Currently smokes')\n\nhist_data = [df1['FVC'], df2['FVC'], df3['FVC']]\n\ngroup_labels = ['Ex-Smokers', 'Never Smoked', 'Current Smokers']\ncolors = ['#393E46', '#2BCDC1', '#F66095']\n\nfig = ff.create_distplot(hist_data, group_labels, colors=colors,bin_size=50,\n                         show_curve=True)\n\n# Add title\nfig.update(layout_title_text='Distribution of FVC categorized by Smoking Status')\nfig.update_layout( width=700,height=600)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The value of FVC for current smokers is mostly concentrated around 3000.\n* For patients who never smoked, the value remains below 4400.\n* For Ex-smokers we see a few higher values around 6000 and a large number of records are between 2000 and 3000","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df1 = train_df.groupby('SmokingStatus').get_group('Ex-smoker')\ndf2 = train_df.groupby('SmokingStatus').get_group('Never smoked')\ndf3 = train_df.groupby('SmokingStatus').get_group('Currently smokes')\n\nhist_data = [df1['Percent'], df2['Percent'], df3['Percent']]\n\ngroup_labels = ['Ex-Smokers', 'Never Smoked', 'Current Smokers']\ncolors = ['#393E46', '#2BCDC1', '#F66095']\n\nfig = ff.create_distplot(hist_data, group_labels, colors=colors,bin_size=1,\n                         show_curve=True)\n\n# Add title\nfig.update(layout_title_text='Distribution of Percent categorized by Smoking Status')\nfig.update_layout( width=700,height=600)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> **NOTE: Double click on the side legend to isolate a category**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Where most values for 'Percent'(the patient's FVC as a percent of the typical FVC for a person of similar characteristics) ranges from 40-120, there is a chunk of data for Current smokers which have values above 140.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"patient1 = train_df[train_df.Patient == 'ID00007637202177411956430']\npatient2 = train_df[train_df.Patient == 'ID00012637202177665765362']\npatient3 = train_df[train_df.Patient == 'ID00082637202201836229724']\n\n\npatient1['text'] ='ID: ' + (patient1['Patient']).astype(str) + '<br>FVC ' + patient1['FVC'].astype(str) + '<br>Percent ' + patient1['Percent'].astype(str) + '<br>Week ' + patient1['Weeks'].astype(str)\npatient2['text'] ='ID: ' + (patient2['Patient']).astype(str) + '<br>FVC ' + patient2['FVC'].astype(str)+ '<br>Percent ' + patient2['Percent'].astype(str)  + '<br>Week ' + patient2['Weeks'].astype(str)\npatient3['text'] ='ID: ' + (patient3['Patient']).astype(str) + '<br>FVC ' + patient3['FVC'].astype(str) + '<br>Percent ' + patient3['Percent'].astype(str) + '<br>Week ' + patient3['Weeks'].astype(str)\n\n\nfig = go.Figure()\nfig.add_trace(go.Scatter(x=patient1['Weeks'], y=patient1['FVC'],hovertext = patient1['text'],\n                    mode='lines+markers',marker=dict(size = 12,line_width = 2),\n                    name='Ex-smoker'))\nfig.add_trace(go.Scatter(x=patient2['Weeks'], y=patient2['FVC'],hovertext = patient2['text'],\n                    mode='lines+markers',marker=dict(size = 12,line_width = 2),\n                    name='Never smoked'))\nfig.add_trace(go.Scatter(x=patient3['Weeks'], y=patient3['FVC'],hovertext = patient3['text'],\n                    mode='lines+markers',marker=dict(size = 12,line_width = 2), name='Currently smokes'))\n\nfig.update(layout_title_text='FVC vs Weeks for 3 different patients')\nfig.update_layout( width=700,height=500)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"patient1['text'] ='ID: ' + (patient1['Patient']).astype(str) + '<br>Percent ' + patient1['Percent'].astype(str) + '<br>FVC ' + patient1['FVC'].astype(str) + '<br>Week ' + patient1['Weeks'].astype(str)\npatient2['text'] ='ID: ' + (patient2['Patient']).astype(str) + '<br>Percent ' + patient2['Percent'].astype(str) + '<br>FVC ' + patient2['FVC'].astype(str) + '<br>Week ' + patient2['Weeks'].astype(str)\npatient3['text'] ='ID: ' + (patient3['Patient']).astype(str) + '<br>Percent ' + patient3['Percent'].astype(str) + '<br>FVC ' + patient3['FVC'].astype(str) + '<br>Week ' + patient3['Weeks'].astype(str)\n\n\nfig = go.Figure()\nfig.add_trace(go.Scatter(x=patient1['Weeks'], y=patient1['Percent'],hovertext = patient1['text'],\n                    mode='lines+markers',marker=dict(size = 12,line_width = 2),\n                    name='Ex-smoker'))\nfig.add_trace(go.Scatter(x=patient2['Weeks'], y=patient2['Percent'],hovertext = patient2['text'],\n                    mode='lines+markers',marker=dict(size = 12,line_width = 2),\n                    name='Never smoked'))\nfig.add_trace(go.Scatter(x=patient3['Weeks'], y=patient3['Percent'],hovertext = patient3['text'],\n                    mode='lines+markers',marker=dict(size = 12,line_width = 2), name='Currently smokes'))\n\nfig.update(layout_title_text='Percent vs Weeks for 3 different patients')\nfig.update_layout( width=700,height=500)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Notice that the **'FVC'** value for **Non-smoker** was the **highest** but trends for **'Percent'** for Non-smoker is less than that of the patient of **currently smokes**.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Visualizing Images","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We have been provided with DICOM files or \"Digital Imaging and Communications in Medicine\" format. It contains an image from a medical scan, like a CT scan + information about the patient. . A DICOM file has two parts: the header and the dataset. The header contains information on the encapsulated dataset. It consists of a File Preamble, a DICOM prefix, and the File Meta Elements. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**HAVING A FIRST LOOK AT THE IMAGE**","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"img = \"../input/osic-pulmonary-fibrosis-progression/train/ID00009637202177434476278/100.dcm\"\nds = pydicom.dcmread(img)\nplt.figure(figsize = (10,10))\nplt.imshow(ds.pixel_array, cmap=plt.cm.bone)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Let's view all of the images for the first patient**","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"\nimage_dir = '../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430'\n\nfig=plt.figure(figsize=(10,10))\ncolumns = 5\nrows = 6\nimage_list = os.listdir(image_dir)\nfor i in range(1, columns*rows +1):\n    filename = image_dir + \"/\" + str(i) + \".dcm\"\n    ds = pydicom.dcmread(filename)\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(ds.pixel_array, cmap=plt.cm.bone)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Looking at the previous two visuals, we notice that the **scans** for both the patients are **different**. Let's have a closer look at the scans of both patients.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"image1 = '../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/8.dcm'\nimage2 = \"../input/osic-pulmonary-fibrosis-progression/train/ID00009637202177434476278/100.dcm\"\n\nfig, ax = plt.subplots(1, 2, figsize=(10, 10))\nds = pydicom.dcmread(image1)\nax[0].set_title('Patient 1')\nax[0].imshow(ds.pixel_array, cmap=plt.cm.bone)\n\nds = pydicom.dcmread(image2)\nax[1].set_title('Patient 2')\nax[1].imshow(ds.pixel_array, cmap=plt.cm.bone)\n\nplt.show","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Notice that the first scan has a circular border and the second one is regular. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Creating An Animation!!**\n\n> If you have seen my previous works, you must know how much I love experimenting with animations. Thanks to Dan Presli for his work https://www.kaggle.com/danpresil1/dicom-basic-preprocessing-and-visualization that I was able to include this animation in my notebook.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def load_scan(path):\n    slices = [pydicom.read_file(path + '/' + s) for s in os.listdir(path)]\n    slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))\n    try:\n        slice_thickness = np.abs(slices[0].ImagePositionPatient[2] - slices[1].ImagePositionPatient[2])\n    except:\n        slice_thickness = np.abs(slices[0].SliceLocation - slices[1].SliceLocation)\n        \n    for s in slices:\n        s.SliceThickness = slice_thickness\n        \n    return slices\n\n\ndef get_pixels_hu(slices):\n    image = np.stack([s.pixel_array for s in slices])\n    # Convert to int16 (from sometimes int16), \n    # should be possible as values should always be low enough (<32k)\n    image = image.astype(np.int16)\n\n    # Set outside-of-scan pixels to 0\n    # The intercept is usually -1024, so air is approximately 0\n    image[image == -2000] = 0\n    \n    # Convert to Hounsfield units (HU)\n    for slice_number in range(len(slices)):\n        \n        intercept = slices[slice_number].RescaleIntercept\n        slope = slices[slice_number].RescaleSlope\n        \n        if slope != 1:\n            image[slice_number] = slope * image[slice_number].astype(np.float64)\n            image[slice_number] = image[slice_number].astype(np.int16)\n            \n        image[slice_number] += np.int16(intercept)\n    \n    return np.array(image, dtype=np.int16)\n\ndef set_lungwin(img, hu=[-1200., 600.]):\n    lungwin = np.array(hu)\n    newimg = (img-lungwin[0]) / (lungwin[1]-lungwin[0])\n    newimg[newimg < 0] = 0\n    newimg[newimg > 1] = 1\n    newimg = (newimg * 255).astype('uint8')\n    return newimg","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"scans = load_scan('../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/')\nscan_array = set_lungwin(get_pixels_hu(scans))\n\n\n\nimageio.mimsave(\"/tmp/gif.gif\", scan_array, duration=0.0001)\nImage(filename=\"/tmp/gif.gif\", format='png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Work in progress.. Stay tuned for more! Do leave your views and suggestions!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}