{"cells":[{"metadata":{},"cell_type":"markdown","source":"# OSIC Pulmonary Fibrosis Progression\n## Predict lung function decline\nPulmonary fibrosis is a lung disease that occurs when lung tissue becomes damaged and scarred. This thickened, stiff tissue makes it more difficult for your lungs to work properly. As pulmonary fibrosis worsens, you become progressively more short of breath.\n\nThe scarring associated with pulmonary fibrosis can be caused by a multitude of factors. But in most cases, doctors can't pinpoint what's causing the problem. When a cause can't be found, the condition is termed idiopathic pulmonary fibrosis.\n\nThe lung damage caused by pulmonary fibrosis can't be repaired, but medications and therapies can sometimes help ease symptoms and improve quality of life. For some people, a lung transplant might be appropriate.\n<img src =\"https://www.mayoclinic.org/-/media/kcms/gbs/patient-consumer/images/2016/08/10/14/57/mcdc7_pulmonaryfibrosis-8col.jpg\">\n\n##### Symptoms\nSigns and symptoms of pulmonary fibrosis may include:\n\n- Shortness of breath (dyspnea)\n- A dry cough\n- Fatigue\n- Unexplained weight loss\n- Aching muscles and joints\n- Widening and rounding of the tips of the fingers or toes (clubbing)\n\n##### Data Description\n\n - train.csv - the training set, contains full history of clinical information\n - test.csv - the test set, contains only the baseline measurement\n - train/ - contains the training patients' baseline CT scan in DICOM format\n - test/ - contains the test patients' baseline CT scan in DICOM format\n - sample_submission.csv - demonstrates the submission format\n \n \n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<img src =\"https://media.giphy.com/media/g7teYpDDWjP7q/giphy.gif\"/>","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\n!pip install dexplot -q\nimport dexplot as dxp\nfrom os import listdir\nprint(list(os.listdir(\"../input/osic-pulmonary-fibrosis-progression\")))\nimport pandas as pd\nimport numpy as np\nimport glob\nimport tqdm\nfrom typing import Dict\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n#plotly\nimport plotly.express as px\nimport plotly.graph_objs as go\nfrom plotly.offline import iplot\nimport cufflinks\ncufflinks.go_offline()\ncufflinks.set_config_file(world_readable=True, theme='pearl')\n\n#color\n\nimport seaborn as sns\n#sns.set(style=\"whitegrid\")\n\n#pydicom\nimport pydicom\n\n# Suppress warnings \nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Settings for pretty nice plots\n#plt.style.use('fivethirtyeight')\n#plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"IMAGE_PATH = \"../input/osic-pulmonary-fibrosis-progressiont/\"\n\npatient_train_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\npatient_test_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\n\nprint('Train data shape:',patient_train_df.shape)\nprint('Test data shape:',patient_test_df.shape)\n\nprint('^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^')\nprint('Train data:',patient_train_df.info())\nprint('Test data:',patient_test_df.info())\npatient_train_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> What does the columns represent?\n\n* Patient- a unique Id for each patient (also the name of the patient's DICOM folder)\n* Weeks- the relative number of weeks pre/post the baseline CT (may be negative)\n* FVC - the recorded lung capacity in ml\n* Percent- a computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics\n* Age- Age of the patient\n* Sex\n* SmokingStatus","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_df = patient_train_df[['Patient', 'Age', 'Sex', 'SmokingStatus']].drop_duplicates()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dxp.count('Sex',data=train_df,figsize=(10,6),title='Count of Patient(Sex Wise)')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = train_df['Sex'].value_counts()[:10].index\nvalues = train_df['Sex'].value_counts()[:10].values\ncolors=['#FA8072',\n '#98adbf']\n\nfig = go.Figure(data=[go.Pie(labels=labels, values=values, textinfo='label+percent',\n                             insidetextorientation='radial',marker=dict(colors=colors))])\nfig.update_layout(title = 'Distribution of Sex for unique patients')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dxp.count('SmokingStatus',data=train_df,figsize=(10,6),title='Count of Patient(Smoker Perspective)',size=0.9)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = train_df['SmokingStatus'].value_counts()[:10].index\nvalues = train_df['SmokingStatus'].value_counts()[:10].values\ncolors=['#FA8072',\n '#98adbf']\n\nfig = go.Figure(data=[go.Pie(labels=labels, values=values, textinfo='label+percent',\n                             insidetextorientation='radial',marker=dict(colors=colors))])\nfig.update_layout(title = 'Distribution of Smoking Status')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dxp.hist(val='Age', data=train_df,title=\"Histogram for Age Column\",figsize=(10,6))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dxp.hist(val='Age', data=train_df, orientation='v', split='Sex', bins=15,figsize=(10,6),title=\"Histogram for Age Column(Gender Wise)\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dxp.kde(x='Age', data=train_df, split='SmokingStatus', split_order=['Ex-smoker','Never smoked','Currently smokes'],xlabel='count',figsize=(10,6),title='Histogram for Age')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dxp.bar(x='Sex', y='Age', data=train_df, aggfunc='median', split='SmokingStatus',figsize=(10,6),size=0.9,stacked=True,title='Age Distribution(Smoker Perspective)')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dxp.count(val='Age', data=train_df, split='SmokingStatus',normalize=True,figsize=(10,6),size=0.9,stacked=True,title=\"Smokers(Patient's Age Perspective)\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.crosstab(index=train_df['SmokingStatus'], columns=train_df['Age'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_train_df['Weeks'].iplot(kind='area',fill=True,opacity=1,xTitle='Weeks',yTitle='Days',title='Area Chart of Weeks Column')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Forced expiratory volume (FEV) measures how much air a person can exhale during a forced breath. The amount of air exhaled may be measured during the first (FEV1), second (FEV2), and/or third seconds (FEV3) of the forced breath.\n\n>**Forced vital capacity (FVC) is the total amount of air exhaled during the FEV test.**\n\n>Forced expiratory volume and forced vital capacity are lung function tests that are measured during spirometry. Forced expiratory volume is the most important measurement of lung function. It is used to:\n\n>Diagnose obstructive lung diseases such as asthma and chronic obstructive pulmonary disease (COPD). A person who has asthma or COPD has a lower FEV1 result than a healthy person.\nSee how well medicines used to improve breathing are working.\nCheck if lung disease is getting worse. Decreases in the FEV1 value may mean the lung disease is getting worse","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import scipy\n\ndata = patient_train_df.FVC.tolist()\nplt.figure(figsize=(18,6))\n# Creating the main histogram\n_, bins, _ = plt.hist(data, 15, density=1, alpha=0.5)\n\n# Creating the best fitting line with mean and standard deviation\nmu, sigma = scipy.stats.norm.fit(data)\nbest_fit_line = scipy.stats.norm.pdf(bins, mu, sigma)\nplt.plot(bins, best_fit_line, color = 'green', linewidth = 3, label = 'fitting curve')\nplt.title(f'FVC Distribution [ mean = {\"{:.2f}\".format(mu)}, standard_dev = {\"{:.2f}\".format(sigma)} ]', fontsize = 18)\nplt.xlabel('FVC')\nplt.show()\n\npatient_train_df['FVC'].iplot(kind='hist',bins=25,color='grey',xTitle='Percent distribution',yTitle='Count')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.histogram(patient_train_df, x=\"FVC\", y=\"Age\", color=\"Sex\", hover_data=patient_train_df.columns)\nfig.update_layout(title='Histogram for FVC w.r.t Patient Gender')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dxp.box(x='Sex', y='FVC', data=patient_train_df,orientation='v',figsize=(10,6),title='Boxplot for FVC')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dxp.box(x='FVC', y='SmokingStatus', data=patient_train_df, \n        split='Sex', split_order=['Male', 'Female'],figsize=(10,6),title='Boxplot for FVC (Gender wise)')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = patient_train_df.pivot_table(index='SmokingStatus', columns='Age', \n                        values='FVC', aggfunc='mean')\ndf = df.fillna(0,axis=1)\ndxp.bar(data=df, orientation='v',figsize=(10,6),title='FVC Interpretation with Smoking Status')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dxp.scatter(x='Weeks', y='FVC', data=patient_train_df,regression=True,split='Sex',figsize=(10,6),title='Week Distribution(Gender Perspective)')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dxp.scatter(x='Weeks', y='FVC', data=patient_train_df,regression=True,split='SmokingStatus',figsize=(10,6),title='Week Distribution(Somker Perspective)')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dxp.scatter(x='Weeks', y='Percent', data=patient_train_df,regression=True,split='Sex',figsize=(10,6),title='Week Distribution(Sex Perspective) with Percentage')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"grp = patient_train_df[['Patient','Percent']].groupby(['Patient'])['Percent'].mean() \ngrp = pd.DataFrame(grp).reset_index()\ngrp = grp[:20]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dxp.bar(x='Patient', y='Percent', data=grp,figsize=(20,6),size=0.9,title='Patients with Mean FCV Percentage ',cmap='jet')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = patient_train_df.pivot_table(index='Patient', columns='SmokingStatus', \n                        values='Percent', aggfunc='mean')\ndf = df.fillna(0,axis=1)\ndf=df[:20]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dxp.bar(data=df, orientation='v',figsize=(20,6),title='Top 10 Patients with High FVC Percentage')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Image Visual","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"img = \"../input/osic-pulmonary-fibrosis-progression/train/ID00010637202177584971671/10.dcm\"\nds = pydicom.dcmread(img)\nplt.figure(figsize = (10,6))\nplt.imshow(ds.pixel_array, cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_dir = '../input/osic-pulmonary-fibrosis-progression/train/ID00027637202179689871102'\n\nfig=plt.figure(figsize=(10,10))\ncolumns = 5\nrows = 6\nimage_list = os.listdir(image_dir)\nfor i in range(1, columns*rows +1):\n    filename = image_dir + \"/\" + str(i) + \".dcm\"\n    ds = pydicom.dcmread(filename)\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(ds.pixel_array, cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### References-\n- https://www.kaggle.com/piantic/osic-pulmonary-fibrosis-progression-basic-eda\n- https://www.kaggle.com/jryoungw/what-should-we-consider-when-handling-dicom\n\n### Please Show your Appreciation by Upvotes !!! Keep Learning!! ✌️\n#### Notebook is in WIP.....\n##### Updated Soon.....\n##### Current version-3\n","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}