{"cells":[{"metadata":{},"cell_type":"markdown","source":"### >>>>>>>>>> for using interactive features please fork it and then run <<<<<<<<<<\n\n![](https://www.osicild.org/uploads/1/2/2/7/122798879/editor/kaggle-v01-clipped_2.png?1569348761)\n## If you want to know details about the Pulmonary Fibrosis Progression, check out my other kernel [Pulmonary Fibrosis for Non-Med People](https://www.kaggle.com/redwankarimsony/pulmonary-fibrosis-for-non-med-people) which contains detailed information about the the disease, symptoms, causes and many more. If you like this kernel, please upvote.. ","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n\n#plotly imports\n!pip install chart_studio\nimport plotly.express as px\nimport chart_studio.plotly as py\nimport plotly.graph_objs as go\nfrom plotly.offline import iplot\nimport cufflinks\ncufflinks.go_offline()\ncufflinks.set_config_file(world_readable=True, theme='pearl')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1. Load Dataframes","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_dir = '../input/osic-pulmonary-fibrosis-progression/train/'\ntest_dir = '../input/osic-pulmonary-fibrosis-progression/test/'\n\n\ntrain_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\ntest_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\nprint('Training Dataframe shape: ', train_df.shape)\n\ntrain_df.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2. Train and Test Dataframe Details","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's have a look at the detailed info about the dataframes\nprint('Training Dataframe Details: ')\nprint(train_df.info())\n\nprint('\\n\\nTest Dataframe Details: ')\nprint(test_df.info())\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3. Number of Patients","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Number of patients in training set:',\n      len(os.listdir(train_dir)))\nprint('Number of patients in test set:',\n     len(os.listdir(test_dir)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4. Creating Individual Patient Profiles \nLet's create a dataframe that will contain all the unique patient IDs. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating unique patient lists and their properties. \npatient_ids = os.listdir(train_dir)\npatient_ids = sorted(patient_ids)\n\n#Creating new rows\nno_of_instances = []\nage = []\nsex = []\nsmoking_status = []\n\nfor patient_id in patient_ids:\n    patient_info = train_df[train_df['Patient'] == patient_id].reset_index()\n    no_of_instances.append(len(os.listdir(train_dir + patient_id)))\n    age.append(patient_info['Age'][0])\n    sex.append(patient_info['Sex'][0])\n    smoking_status.append(patient_info['SmokingStatus'][0])\n\n#Creating the dataframe for the patient info    \npatient_df = pd.DataFrame(list(zip(patient_ids, no_of_instances, age, sex, smoking_status)), \n                                 columns =['Patient', 'no_of_instances', 'Age', 'Sex', 'SmokingStatus'])\nprint(patient_df.info())\npatient_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5. Gender Distribution\nIt seems that among the unique patients 78% are Male and rest of them are female. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df['Sex'].value_counts(normalize = True).iplot(kind = 'bar', \n                                                        color = 'blue', \n                                                        yTitle = 'Unique patient count',\n                                                        xTitle = 'Gender',\n                                                        title = 'Gender Distribution of the unique patients')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 6. Age Distribution of the Patients\nNow let's have a look at the patients age distributions. We observe that the age distribution starts from near 50 years and it tops to 90 years. Therefore, the given data about the pulmonary fibrosis is mainly of the older people.  ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import scipy\n\ndata = patient_df.Age.tolist()\nplt.figure(figsize=(18,6))\n# Creating the main histogram\n_, bins, _ = plt.hist(data, 15, density=1, alpha=0.5)\n\n# Creating the best fitting line with mean and standard deviation\nmu, sigma = scipy.stats.norm.fit(data)\nbest_fit_line = scipy.stats.norm.pdf(bins, mu, sigma)\nplt.plot(bins, best_fit_line, color = 'b', linewidth = 3, label = 'fitting curve')\nplt.title(f'Age Distribution [ mean = {\"{:.2f}\".format(mu)}, standard_dev = {\"{:.2f}\".format(sigma)} ]', fontsize = 18)\nplt.xlabel('Age -->')\nplt.show()\n\npatient_df['Age'].iplot(kind='hist',bins=25,color='blue',xTitle='Percent distribution',yTitle='Count')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"let's see the average age of patients based on their sex. However, it is observed that the average age of male and female patients are almost same. However, male patients are mostly sick near the age of 70 ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsns.kdeplot(patient_df.loc[patient_df['Sex'] == 'Male', 'Age'], label = 'Male',shade=True)\nsns.kdeplot(patient_df.loc[patient_df['Sex'] == 'Female', 'Age'], label = 'Female',shade=True)\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 7. Study of SmokingStatus \nIt seems that almost 70% of the patients are ex-smokers (at some point of their lives, they smoked). However, among the patients about 27% of the patients have never smoked in their life. Hilarious thing is that 5% of the patients are still smoking.. LOL","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df['SmokingStatus'].value_counts(normalize=False ).iplot(kind='bar',\n                                                      yTitle='Percentage', \n                                                      linecolor='black', \n                                                      opacity=0.8,\n                                                      color='blue',\n                                                      theme='pearl',\n                                                      bargap=0.5,\n                                                      title='SmokingStatus Distribution')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 8. Study of Gender vs SmokingStatus","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df.groupby(['SmokingStatus', 'Sex']).count()['Patient'].unstack().iplot(kind='bar', \n                                                                                yTitle = 'Unique Patient Count',\n                                                                                title = 'Gender vs SmokingStatus' )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 9. Age Distribution of the Patients based on Smoking Status","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsns.kdeplot(patient_df.loc[patient_df['SmokingStatus'] == 'Ex-smoker', 'Age'], label = 'Ex-smoker',shade=True)\nsns.kdeplot(patient_df.loc[patient_df['SmokingStatus'] == 'Never smoked', 'Age'], label = 'Never smoked',shade=True)\nsns.kdeplot(patient_df.loc[patient_df['SmokingStatus'] == 'Currently smokes', 'Age'], label = 'Currently smokes',shade=True)\n# Labeling of plot\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 10. Interactive Patient Lookup (FVC Decay)\nIf you run the following cell, you will get an dropdown menu where you can select any patient ID and it will show you the details present for that patient in the training data frame. It will show you all the information of that patient from the train data frame and also plot the gradual fibrosis progression over time. It plots the decrease of FVC over time.  \n#### Observations:\n* ***Smoking Status doesn't change over time.*** So smoking status for a single patient is always unique. \n* ***Age of the patient doesn't change over time*** Whatever the age in the beginning of the diagnosis, even after more than 52 weeks patient's age didn't change in the dataframe. \n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from ipywidgets import interact  #, interactive, IntSlider, ToggleButtons\n\ndef patient_lookup(patient_id):\n    print(train_df[train_df['Patient'] == patient_id])\n    patient_info = train_df[train_df['Patient'] == patient_id].reset_index()\n    fig, (ax1,ax2) = plt.subplots(1,2, figsize = (15, 5))\n    ax1.plot(patient_info['Weeks'].tolist() , patient_info['FVC'].tolist(), marker = '*', linewidth = 3,color = 'r', markeredgecolor = 'b')\n    ax1.set_title('FVC Deterioriation over the Weeks')\n    ax1.set_xlabel('Weeks -->')\n    ax1.set_ylabel('FVC')\n    ax1.grid(True)\n    \n    ax2.plot(patient_info['Weeks'].tolist() , patient_info['Percent'].tolist(),marker = '*', linewidth = 3,\n            color = 'r', markeredgecolor = 'b' )\n    ax2.set_title('Percent change over the weeks')\n    ax2.set_xlabel('Weeks -->')\n    ax2.set_ylabel('Percent(of adult capacity)')\n    ax2.grid(True)\n    fig.suptitle(f'P_ID: {patient_id}', fontsize = 20) \n    \n    \n    \ninteract(patient_lookup, patient_id = patient_ids)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 11. Interactive Patient Lookup (CT Scans)\nHere from the dropdown menu, please select any patient id, and it will show you 16 CT-Scans over time for that particular patient. ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import random\nimport pydicom\ndef explore_dicoms(patient_id, instance):\n    RefDs = pydicom.dcmread('../input/osic-pulmonary-fibrosis-progression/train/' + \n                            patient_id +\n                            '/' + \n                            str(instance) + '.dcm')\n    plt.figure(figsize=(10, 5))\n    plt.imshow(RefDs.pixel_array, cmap='gray');\n    plt.title(f'P_ID: {patient_id}\\nInstance: {instance}')\n    plt.axis('off')\n\n\ndef show_ct_scans(patient_id):\n    no_of_instances = int(patient_df[patient_df['Patient'] == patient_id]['no_of_instances'].values[0])\n    files = sorted(random.sample(range(1, no_of_instances), 9))\n    rows = 3\n    cols = 3\n    fig = plt.figure(figsize=(12,12))\n    for idx in range(1, rows*cols+1):\n        fig.add_subplot(rows, cols, idx)\n        RefDs = pydicom.dcmread(train_dir + patient_id + '/' + str(files[idx-1]) + '.dcm')\n        plt.imshow(RefDs.pixel_array, cmap='gray')\n        plt.title(f'Instance: {files[idx-1]}')\n        plt.axis(False)\n        fig.add_subplot\n    fig.suptitle(f'P_ID: {patient_id}') \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# show_ct_scans(patient_ids[0])\ninteract(show_ct_scans,patient_id = patient_ids)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### 12. Sumarizing the Unique Patient Profile","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\nimport pandas_profiling as pdp\nunique_patient_profile  = pdp.ProfileReport(patient_df)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unique_patient_profile","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}