{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Overview of Pulmonary Fibrosis\n**By Chase Eby**\n\nThis is my first public notebook on kaggle. Im taking this as an opportunity to make a entry point for the Data Exploration Process. Hopefully this will be a good starting point for people new to the Data Science field like me.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Pulmonary Fibrosis Definition**\n\nThe word “pulmonary” means lung and the word “fibrosis” means scar tissue— similar to scars that you may have on your skin from an old injury or surgery. So, in its simplest sense, pulmonary fibrosis (PF) means scarring in the lungs. Over time, the scar tissue can destroy the normal lung and make it hard for oxygen to get into your blood. Low oxygen levels (and the stiff scar tissue itself) can cause you to feel short of breath, particularly when walking and exercising. Pulmonary fibrosis isn’t just one disease. It is a family of more than 200 different lung diseases that all look very much alike. The PF family of lung diseases falls into an even larger group of diseases called the interstitial lung diseases (also known as ILD), which includes all of the diseases that have inflammation and/or scarring in the lung. Some interstitial lung diseases don’t include scar tissue. When an interstitial lung disease does include scar tissue in the lung, we call it pulmonary fibrosis.\n\n**Ideopathic Pulmonary Fibrosis occurs when no cause of the disease can be identified** \n\nSource \"https://www.pulmonaryfibrosis.org/life-with-pf/about-pf\"","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Basic Data Exploration","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\ntest = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Training Set Shape = {train.shape} - Patients = {train[\"Patient\"].nunique()}')\nprint(f'Test Set Shape = {test.shape} - Patients = {test[\"Patient\"].nunique()}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Change plot colors from default to colorblind.\ncb = sns.color_palette(\"colorblind\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Exploring Patients ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check for nulls in the dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Both the training and test sets have zero null values","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"# https://www.kaggle.com/andradaolteanu/pulmonary-fibrosis-competition-eda-dicom-prep\n# Select unique bio info for the patients\ndata = train.groupby(by=\"Patient\")[[\"Patient\", \"Age\", \"Sex\", \"SmokingStatus\"]].first().reset_index(drop=True)\n\nf, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize = (20, 5))\n\n# Distribution of Age\nage = sns.distplot(data['Age'],ax=ax1, color = cb[2])\nage.set_title(\"Patient Age Distribution\", fontsize=16)\n\nsex = sns.countplot(data['Sex'],ax=ax2, palette = cb[3:6])\nsex.set_title(\"Patient Sex\", fontsize=16)\n\nsmoke = sns.countplot(data[\"SmokingStatus\"],ax=ax3, palette = cb[2:5])\nsmoke.set_title(\"Smoking Status\", fontsize=16)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The average age of patients happens to be a little under the age of 70. \n* Most of the patients are male\n    * Lets look to see the different smoking status for each sex\n* Many of the patients are ex-smoker.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(['Sex', 'SmokingStatus'])['FVC'].agg(['mean','std','count'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f, (ax1) = plt.subplots(1, figsize = (8, 6))\nax = sns.swarmplot(x=\"SmokingStatus\", y=\"FVC\", hue=\"Sex\",\n              palette=cb[1:3], data=train)\nax.set_title(\"Smoking Status(FVC Output) for Each Sex\", fontsize=16)\nax","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"According to our graph, women tend to be on the lower end of FVC for all 3 of our categories.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Which category of smoking status is has the highest FVC.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Figure\nf, (ax1, ax2) = plt.subplots(1,2, figsize = (16, 6))\n\na = sns.barplot(x = train[\"SmokingStatus\"], y = train[\"FVC\"], ax=ax1, palette=cb[0:4])\nb = sns.barplot(x = train[\"SmokingStatus\"], y = train[\"Percent\"], ax=ax2, palette=cb[4:7])\n\na.set_title(\"Mean FVC per Smoking Status\", fontsize=16)\nb.set_title(\"Mean Perc per Smoking Status\", fontsize=16)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**This is surprising because the category of smoker that has the highest FCV currently smoke.**\n* Lets look to see and see if there is any information we can gleam more insight into the smoking categories.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Smoking Status By Category\n ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"f, (ax1, ax2) = plt.subplots(1,2, figsize = (16, 6))\na = sns.barplot(x = train[\"SmokingStatus\"], y = train[\"Age\"],ax=ax1, palette=cb[0:4])\n\nb = sns.violinplot(x=train[\"SmokingStatus\"], y= train['Age'],ax=ax2, palette=cb[7:10])\n\na.set_title(\"Average Age for each Smoking Status\", fontsize=16)\nb.set_title(\"Distribution of Age For Each Smoking Status\", fontsize=16)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"While the average age of each smoking status very similar we can see that the distribution for each group is slighly different.\n* Current smokers is slghtly skewed more negativly vs the other two groups.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Lets look at the FVC value for each point of age and each SmokingStatus.  Is there any insights that can be made from the data?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data = train.groupby(['Age', 'SmokingStatus'])['FVC'].agg(['mean','std'])\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f, (ax1) = plt.subplots(1, figsize = (16, 6))\n\nax = sns.lineplot(x=\"Age\", y=\"FVC\", hue = 'SmokingStatus', data=train)\nax.set_title(\"Average FVC for Each Smoking Status\", fontsize=16)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see current smokers tend to have a higher output on their FVC tests.  \n\nEx-smokers seem to be the most stable of the groups by slowly losing FVC in the later years of life. \n\nNon-Smokers tend to have the lower output compared to the other groups.  I wonder if many of the non smokers have other underlying health issues that may have affected their lung capacity.  I don't know if this data contains both **Idiopathic** and **Non-Idiopathic** pulmonary fibrosis patients.\n\n**Potential Causes**\n\n**Causes of pulmonary fibrosis include environmental pollutants, some medicines, some connective tissue diseases, and interstitial lung disease. Interstitial lung disease is the name for a large group of diseases that inflame or scar the lungs. In most cases, the cause cannot be found. This is called idiopathic pulmonary fibrosis.**\nhttps://medlineplus.gov/pulmonaryfibrosis.html#:~:text=Causes%20of%20pulmonary%20fibrosis%20include,the%20cause%20cannot%20be%20found.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"raw","source":"","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}