{"cells":[{"metadata":{},"cell_type":"markdown","source":"# 1. Introduction\n\n### What is Pulmonary Fibrosis? \n\nPulmonary fibrosis is a lung disease that occurs when lung tissue becomes damaged and scarred. This thickened, stiff tissue makes it more difficult for your lungs to work properly. As pulmonary fibrosis worsens, you become progressively more short of breath.\n\nThe scarring associated with pulmonary fibrosis can be caused by a multitude of factors. But in most cases, doctors can't pinpoint what's causing the problem. When a cause can't be found, the condition is termed idiopathic pulmonary fibrosis.\n\nThe lung damage caused by pulmonary fibrosis can't be repaired, but medications and therapies can sometimes help ease symptoms and improve quality of life. For some people, a lung transplant might be appropriate.\n\n![Image from Mayoclinic](https://www.mayoclinic.org/-/media/kcms/gbs/patient-consumer/images/2016/08/10/14/57/mcdc7_pulmonaryfibrosis-8col.jpg)\n\n### What are the Symptons of Pulmonary Fibrosis?\n\nSigns and symptoms of pulmonary fibrosis may include:\n\n* Shortness of breath (dyspnea)\n* A dry cough\n* Fatigue\n* Unexplained weight loss\n* Aching muscles and joints\n* Widening and rounding of the tips of the fingers or toes (clubbing)\n\nThe course of pulmonary fibrosis — and the severity of symptoms — can vary considerably from person to person. Some people become ill very quickly with severe disease. Others have moderate symptoms that worsen more slowly, over months or years.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Install and Import Necessary Libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"! pip -q install chart_studio","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport pydicom\nfrom tqdm.notebook import tqdm\nimport glob\nimport random\nimport os\n\nimport matplotlib.animation as animation\nfrom matplotlib.widgets import Slider\nfrom IPython.display import HTML, Image\n\nimport plotly.express as px\nimport plotly.figure_factory as ff\nimport chart_studio.plotly as py\nimport plotly.graph_objs as go\nfrom plotly.offline import iplot\n\nfrom mpl_toolkits.mplot3d.art3d import Poly3DCollection\nimport scipy.ndimage\nfrom skimage import morphology\nfrom skimage import measure\n\nfrom colorama import Fore, Back, Style\n\nDATA_DIR = \"../input/osic-pulmonary-fibrosis-progression\"\nplt.style.use(\"fivethirtyeight\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. EDA\nLet's now explore the data.","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(os.path.join(DATA_DIR, \"train.csv\"))\ntest_df = pd.read_csv(os.path.join(DATA_DIR, \"test.csv\"))\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.1 Dry EDA\nLet's do some basic Data Statistics to get ourselves warmed up","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 2.1.1 Unique Patients","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(Fore.BLUE+f\"In Total there are {train_df['Patient'].count()} patients ids in train set and {test_df['Patient'].count()} patient ids in test set\"+Style.RESET_ALL)\nprint(Fore.GREEN+f\"Of which there are {train_df['Patient'].nunique()} unique patients in the train set\"+Style.RESET_ALL)\nprint(Fore.YELLOW+f\"And {test_df['Patient'].nunique()} unique patients in the test set\"+Style.RESET_ALL)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.1.2 Null Values","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(Fore.GREEN+f\"There are {train_df.any().isna().sum()} null values in the train set\"+Style.RESET_ALL)\nprint(Fore.YELLOW+f\"There are {test_df.any().isna().sum()} null values in the test set\"+Style.RESET_ALL)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.1.3 Intersecting Samples\n\nLet's check if any samples intersect between training and testing sets.\nAs it turns out, all the 5 samples in test set are present in training set. This may cause overfitting in future.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_set = set(train_df['Patient'])\ntest_set = set(test_df['Patient'])\ninter = train_set.intersection(test_set)\n\nprint(Fore.CYAN + f\"There are {len(inter)} Same Samples between both datasets\" + Style.RESET_ALL)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.1.4 DICOM Data Statistics","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"nb_train_imgs = glob.glob(os.path.join(DATA_DIR, \"train/**/*.dcm\"))\nnb_test_imgs = glob.glob(os.path.join(DATA_DIR, \"test/**/*.dcm\"))\n\nprint(Fore.BLUE + f\"There are {len(nb_train_imgs)+len(nb_test_imgs)} total DICOM files in this dataset.\"+Style.RESET_ALL)\nprint(Fore.GREEN+f\"In training set, there are: {len(nb_train_imgs)} DICOM files\"+Style.RESET_ALL)\nprint(Fore.YELLOW+f\"In testing set, there are {len(nb_test_imgs)} DICOM files\"+Style.RESET_ALL)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"avg_train_imgs = len(nb_train_imgs) // train_df['Patient'].count()\navg_test_imgs = len(nb_test_imgs) // test_df['Patient'].count()\n\nprint(Fore.GREEN+f\"In training set, an average patient has: {avg_train_imgs} images\"+Style.RESET_ALL)\nprint(Fore.YELLOW+f\"In testing set, an average patient has: {avg_test_imgs} images\"+Style.RESET_ALL)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.2 In-depth Analysis\n\nBefore we continue, let's make a new dataframe only with `Patient`, `Age`, `Sex` and `Smoking Status` that doesn't have duplicate values and shuffle it so we can easily create some visualizations from it.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"new_df = train_df[['Patient', 'Age', 'Sex', 'SmokingStatus']].drop_duplicates()\nnew_df = new_df.sample(frac=1).reset_index(drop=True)\nnew_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.2.1 Smoking Status Viz.\nLet's now visualize how many of all the patients are `Currently smoker`, `Ex-smoker` and `Non-Smoker` with the help of a pie chart. I have seen many excellent notebook and visualizations that use Bar Graphs for this purpose, but I (personally) think that Pie-charts are more intuitive and severely under-rated. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"vals = new_df['SmokingStatus'].value_counts().tolist()\nidx = ['Ex-Smoker', 'Never Smoked', 'Currently Smokes']\nfig = px.pie(\n    values=vals,\n    names=idx,\n    title='Smoking Status of Patients',\n    color_discrete_sequence=['cyan', 'blue', 'darkblue']\n)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see that the majority of the patients were ex-smokers followed by non-smoker and current smoker. The reason that ex-smokers make up such a majority can be attributed to the fact that smoking has long-lasting effects, even after you quit it. If I am wrong, please correct me!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 2.2.2 Gender Values Pie Chart\nLet's now use our friendly pie-chart to take a look at how many patients are Male and how many are Female?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"vals = new_df['Sex'].value_counts().tolist()\nidx = new_df['Sex'].value_counts().keys().tolist()\nfig = px.pie(\n    values=vals,\n    names=idx,\n    title='Gender Distribution of Patients',\n    color_discrete_sequence=['blue', 'magenta']\n)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.2.3 Unique Patients Age Distribution\nAlso take a look at Age Distribution of both genders.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.histogram(\n    new_df, x=\"Age\",\n    marginal=\"violin\",\n    hover_data=new_df.columns,\n    color='Sex',\n    color_discrete_sequence=['blue', 'magenta'],\n    title=f\"Unique Patients Age Distribution [\\u03BC : ~{int(new_df.mean())} years | \\u03C3 : ~{int(new_df.std())} years]\",\n)\n\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, most of the patients are between the ages `60-75`. Which confirms that the main targets of Pulmonary Fibrosis are old-aged citizens.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 2.2.4 Weeks Distribution\nSimilar to Age Distribution Above, now let's plot a histogram of total Weeks Distribution of both genders.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.histogram(\n    train_df, x=\"Weeks\",\n    marginal=\"box\",\n    hover_data=train_df.columns,\n    color='Sex',\n    title=f\"Weeks Distribution [\\u03BC : ~{int(train_df['Weeks'].mean())} weeks | \\u03C3 : ~{int(train_df['Weeks'].std())} weeks]\",\n)\n\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(Fore.BLUE + f\"Maximum Weeks for a Male Patient are: {train_df.loc[train_df['Sex']=='Male', 'Weeks'].max()}, average are: {int(train_df.loc[train_df['Sex']=='Male', 'Weeks'].mean())} and minimum are: {train_df.loc[train_df['Sex']=='Male', 'Weeks'].min()}\" + Style.RESET_ALL)\nprint(Fore.MAGENTA + f\"Maximum Weeks for a Female Patient are: {train_df.loc[train_df['Sex']=='Female', 'Weeks'].max()}, average are: {int(train_df.loc[train_df['Sex']=='Female', 'Weeks'].mean())} and minimum are: {train_df.loc[train_df['Sex']=='Female', 'Weeks'].min()}\" + Style.RESET_ALL)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.2.5 Age Distribution for Male and Female Patients","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsns.kdeplot(new_df.loc[new_df['Sex'] == 'Male', 'Age'], label = 'Male',shade=True)\nsns.kdeplot(new_df.loc[new_df['Sex'] == 'Female', 'Age'], label = 'Female',shade=True)\n\n# Labeling of plot\nplt.xlabel('Age')\nplt.ylabel('Density')\nplt.title('Distribution of Ages for Male and Female Patients')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.2.6 Age Distribution for different smoking status categories","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsns.kdeplot(new_df.loc[new_df['SmokingStatus'] == 'Currently smokes', 'Age'], label = 'Currently Smokes',shade=True)\nsns.kdeplot(new_df.loc[new_df['SmokingStatus'] == 'Never smoked', 'Age'], label = 'Never Smoked',shade=True)\nsns.kdeplot(new_df.loc[new_df['SmokingStatus'] == 'Ex-smoker', 'Age'], label = 'Ex-Smoker',shade=True)\n\nplt.xlabel('Age')\nplt.ylabel('Density')\nplt.title('Distribution of Ages vs SmokingStatus Category')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can note few things from the above plot:\n\n- Most Current Smokers lie in the age range of `~62-72`\n- Most People who never smoked lie in the age range of `~55-75` \n- Most Ex-smokers lie in the age range of `~65-80`\n\nIn simple words. **Non-smokers tend to be younger, Current Smokers tend to be between `60-70`(let's say Old) and Ex-smokers tend be in `70-80` (let's say oldest).**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 2.2.7 FVC Capacity\nThe Histogram and violin plot below shows the count of different ranges of Forces Vital Capacity (FVC)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.histogram(\n    train_df,\n    x='FVC',\n    marginal='violin',\n    hover_data=train_df.columns,\n    color_discrete_sequence=['maroon'],\n    title=f\"FVC Count Distribution [ \\u03BC : {int(train_df['FVC'].mean())} ml. | \\u03C3 : {int(train_df['FVC'].std())} ml. ]\"\n)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that, the average Forced Vital Capacity is 2690 mililiter with a standard deviation of 830 mililiter. We also have a few outliers such as the one lying on the far-right of plot with and FVC of ~6400 ml. That patient is at 0-weeks, is 71-Male and an Ex-Smoker. The reason for such a high FVC can be explained since they are at their 0th Week.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 2.2.8 Patient's Lung Capacity over weeks\nWe'll now take a look at the lung capacity of 3 randomly picked patients with different smoking habits.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"smoker = random.choice(train_df.query(\"SmokingStatus == 'Currently smokes'\")['Patient'].unique())\nnon_smoker = random.choice(train_df.query(\"SmokingStatus == 'Never smoked'\")['Patient'].unique())\nexsmoker = random.choice(train_df.query(\"SmokingStatus == 'Ex-smoker'\")['Patient'].unique())\n\nfig = go.Figure()\nfig.add_trace(go.Scatter(x=train_df[train_df.Patient==smoker]['Weeks'], y=train_df[train_df.Patient==smoker]['FVC'],\n                    mode='lines+markers',\n                    name='Current smoker'))\nfig.add_trace(go.Scatter(x=train_df[train_df.Patient==non_smoker]['Weeks'], y=train_df[train_df.Patient==non_smoker]['FVC'],\n                    mode='lines+markers',\n                    name='Non-smoker'))\nfig.add_trace(go.Scatter(x=train_df[train_df.Patient==exsmoker]['Weeks'], y=train_df[train_df.Patient==exsmoker]['FVC'],\n                    mode='lines+markers', name='Ex-smoker'))\n\nfig.update_layout(\n    title=\"Patient Lung Capacity over weeks\",\n    xaxis_title=\"Weeks\",\n    yaxis_title=\"Lung Capacity (in ml)\",\n    legend_title=\"Smoker Status\",\n)\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One thing to note here is that the Non-smoker has had a low lung capacity since the very beginning. I'm sure that I am wrong or missing something out as I am a beginner, but if you can point out what I am doing/guessing wrong, please commnet below, it'll be a learning experience for me 😀","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 2.3 DICOM Image Analysis 💻\n\nLet's now analyze DICOM images and see how they actually look like.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 2.3.1 Multiple DICOM Image Plot\nFirst, let's plot 20 DICOM images of a patient","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_dicom(patient_id=\"ID00019637202178323708467\", cmap='jet'):\n    image_dir = os.path.join(\"../input/osic-pulmonary-fibrosis-progression/train/\", patient_id)\n    fig = plt.figure(figsize=(12, 12))\n    cols = 4\n    row = 5\n    img_list = os.listdir(image_dir)\n    plt.title(f\"DICOM Images of Patient: {patient_id}\")\n    for i in range(1, row*cols+1):\n        filename = os.path.join(image_dir, str(i)+\".dcm\")\n        image = pydicom.dcmread(filename)\n        fig.add_subplot(row, cols, i)\n        plt.grid(False)\n        plt.imshow(image.pixel_array, cmap=cmap)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_dicom()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.3.2 Lung CT Scan Animation\nLet's make an animation out the of CT Scans to see them change in real-time (no not real time, shouldn't have said that 😂)\nLearnt how to make animations from [Pranav's Kernel](https://www.kaggle.com/pranavkasela/interactive-lung-ct-scan-with-animation)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%capture\n# Choose a patient id and then get all the dicom images of that patient\npatient_id = \"ID00012637202177665765362\"\ndicom_path = \"/kaggle/input/osic-pulmonary-fibrosis-progression/train\"\n\n# Just Sort all the dicom files based only on their first names and then put back all the sorted files with their .dcm extensions\nfiles = np.array([f.replace(\".dcm\",\"\") for f in os.listdir(f\"{dicom_path}/{patient_id}/\")])\nfiles = np.sort(files.astype(\"int\"))\ndicoms = [f\"{dicom_path}/{patient_id}/{f}.dcm\" for f in files]\n\n# Iterate through all the dicom images, read every image and reshape it\n# Then save the plt.imshow(img) in img_ and append it to an array \n# At the end, pass it to the ArtistAnimation function with 0.1s interval and 1s delay to make an animation out of them\nims = []\nfig = plt.figure()\nfor img in dicoms:\n    img_ = pydicom.dcmread(img).pixel_array.reshape(512, 512)\n    img_ = plt.imshow(img_, animated=True, cmap='gray')\n    plt.axis(\"off\")\n    ims.append([img_])\n\nani = animation.ArtistAnimation(fig, ims, interval=100, blit=False, repeat_delay=1000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now just show the animations as an HTML component.\nHTML(ani.to_jshtml())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Conclusion\nThis marks the end of this kernel. If you liked my work, please consider giving it an upvote and if I have missed something or maybe wrote something wrong, please make sure to correct me down below!\n\nI have learnt and implemented a lot of things from [Heroseo's Kernel](https://www.kaggle.com/piantic/osic-pulmonary-fibrosis-progression-basic-eda) and also did a little bit of experimentation of my own. \n\nThank you!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}