{"cells":[{"metadata":{"trusted":true},"cell_type":"markdown","source":"**HELLO KAGGLERS. THIS IS AN IN DEPTH AND THOROUGH VISUALIZATION OF OSIC PULMONARY FIBROSIS PROGRESSION DATASET. I HAVE COMPARED ALMOST EVERY FEATURE WITH ONE ANOTHER. HOPE YOU LIKE MY WORK. IF YOU WANNA ADD SOMETHING JUST TELL IN COMMENTS. IF YOU LIKE MY WORKK UPVOTE.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Pulmonary fibrosis is a lung disease that occurs when lung tissue becomes damaged and scarred. This thickened, stiff tissue makes it more difficult for your lungs to work properly.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Imagine one day, your breathing became consistently labored and shallow. Months later you were finally diagnosed with pulmonary fibrosis, a disorder with no known cause and no known cure, created by scarring of the lungs. If that happened to you, you would want to know your prognosis. That’s where a troubling disease becomes frightening for the patient: outcomes can range from long-term stability to rapid deterioration, but doctors aren’t easily able to tell where an individual may fall on that spectrum. Your help, and data science, may be able to aid in this prediction, which would dramatically help both patients and clinicians.\n\n\n\nCurrent methods make fibrotic lung diseases difficult to treat, even with access to a chest CT scan. In addition, the wide range of varied prognoses create issues organizing clinical trials. Finally, patients suffer extreme anxiety—in addition to fibrosis-related symptoms—from the disease’s opaque path of progression.\n\nOpen Source Imaging Consortium (OSIC) is a not-for-profit, co-operative effort between academia, industry and philanthropy. The group enables rapid advances in the fight against Idiopathic Pulmonary Fibrosis (IPF), fibrosing interstitial lung diseases (ILDs), and other respiratory diseases, including emphysematous conditions. Its mission is to bring together radiologists, clinicians and computational scientists from around the world to improve imaging-based treatments.\n\nIn this competition, you’ll predict a patient’s severity of decline in lung function based on a CT scan of their lungs. You’ll determine lung function based on output from a spirometer, which measures the volume of air inhaled and exhaled. The challenge is to use machine learning techniques to make a prediction with the image, metadata, and baseline FVC as input.\n\nIf successful, patients and their families would better understand their prognosis when they are first diagnosed with this incurable lung disease. Improved severity detection would also positively impact treatment trial design and accelerate the clinical development of novel treatments.\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np \nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/osic-pulmonary-fibrosis-progression/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# DATA ANALYSIS","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"LEST FIRST HAVE A LOOK THAT HOW MUCH NULL DATA WE HAVE.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Data Cleaning","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10)) \nsns.heatmap(train.isnull(), yticklabels=False, cbar=False, cmap='viridis')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"ABOVE VISUALIZATION CLEARLY SHOWS WE HAVE NO OF NULL DATA","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"len(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['SmokingStatus'].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(train['Patient'].unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**THIS MEANS ALTHOUGH THERE ABOUT 1500 EXAMPLES, BUT THERE ARE ONLY 176 UNIQUE PATIENTS**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# DATA VISUALIZATON","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Visualization Related to Smoking","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = ['Ex-smoker', 'Never smoked', 'Currently smokes']\nsizes = train['SmokingStatus'].value_counts()\ncolors = plt.cm.afmhot(np.linspace(0, 1, 5))\nexplode = [0.1, 0.1, 0.1,]\n\nplt.rcParams['figure.figsize'] = (9, 9)\nplt.pie(sizes, labels = labels, colors = colors,explode=explode, shadow = True)\nplt.title('Distribution of Smoking Status', fontsize = 20)\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"1. **WE CAN INFER THAT MAJOR PATIENTS ARE EX-SMOKERS AND LEAST PEOPLE SMOKE CURRENTLY ALSO**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.style.use('seaborn-white')\n\nsns.countplot(x='SmokingStatus',  data=train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10)) \n\nsns.countplot(x='SmokingStatus',data=train,hue='Sex')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,10)) \n\nsns.countplot(x='SmokingStatus',data=train,hue='Age')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* MAJORLY EX-SMOKERS ARE IN 70s RANGE OF AGE\n* MAJORLY WHO NEVER SMOKED ARE IN RANGE OF 60s\n* ALTHOUGH VERY LESS PATIENTS CURRENTLY SMOKE BUT MAJORY ARE YOUNG","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.kdeplot(train.loc[train['SmokingStatus'] == 'Ex-smoker', 'Age'], label = 'Ex-smoker',shade=True)\n\nsns.kdeplot(train.loc[train['SmokingStatus'] == 'Never smoked', 'Age'], label = 'Never smoked',shade=True)\n\n# Labeling of plot\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Smokers over Age');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.style.use('dark_background')\n\nplt.rcParams['figure.figsize'] = (15, 7)\nax = sns.violinplot(x = train['SmokingStatus'], y = train['Age'], palette = 'Reds')\nax.set_xlabel(xlabel = 'Smoking habit', fontsize = 15)\nax.set_ylabel(ylabel = 'Age', fontsize = 15)\nax.set_title(label = 'Distribution of Smokers over Age', fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"THIS JUSTIFIES OUR ABOVE INFERENCE","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.style.use('dark_background')\n\nplt.rcParams['figure.figsize'] = (15, 7)\nax = sns.violinplot(x = train['SmokingStatus'], y = train['Percent'], palette = 'Reds')\nax.set_xlabel(xlabel = 'Smoking Habit', fontsize = 15)\nax.set_ylabel(ylabel = 'Percent', fontsize = 15)\nax.set_title(label = 'Distribution of Smoking Status Over Percentage', fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualization Related to Age","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.style.use('dark_background')\ntrain['Age'].value_counts().head(80).plot.bar(color = 'orange', figsize = (20, 7))\nplt.title('Different Ages in Data', fontsize = 30, fontweight = 20)\nplt.xlabel('Different Age Groups')\nplt.ylabel('count')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"MAXIMUM PATIENTS ARE OF 66 AGE","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.style.use('seaborn-white')\n\nsns.scatterplot(x='Age',y='Percent',data=train, color='Red')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.scatterplot(x='Weeks',y='Age',data=train, color = 'Black')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**MAJORITY OF PATEINTS WHO HAVE THEIR AGE FROM 60-75 ARE HAVING CHECKUPS OF AROUND 0-60 NUMBER OF TIMES**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,7))\nsns.distplot(train['Age'], color='Black',  bins = 30 )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.style.use('dark_background')\n\nsns.kdeplot(train.loc[train['Sex'] == 'Male', 'Age'], label = 'Male',shade=True)\n\nsns.kdeplot(train.loc[train['Sex'] == 'Female', 'Age'], label = 'Female',shade=True)\n\n# Labeling of plot\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages over Gender');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.style.use('seaborn-white')\nplt.rcParams['figure.figsize'] = (15, 8)\nax = sns.boxplot(x = train['Sex'], y = train['Age'], palette = 'viridis')\nax.set_xlabel(xlabel = 'Sex', fontsize = 9)\nax.set_ylabel(ylabel = 'Age', fontsize = 9)\nax.set_title(label = 'Distribution of Ages as per Sex', fontsize = 20)\nplt.xticks(rotation = 90)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**AVERAGE AGE FOR FEMALE PATIENTS IS LESS THAN MALES**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lineplot(train['Age'], train['Percent'] , color = 'black')\nplt.title('Age vs Percent', fontsize = 20)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lineplot(train['Age'], train['FVC'] , color = 'black')\nplt.title('Age vs FVC', fontsize = 20)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lineplot(train['Age'], train['Weeks'] , color = 'black')\nplt.title('Age vs FVC', fontsize = 20)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,30))\na = plt.subplot(10, 1, 1)\nsns.pointplot(train.Age ,train.Percent)\nplt.title(\"Patinets Perecent over Age\" , fontsize = 25)\nplt.ylabel('Percent', fontsize = 15)\nplt.xlabel('Age', fontsize = 15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,30))\na = plt.subplot(10, 1, 1)\nsns.pointplot(train.Sex ,train.Age)\nplt.title(\"Patinets Age by Sex\" , fontsize = 25)\nplt.ylabel('Age', fontsize = 15)\nplt.xlabel('Sex', fontsize = 15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,30))\na = plt.subplot(8, 1, 1)\nsns.pointplot(train.Age ,train.FVC)\nplt.title(\"Patinets FVC over Age\" , fontsize = 25)\nplt.ylabel('FVC', fontsize = 15)\nplt.xlabel('Age', fontsize = 15)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Visualization Related to FVC","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.scatterplot(x='FVC',y='Percent',data=train, color='Black')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**FVC is highly co-related to Percentage. They both share Linear Relationship. if Percentage Increases, so does FVC.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.scatterplot(x='FVC',y='Age',data=train, color ='Red')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.scatterplot(x='FVC',y='Weeks',data=train, color='magenta')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Majority Patients who have FVC from 1500-3000 are having 0-40 weeks. **","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.corr()['FVC'].sort_values()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.style.use('seaborn-white')\nplt.figure(figsize=(10,7))\nsns.distplot(train['FVC'], color='Blue')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**MAXIMUM PATIENTS HAVE FVC AROUND 3000.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lineplot(train['Percent'], train['FVC'] , color = 'black')\nplt.title('Percent vs FVC', fontsize = 20)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Visualization Related to Weeks","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.style.use('seaborn-white')\ntrain['Weeks'].value_counts().head(80).plot.bar(color = 'red', figsize = (25, 7))\nplt.title('Number of Weeks in Data', fontsize = 50, fontweight = 20)\nplt.xlabel('Weeks')\nplt.ylabel('count')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.scatterplot(x='Weeks',y='Percent',data=train, color ='blue')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.scatterplot(x='Weeks',y='Age',data=train, color ='blue')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.scatterplot(x='Weeks',y='FVC',data=train, color ='blue')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,7))\nsns.distplot(train['Weeks'], color='Blue')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**MOST PATIENTS HAVE CHECKED UP BY 8-10 TIMES.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lineplot(train['Weeks'], train['Percent'] , color = 'black')\nplt.title('Percent vs Week', fontsize = 20)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Visualization Related to Sex","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(x='Sex',  data=train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**MAJORITY ARE MALES **","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,30))\na = plt.subplot(10, 1, 1)\nsns.pointplot(train.Sex ,train.Percent)\nplt.title(\"Patinets Perecent over Sex\" , fontsize = 25)\nplt.ylabel('Percent', fontsize = 15)\nplt.xlabel('Sex', fontsize = 15)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,30))\na = plt.subplot(10, 1, 1)\nsns.pointplot(train.Sex ,train.FVC)\nplt.title(\"Patinet's FVC over Sex\" , fontsize = 25)\nplt.ylabel('Percent', fontsize = 15)\nplt.xlabel('Sex', fontsize = 15)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,30))\na = plt.subplot(10, 1, 1)\nsns.pointplot(train.Sex ,train.Weeks)\nplt.title(\"Patinet's Weeks over Sex\" , fontsize = 25)\nplt.ylabel('Percent', fontsize = 15)\nplt.xlabel('Sex', fontsize = 15)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,7))\nsns.distplot(train['Percent'], color='Blue')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MAJOR POINTS OF INFERRENCE:\n\n\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# MAJOR POINTS OF INFERRENCE:\n\n* DATA WE HAVE, CLEARLY SHOWS WE HAVE NO OF NULL DATA.\n* THIS MEANS ALTHOUGH THERE ABOUT 1500 EXAMPLES, BUT THERE ARE ONLY 176 UNIQUE PATIENTS.\n* WE CAN INFER THAT MAJOR PATIENTS ARE EX-SMOKERS AND LEAST PEOPLE SMOKE CURRENTLY ALSO.\n* MAJORLY EX-SMOKERS ARE IN 70s RANGE OF AGE\n* MAJORLY WHO NEVER SMOKED ARE IN RANGE OF 60s\n* ALTHOUGH VERY LESS PATIENTS CURRENTLY SMOKE BUT MAJORY ARE YOUNG\n* MAXIMUM PATIENTS ARE OF 66 AGE.\n* MAJORITY OF PATEINTS WHO HAVE THEIR AGE FROM 60-75 ARE HAVING CHECKUPS OF AROUND 0-60 NUMBER OF TIMES.\n* AVERAGE AGE FOR FEMALE PATIENTS IS LESS THAN MALES\n* FVC is highly co-related to Percentage. They both share Linear Relationship. if Percentage Increases, so does FVC.\n* Majority Patients who have FVC from 1500-3000 are having 0-40 weeks. \n* MOST PATIENTS HAVE CHECKED UP BY 8-10 TIMES.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_path = '../input/osic-pulmonary-fibrosis-progression/train'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pydicom as dicom","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fs = dicom.dcmread(\"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/1.dcm\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(fs.pixel_array) \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}