{"cells":[{"metadata":{},"cell_type":"markdown","source":"<img src='https://www.pulmonaryfibrosis.org/images/default-source/default-album/normal-and-impaired-gas-exchange.png?sfvrsn=c3b0918d_0'>\n<h1><center>OSIC Pulmonary Fibrosis Progression - EDA</center><h1>\n    \n# 1. Let's Understand Whats is this disease? ▶\n    \n###  1.1 What is Pulmonary fibrosis?\n* [The word “pulmonary” means lung and the word “fibrosis” means scar tissue— similar to scars](https://www.pulmonaryfibrosis.org/life-with-pf/about-pf) that you may have on your skin from an old injury or surgery. So, in its simplest sense, pulmonary fibrosis (PF) means scarring in the lungs. Over time, the scar tissue can destroy the normal lung and make it hard for oxygen to get into your blood. Low oxygen levels (and the stiff scar tissue itself) can cause you to feel short of breath, particularly when walking and exercising.\n    \n###  1.2 What has to be done in this competition?\nThe aim of this competition is to predict a patient’s severity of decline in lung function based on a CT scan of their lungs. Lung function is assessed based on output from a spirometer, which measures the forced vital capacity (FVC), i.e. the volume of air exhaled.\n\nIn the dataset, you are provided with a baseline chest CT scan and associated clinical information for a set of patients. A patient has an image acquired at time Week = 0 and has numerous follow up visits over the course of approximately 1-2 years, at which time their FVC is measured.\n\nIn the training set, you are provided with an anonymized, baseline CT scan and the entire history of FVC measurements.\nIn the test set, you are provided with a baseline CT scan and only the initial FVC measurement. You are asked to predict the final three FVC measurements for each patient, as well as a confidence value in your prediction.\n    \n- Files\n\nThis is a synchronous rerun code competition. The provided test set is a small representative set of files (copied from the training set) to demonstrate the format of the private test set. When you submit your notebook, Kaggle will rerun your code on the test set, which contains unseen images.\n\ntrain.csv - the training set, contains full history of clinical information\ntest.csv - the test set, contains only the baseline measurement\ntrain/ - contains the training patients' baseline CT scan in DICOM format\ntest/ - contains the test patients' baseline CT scan in DICOM format\nsample_submission.csv - demonstrates the submission format\n    \n- The leaderboard of this competition is calculated with approximately 1% of the test data. The final results will be based on the other 99%, so the final standings may be different.\n    \n- EXTERNAL DATA: Publicly and freely available external data is permitted if it is available for use that includes research or academic purposes","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 1. Import libraries ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nfrom os import listdir\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n#plotly\n!pip install chart_studio\nimport plotly.express as px\nimport chart_studio.plotly as py\nimport plotly.graph_objs as go\nfrom plotly.offline import iplot\nimport cufflinks\ncufflinks.go_offline()\ncufflinks.set_config_file(world_readable=True, theme='pearl')\n\nimport seaborn as sns\nsns.set(style=\"whitegrid\")\n\n\n#pydicom\nimport pydicom\n\n# Beautiful plot scheme\nplt.style.use('fivethirtyeight')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Reading training Meta data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# List files available\nlist(os.listdir(\"../input/osic-pulmonary-fibrosis-progression\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_PATH = \"../input/osic-pulmonary-fibrosis-progressiont/\"\n\ntrain_data = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\ntest_data = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\n\nprint('Training data shape: ', train_data.shape)\ntrain_data.head(5)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_data['SmokingStatus'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.groupby(['SmokingStatus','Sex']).count()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Feast for the eyes: Visualization","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Lets Explore the  data\nprint('Train mets data Set !!')\nprint(train_data.info())\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Test meta dataSet !!')\nprint(test_data.info())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Total number of ecords in the Meta dataset(train+test)\nprint(\"Total Patient in Train data set: \",train_data['Patient'].count())\nprint(\"Total Patient in Test  data set: \",test_data['Patient'].count())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n**In the dataset, you are provided with a baseline chest CT scan and associated clinical information for a set of patients. A patient has an image acquired at time Week = 0 and has numerous follow up visits over the course of approximately 1-2 years, at which time their FVC is measured.**\n\n## Lets Check the Unique Patients in Train and Test data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"The total patient ids are : \") \nprint(train_data['Patient'].count())\nprint(\"Total Uniquw unique patients are :\") \nprint(train_data['Patient'].value_counts().shape[0] )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"columns = train_data.keys()\ncolumns = list(columns)\nprint(columns)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## People Smoke?  Hmm.. Lets Check it out!!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['SmokingStatus'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"iplot(train_data.SmokingStatus.iplot(asFigure=True, kind='histogram', title='Smoking Distribution Data', dimensions=(1000,400)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Weeks distribution : The relative number of weeks pre/post the baseline CT (may be negative)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['Weeks'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Number of patients count pre/post the baseline CT: week Wise","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"grpdata=train_data.groupby(['Weeks']).count()[\"Patient\"]\ngrpdata","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.options.plotting.backend = \"plotly\"\ntrain_data.groupby(['Weeks']).count()[\"Patient\"].plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ngrpval=train_data.groupby(['FVC']).count()[\"Patient\"]\ngrpval","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Weeks vs SmokingStatus","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"z=train_data.groupby(['SmokingStatus','Weeks'])['FVC'].count().to_frame().reset_index()\nz.style.background_gradient(cmap='Reds') ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## FVC VS Patients : How many patients are at same stage of disease \n**Lung function is assessed based on output from a spirometer, which measures the forced vital capacity (FVC), i.e. the volume of air exhaled.******","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.groupby(['FVC']).count()[\"Patient\"].plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Does Gender has any relationship with FVC?\n### FVC VS Percentage VS SEX","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.groupby(['FVC']).count()[\"Patient\"].plot()\n\nimport plotly.express as px\nfig = px.line(train_data, x='FVC', y='Percent', color='Sex')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Gender has any relationship with FVC and Smoking habits?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsns.kdeplot(train_data.loc[train_data['SmokingStatus'] == 'Ex-smoker', 'FVC'], label = 'Ex-smoker',shade=True)\nsns.kdeplot(train_data.loc[train_data['SmokingStatus'] == 'Never smoked', 'FVC'], label = 'Never smoked',shade=True)\nsns.kdeplot(train_data.loc[train_data['SmokingStatus'] == 'Currently smokes', 'FVC'], label = 'Currently smokes',shade=True)\n\n# Labeling of plot\nplt.xlabel('FVC'); plt.ylabel('Density'); plt.title('Distribution of Gender');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## How many patients at different level of disease?\n### Percent","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"A computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['Percent'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['Percent'].iplot(kind='hist',bins=35,color='green',xTitle='Percent distribution',yTitle='No Of Patients')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## FVC distribution at different Percent for different Genders ?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsns.kdeplot(train_data.loc[train_data['Sex'] == 'Male', 'Percent'], label = 'Male',shade=True)\nsns.kdeplot(train_data.loc[train_data['Sex'] == 'Female', 'Percent'], label = 'Female',shade=True)\n\n# Labeling of plot\nplt.xlabel('Perecent of FVC '); plt.ylabel('Density'); plt.title('Distribution of Gender for FVC');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Aged patients are more prone to Pulmonary Fibrosis Progression?\n### Lets Check","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['Age'].iplot(kind='hist',bins=30,color='red',xTitle='Age distribution',yTitle='Count')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Smoking has any relationship with Age?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['SmokingStatus'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsns.kdeplot(train_data.loc[train_data['SmokingStatus'] == 'Ex-smoker', 'Age'], label = 'Ex-smoker',shade=True)\nsns.kdeplot(train_data.loc[train_data['SmokingStatus'] == 'Never smoked', 'Age'], label = 'Never smoked',shade=True)\nsns.kdeplot(train_data.loc[train_data['SmokingStatus'] == 'Currently smokes', 'Age'], label = 'Currently smokes',shade=True)\n\n# Labeling of plot\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Distribution of Age vs gender\n## Is it really Helpful: Check out yourself","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsns.kdeplot(train_data.loc[train_data['Sex'] == 'Female', 'Age'], label = 'Female',shade=True)\nsns.kdeplot(train_data.loc[train_data['Sex'] == 'Male', 'Age'], label = 'Male',shade=True)\n\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Gender distribution in train data\n### Males are more affected from Pulmonary Fibrosis Progression??","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['Sex'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"iplot(train_data.Sex.iplot(asFigure=True, kind='histogram', title='Sex Distribution Data', dimensions=(1000,400)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Males have bad habit of smoking?? \n### Gender vs SmokingStatus","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\na = sns.countplot(data=train_data, x='SmokingStatus', hue='Sex')\n\nfor p in a.patches:\n    a.annotate(format(p.get_height(), ','), \n           (p.get_x() + p.get_width() / 2., \n            p.get_height()), ha = 'center', va = 'center', \n           xytext = (0, 4), textcoords = 'offset points')\n\nplt.title('Gender split by SmokingStatus', fontsize=16)\nsns.despine(left=True, bottom=True);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## How are the images?\n###  Visualising DECOM images\n> A DICOM file is an image saved in the Digital Imaging and Communications in Medicine (DICOM) format. It contains an image from a medical scan, such as an ultrasound or MRI. DICOM files may also include identification data for patients so that the image is linked to a specific individual.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Train .dcm number of images:', len(list(os.listdir('../input/osic-pulmonary-fibrosis-progression/train'))), '\\n' +\n      'Test .dcm number of images:', len(list(os.listdir('../input/osic-pulmonary-fibrosis-progression/test'))), '\\n' +\n      '--------------------------------', '\\n' +\n      'There is the same number of images as in train/ test .csv datasets')\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Only 176 Images why??\n**Because there are only 176 unique patients in database**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":" ## Let's see what is in image","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"filename = \"/kaggle/input/osic-pulmonary-fibrosis-progression/train/ID00123637202217151272140/137.dcm\"\nds = pydicom.dcmread(filename)\nplt.imshow(ds.pixel_array, cmap=plt.cm.bone) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# directory for a patient\nimdir = \"/kaggle/input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430\"\nprint(\"total images for patient ID00007637202177411956430: \", len(os.listdir(imdir)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"images for patient ID00007637202177411956430 :\")\nmylist = os.listdir(imdir)\nmylist.sort()\nprint(mylist)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# view first (columns*rows) images in order\nw=10\nh=10\nfig=plt.figure(figsize=(12, 12))\ncolumns = 4\nrows = 5\nimglist = os.listdir(imdir)\nfor i in range(1, columns*rows +1):\n    filename = imdir + \"/\" + str(i) + \".dcm\"\n    ds = pydicom.dcmread(filename)\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(ds.pixel_array, cmap=plt.cm.bone)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## pydicom Hidden Information that could be useful\nhttps://pydicom.github.io/pydicom/stable/old/getting_started.html. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import glob\ntrain_image_path = '../input/osic-pulmonary-fibrosis-progression/train'\ntrain_image_files = glob.glob(os.path.join(train_image_path, '*', '*.dcm'))\n\ntrain_image_data = pydicom.read_file(train_image_files[0])\ntrain_image_data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ideas taken from https://www.kaggle.com/piantic/osic-pulmonary-fibrosis-progression-basic-eda","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}