{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Problem Statement\n\nIn this competition, you’ll predict a patient’s severity of decline in lung function based on a CT scan of their lungs. You’ll determine lung function based on output from a spirometer, which measures the volume of air inhaled and exhaled. The challenge is to use machine learning techniques to make a prediction with the image, metadata, and baseline FVC as input.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Let's first know a bit about Pulmonary Fibrosis","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import IFrame, YouTubeVideo\nYouTubeVideo('cRVRAKM5ono',width=600, height=400)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I hope it helped. If you like the kernel please upvote the kernel :)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Hope you already read the data description. If you haven't, its give below:\n\n## Data Description\n\nThe aim of this competition is to predict a patient’s severity of decline in lung function based on a CT scan of their lungs. Lung function is assessed based on output from a spirometer, which measures the forced vital capacity (FVC), i.e. the volume of air exhaled.\n\nIn the dataset, you are provided with a baseline chest CT scan and associated clinical information for a set of patients. A patient has an image acquired at time Week = 0 and has numerous follow up visits over the course of approximately 1-2 years, at which time their FVC is measured.\n\nIn the training set, you are provided with an anonymized, baseline CT scan and the entire history of FVC measurements.\nIn the test set, you are provided with a baseline CT scan and only the initial FVC measurement. You are asked to predict the final three FVC measurements for each patient, as well as a confidence value in your prediction.\n\nSince this is real medical data, you will notice the relative timing of FVC measurements varies widely. The timing of the initial measurement relative to the CT scan and the duration to the forecasted time points may be different for each patient. This is considered part of the challenge of the competition. To avoid potential leakage in the timing of follow up visits, you are asked to predict every patient's FVC measurement for every possible week. Those weeks which are not in the final three visits are ignored in scoring.\n\n## Files\nThis is a synchronous rerun code competition. The provided test set is a small representative set of files (copied from the training set) to demonstrate the format of the private test set. When you submit your notebook, Kaggle will rerun your code on the test set, which contains unseen images.\n\n* train.csv - the training set, contains full history of clinical information\n* test.csv - the test set, contains only the baseline measurement\n* train/ - contains the training patients' baseline CT scan in DICOM format\n* test/ - contains the test patients' baseline CT scan in DICOM format\n* sample_submission.csv - demonstrates the submission format\n\n## Columns\ntrain.csv and test.csv\n* Patient- a unique Id for each patient (also the name of the patient's DICOM folder)\n* Weeks- the relative number of weeks pre/post the baseline CT (may be negative)\n* FVC - the recorded lung capacity in ml\n* Percent- a computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics\n* Age\n* Sex\n* SmokingStatus","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Importing Libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install fastai2 -q","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n#Load the dependancies\nfrom fastai2.basics import *\nfrom fastai2.callback.all import *\nfrom fastai2.vision.all import *\nfrom fastai2.medical.imaging import *\n\nimport pydicom","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/train.csv\")\ndf_test = pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.shape,df_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test.nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_weeks = df_train.groupby(\"Patient\").agg({\"Weeks\":\"nunique\",\"Age\":\"nunique\"}).reset_index()\nfig, (ax1,ax2) = plt.subplots(1,2,figsize=(10, 5))\nsns.countplot(df_weeks.Weeks,ax = ax1);\nsns.countplot(df_weeks.Age,ax =ax2);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What we see:\n* More than 120 out of 176 Patients have 9 weeks and about 30 Patients have 8 weeks of clinical information/recordings \n* All of them have 1 age throught their weeks of clinical information/recordings","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_patients = df_train[[\"Patient\",\"Sex\",\"SmokingStatus\",\"Age\"]].drop_duplicates()\nfig, (ax1,ax2,ax3) = plt.subplots(1,3,figsize=(20, 5),gridspec_kw={'width_ratios': [1,1,2]})\nsns.countplot(df_patients.Sex,ax = ax1);\nsns.countplot(df_patients.SmokingStatus,ax =ax2);\nsns.countplot(df_patients.Age,ax =ax3);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What we see:\n*  ~140 out of 176 Patients are Male. Less than 40 Patients(22%) are female \n*  ~120(68%) out of 176 Patients were Ex-Smokers. ~ 50 Patients never smoked\n* Age shows a normal distriution where 64-74 ages Patients show the highest peaK","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Let's look at the FVC of one Patient over the weeks**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lineplot(x = \"Weeks\", y = \"FVC\", data = df_train[df_train.Patient==\"ID00007637202177411956430\"]);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, to understand DICOMs which is the format in which the CT Scans are shared, I went through this kernel : https://www.kaggle.com/avirdee/understanding-dicoms\n\nDICOM(Digital Imaging and COmmunications in Medicine) is the de-facto standard that establishes rules that allow medical images(X-Ray, MRI, CT) and associated information to be exchanged between imaging equipment from different vendors, computers, and hospitals.\n\nDICOM files typically have a .dcm extension and provides a means of storing data in separate 'tags' such as patient information as well as image/pixel data. A DICOM file consists of a header and image data sets packed into a single file. \n\nTo access the files I will be using fastai2.medical.imaging module. Under the hood fastai uses pydicom to access the dicom files.\n\nPydicom is a python package for parsing DICOM files and makes it easy to covert DICOM files into pythonic structures for easier manipulation. Files are opened using pydicom.dcmread","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_DATA = \"../input/osic-pulmonary-fibrosis-progression/train\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_files = get_dicom_files(TRAIN_DATA)\ntrain_files","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are 33,026 DICOM files.Let's look at one of the file","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"info_view = train_files[0]\ndimg = dcmread(info_view)\ndimg","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dimg.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}