{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<h1><center>OSIC Pulmonary Fibrosis Progression</center></h1>","metadata":{}},{"cell_type":"markdown","source":"### 1.Introduction\nPulmonary fibrosis or scarring in the lungs is a family of more than 200 different lung diseases that are very similar. Over time, the scar tissue blocks the movement of oxygen from inside the tiny air sacs in the lungs into the bloodstream. There are five main categories of identifiable causes of pulmonary fibrosis: \n* drug-induced\n* radiation-induced\n* environmental\n* autoimmune\n* occupational\n\nWith all this, it can still be very challenging for doctors to determine the exact cause of a pulmonary fibrosis case. A pulmonary fibrosis case of unknown cause is called \"idiopathic\". With no known cure, current methods make fibrotic lung diseases difficult to treat even with access to a chest CT scan. On top of the disease having no cure, the prognosis of outcomes range from long-term stability to rapid deterioration, with doctors having no accurate way to determine where exactly a patient falls on that spectrum.<br>\n### 2.Problem statement\nWith the use of neural networks this notebook aims to predict a patient's severity of decline in lung function based on a CT scan of their lungs, and several other data which we will describe later. Lung function is being measured with the help of a spirometer, which measures the volume of air inhaled and exhaled. This value is called a FVC, which stands for Forced Vital Capacity.<br>\n### 3.Data\n* train.csv - the training set, contains full history of clinical information\n* test.csv - the test set, contains only the baseline measurement\n* train/ - contains the training patients' baseline CT scan in DICOM format\n* test/ - contains the test patients' baseline CT scan in DICOM format\n* sample_submission.csv - demonstrates the submission format","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport plotly.express as px\nimport plotly.graph_objs as go\n\nimport pydicom\nimport glob\nimport imageio\nfrom IPython.display import Image","metadata":{"execution":{"iopub.status.busy":"2022-01-20T01:02:19.650863Z","iopub.execute_input":"2022-01-20T01:02:19.651478Z","iopub.status.idle":"2022-01-20T01:02:21.337650Z","shell.execute_reply.started":"2022-01-20T01:02:19.651381Z","shell.execute_reply":"2022-01-20T01:02:21.336610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/train.csv\")\ntest_df = pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/test.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-01-20T01:02:21.339283Z","iopub.execute_input":"2022-01-20T01:02:21.339608Z","iopub.status.idle":"2022-01-20T01:02:21.374994Z","shell.execute_reply.started":"2022-01-20T01:02:21.339574Z","shell.execute_reply":"2022-01-20T01:02:21.374312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-20T01:02:21.377058Z","iopub.execute_input":"2022-01-20T01:02:21.377706Z","iopub.status.idle":"2022-01-20T01:02:21.400386Z","shell.execute_reply.started":"2022-01-20T01:02:21.377656Z","shell.execute_reply":"2022-01-20T01:02:21.399564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-20T01:02:21.402259Z","iopub.execute_input":"2022-01-20T01:02:21.402482Z","iopub.status.idle":"2022-01-20T01:02:21.415318Z","shell.execute_reply.started":"2022-01-20T01:02:21.402455Z","shell.execute_reply":"2022-01-20T01:02:21.414302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"|Column name|Meaning|\n|:------------:|:-----------:|\n|Patient|a unique Id for each patient (also the name of the patient's DICOM folder)|\n|Weeks|the relative number of weeks pre/post the baseline CT (may be negative)|\n|FVC|the recorded lung capacity in ml|\n|Percent|a computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics|\n|Age|age of patient|\n|Sex|sex of patient|\n|SmokingStatus|smoking status of patient|","metadata":{}},{"cell_type":"code","source":"## Exploring the data\nprint('Shape of Training data: ', train_df.shape)\nprint('Shape of Test data: ', test_df.shape)","metadata":{"execution":{"iopub.status.busy":"2022-01-20T01:02:21.416793Z","iopub.execute_input":"2022-01-20T01:02:21.417636Z","iopub.status.idle":"2022-01-20T01:02:21.428171Z","shell.execute_reply.started":"2022-01-20T01:02:21.417582Z","shell.execute_reply":"2022-01-20T01:02:21.427182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-01-20T01:02:21.429306Z","iopub.execute_input":"2022-01-20T01:02:21.429549Z","iopub.status.idle":"2022-01-20T01:02:21.461498Z","shell.execute_reply.started":"2022-01-20T01:02:21.429513Z","shell.execute_reply":"2022-01-20T01:02:21.460645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Number of unique ids are {train_df['Patient'].value_counts().shape[0]} \")","metadata":{"execution":{"iopub.status.busy":"2022-01-20T01:02:21.462649Z","iopub.execute_input":"2022-01-20T01:02:21.462869Z","iopub.status.idle":"2022-01-20T01:02:21.470022Z","shell.execute_reply.started":"2022-01-20T01:02:21.462842Z","shell.execute_reply":"2022-01-20T01:02:21.469156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since the number of unique ids is smaller than the number of entries(1549), it means that there are several entries per patient. The FVC value, as well as the percent value was probably measured more often.","metadata":{}},{"cell_type":"code","source":"new_df = train_df.groupby([train_df.Patient,train_df.Age,train_df.Sex, train_df.SmokingStatus])['Patient'].count()\nnew_df.index = new_df.index.set_names(['id','Age','Sex','SmokingStatus'])\nnew_df = new_df.reset_index()\nnew_df.rename(columns = {'Patient': 'freq'},inplace = True)\nnew_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-20T01:02:21.471379Z","iopub.execute_input":"2022-01-20T01:02:21.471929Z","iopub.status.idle":"2022-01-20T01:02:21.494540Z","shell.execute_reply.started":"2022-01-20T01:02:21.471880Z","shell.execute_reply":"2022-01-20T01:02:21.493666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.bar(new_df, x='id',y ='freq',color='freq')\nfig.update_layout(xaxis={'categoryorder':'total ascending'},title='No. of observations for each patient')\nfig.update_xaxes(showticklabels=False)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-20T01:02:21.496004Z","iopub.execute_input":"2022-01-20T01:02:21.496722Z","iopub.status.idle":"2022-01-20T01:02:22.736010Z","shell.execute_reply.started":"2022-01-20T01:02:21.496671Z","shell.execute_reply":"2022-01-20T01:02:22.734840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(new_df, x='Age',nbins = 42)\nfig.update_traces(marker_color='rgb(158,202,225)', marker_line_color='rgb(8,48,107)', marker_line_width=1.5, opacity=0.6)\nfig.update_layout(title = 'Distribution of Age for unique patients')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-20T01:02:22.738941Z","iopub.execute_input":"2022-01-20T01:02:22.739248Z","iopub.status.idle":"2022-01-20T01:02:22.838870Z","shell.execute_reply.started":"2022-01-20T01:02:22.739214Z","shell.execute_reply":"2022-01-20T01:02:22.837861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(new_df, x='Sex')\nfig.update_traces(marker_color='rgb(202,158,225)', marker_line_color='rgb(48,8,107)',\n                 marker_line_width=2, opacity=0.8)\nfig.update_layout(title = 'Distribution of Sex for unique patients')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-20T01:02:22.840267Z","iopub.execute_input":"2022-01-20T01:02:22.840481Z","iopub.status.idle":"2022-01-20T01:02:22.909605Z","shell.execute_reply.started":"2022-01-20T01:02:22.840455Z","shell.execute_reply":"2022-01-20T01:02:22.909047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(new_df, x='SmokingStatus')\nfig.update_traces(marker_color='rgb(202,225,158)', marker_line_color='rgb(48,107,8)',\n                 marker_line_width=2, opacity=0.8)\nfig.update_layout(title = 'Distribution of SmokingStatus for unique patients')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-20T01:02:22.910404Z","iopub.execute_input":"2022-01-20T01:02:22.910612Z","iopub.status.idle":"2022-01-20T01:02:22.978729Z","shell.execute_reply.started":"2022-01-20T01:02:22.910585Z","shell.execute_reply":"2022-01-20T01:02:22.978144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(new_df, x='SmokingStatus',color = 'Sex')\nfig.update_traces(marker_line_color='black',marker_line_width=2, opacity=0.85)\nfig.update_layout(title = 'Distribution of SmokingStatus for unique patients')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-20T01:02:22.979643Z","iopub.execute_input":"2022-01-20T01:02:22.980385Z","iopub.status.idle":"2022-01-20T01:02:23.064003Z","shell.execute_reply.started":"2022-01-20T01:02:22.980348Z","shell.execute_reply":"2022-01-20T01:02:23.063220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3.DICOM\nFor every patient there is a folder named with their unique ids. These folders contain images from a medical scan, like a CT scan and information about the patient. It is important to understand that each folder has a number of images, but they are all from the same scan, carried out on the same day. CT scans are pictures that slice from the beginning of the torso up to the neck. FVC measurements were carried out before, after and sometimes on the day of the CT scan. These images are called DICOM files. DICOM stands for \"Digital Imaging and Communications in Medicine\" and has two parts: the header and the dataset. The header contains information on the encapsulated dataset. It consists of a File Preamble, a DICOM prefix, and the File Meta Elements.","metadata":{}},{"cell_type":"code","source":"img = \"../input/osic-pulmonary-fibrosis-progression/train/ID00009637202177434476278/100.dcm\"\nds = pydicom.dcmread(img)\nplt.figure(figsize = (10,10))\nplt.imshow(ds.pixel_array, cmap=plt.cm.bone)","metadata":{"execution":{"iopub.status.busy":"2022-01-20T01:02:23.065293Z","iopub.execute_input":"2022-01-20T01:02:23.066115Z","iopub.status.idle":"2022-01-20T01:02:23.694093Z","shell.execute_reply.started":"2022-01-20T01:02:23.066061Z","shell.execute_reply":"2022-01-20T01:02:23.693051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}}]}