{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"### This is an interactive notebook. To use the interactive features, please fork it and then run again. \n
\n OSIC Pulmonary Fibrosis Progression \n
\n\n
\n Predict lung function decline \n
\n\n \n\n\n\n\n\nImagine one day, your breathing became consistently labored and shallow. Months later you were finally diagnosed with pulmonary fibrosis, a disorder with no known cause and no known cure, created by scarring of the lungs. If that happened to you, you would want to know your prognosis. That’s where a troubling disease becomes frightening for the patient: outcomes can range from long-term stability to rapid deterioration, but doctors aren’t easily able to tell where an individual may fall on that spectrum. Your help, and data science, may be able to aid in this prediction, which would dramatically help both patients and clinicians.\n\n\n\n[Open Source Imaging Consortium (OSIC)](https://www.osicild.org/) is a **not-for-profit, co-operative effort between academia, industry and philanthropy**. The group enables rapid advances in the fight against **Idiopathic Pulmonary Fibrosis (IPF)**, **fibrosing interstitial lung diseases (ILDs)**, and other respiratory diseases, including emphysematous conditions. Its mission is to bring together radiologists, clinicians and computational scientists from around the world to improve imaging-based treatments.\n\n \n\n\n**Common difficulties in treating lung diseases**\n* Even with access to a CT-scan it is difficult to identify lung disease.\n* Varied prognosis to different lung diseases create additional difficulties in clinical treatment. \n* Patients become anxious with fibrosis-related symptoms as it is often unclear what is causing problem in the first place. \n\n**In this competition, you’ll:**\n* Predict a patient’s severity of decline in lung function based on a CT scan of their lungs.\n* Determine lung function based on output from a spirometer, which measures the volume of air inhaled and exhaled;\n* Challenge to use machine learning techniques to make a prediction with the image, metadata, and baseline FVC as input.\n\n**Goals:**\n* Patients and their families would better understand their prognosis when they are first diagnosed with this incurable lung disease;\n* improved severity detection would positively impact treatment trial design and accelerate the clinical development of novel treatments.\n\nBefore diving into the competition, let's have a look at pulmonary fibrosis, its causes, symptoms and complications and other non cs stuffs. Though I am not a medical specialist, I collected bits and pieces from here and there and put together in this notebook. If something is not right, please comment so that all are benifitted from that knowledge. ","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"\n\n\n\n## Pulmonary fibrosis\nPulmonary fibrosis is a lung disease that occurs when lung tissue becomes damaged and scarred. This thickened, stiff tissue makes it more difficult for your lungs to work properly. As pulmonary fibrosis worsens, you become progressively more short of breath.\n\nThe scarring associated with pulmonary fibrosis can be caused by a multitude of factors. But in most cases, doctors can't pinpoint what's causing the problem. When a cause can't be found, the condition is termed idiopathic pulmonary fibrosis.\n\nThe lung damage caused by pulmonary fibrosis can't be repaired, but medications and therapies can sometimes help ease symptoms and improve quality of life. For some people, a lung transplant might be appropriate.\n\n\n \n\n\"Drawing\"\n\n\n## Symptoms of Pulmonary fibrosis\n\nSigns and symptoms of pulmonary fibrosis may include:\n* Shortness of breath (dyspnea)\n* A dry cough\n* Fatigue\n* Unexplained weight loss\n* Aching muscles and joints\n* Widening and rounding of the tips of the fingers or toes (clubbing)\n\nThe course of pulmonary fibrosis — and the severity of symptoms — can vary considerably from person to person. Some people become ill very quickly with severe disease. Others have moderate symptoms that worsen more slowly, over months or years.\n\nSome people may experience a rapid worsening of their symptoms (acute exacerbation), such as severe shortness of breath, that may last for several days to weeks. People who have acute exacerbations may be placed on a mechanical ventilator.\n[Source](https://www.mayoclinic.org/diseases-conditions/pulmonary-fibrosis/symptoms-causes/syc-20353690#:~:text=Pulmonary%20fibrosis%20is%20a%20lung,progressively%20more%20short%20of%20breath.)\n\n\n \n\n\"Drawing\"\n\n## Risk factors in pulmonary fibrosis \n\n* **Age:** Although pulmonary fibrosis has been diagnosed in children and infants, the disorder is much more likely to affect middle-aged and older adults.\n* **Sex:** Idiopathic pulmonary fibrosis is more likely to affect men than women.\n* **Smoking:** Far more smokers and former smokers develop pulmonary fibrosis than do people who have never smoked. Pulmonary fibrosis can occur in patients with emphysema.\n* **Certain occupations:** You have an increased risk of developing pulmonary fibrosis if you work in mining, farming or construction or if you're exposed to pollutants known to damage your lungs.\n* **Cancer treatments:** Having radiation treatments to your chest or using certain chemotherapy drugs can increase your risk of pulmonary fibrosis.\n* **Genetic factors:** Some types of pulmonary fibrosis run in families, and genetic factors may be a component.\n[source](https://www.lungsandyou.com/facts/ipf-risk-factors)\n\n\n \n\n## Complications\n\n* **High blood pressure in your lungs (pulmonary hypertension):** Unlike systemic high blood pressure, this condition affects only the arteries in your lungs. It begins when the smallest arteries and capillaries are compressed by scar tissue, causing increased resistance to blood flow in your lungs.This in turn raises pressure within the pulmonary arteries and the lower right heart chamber (right ventricle). Some forms of pulmonary hypertension are serious illnesses that become progressively worse and are sometimes fatal.\n\n* **Right-sided heart failure (cor pulmonale):** This serious condition occurs when your heart's lower right chamber (ventricle) has to pump harder than usual to move blood through partially blocked pulmonary arteries.\n* **Respiratory failure:** This is often the last stage of chronic lung disease. It occurs when blood oxygen levels fall dangerously low.\n* **Lung cancer:** Long-standing pulmonary fibrosis also increases your risk of developing lung cancer.\n* **Lung complications:** As pulmonary fibrosis progresses, it may lead to complications such as blood clots in the lungs, a collapsed lung or lung infections.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now Let's have a look at the data itself. ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from IPython.lib.display import YouTubeVideo\nYouTubeVideo('AfK9LPNj-Zo', width=800, height=600)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport json\nfrom pathlib import Path\nfrom glob import glob\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\ntrain_data_dir = '../input/osic-pulmonary-fibrosis-progression/train/'\ntest_data_dir = '../input/osic-pulmonary-fibrosis-progression/test/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## There are 176 patients in training set and in the test set, there are only 5 patients. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_ids = os.listdir(train_data_dir)\nprint('Training Patient NOs:', len(patient_ids))\n\npatient_ids_test = os.listdir(test_data_dir)\nprint('Test Patient NOs:', len(patient_ids_test))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Read DICOM Files\n### What is a DICOM file?\nA DICOM file is an image saved in the Digital Imaging and Communications in Medicine (DICOM) format. It contains an image from a medical scan, such as an ultrasound or MRI. DICOM files may also include identification data for patients so that the image is linked to a specific individual. The DICOM format was developed by the NEMA (National Electrical Manufacturers Association). It was designed for the exchanging and viewing of medical images, such as CT scans, MRIs, and ultrasound images.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_image_paths_dcom = glob(train_data_dir + '*/*.dcm')\nprint(f'Total train images {len(train_image_paths_dcom)}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pydicom\nfrom pydicom.data import get_testdata_files\nprint(__doc__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A ton of metadata is associated with each of the dicom files. However, in order to get each of the instances we just have to consider the attribute with a . sign. For example. if we want to access the rows and columns of the image. we can just write \n```Python\n# For getting the row and columns\nRefDs.Rows\nRefDs.Columns\n\n##For getting the actual pixel values\nRefDs.PixelData\n```","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_image_paths_dcom[0])\nRefDs = pydicom.dcmread(train_image_paths_dcom[0])\nprint(f'Image size: {RefDs.Rows}x{RefDs.Columns}' )\nRefDs","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Interactive Metadata Visualizations\nLet's look at the few metadata attached with one of the files. Just select any patient ID from the dropdown list and then from the slider, select the instance. If that instance is present, you can see the metatada of that patient. Feel free to change the range of instance in the calling function to see more diversified instances of any patient.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from ipywidgets import interact, interactive, IntSlider, ToggleButtons\ndef explore_patients_metadata(patient_id, instance):\n RefDs = pydicom.dcmread('../input/osic-pulmonary-fibrosis-progression/train/' + \n patient_id +'/' + \n str(instance) + '.dcm')\n pat_name = RefDs.PatientName\n display_name = pat_name.family_name + \", \" + pat_name.given_name\n print(\"Patient's name................:\", display_name)\n print(\"Patient id....................:\", RefDs.PatientID)\n print(\"Scan Instance.................:\", RefDs.InstanceNumber)\n print(\"Modality......................:\", RefDs.Modality)\n print(\"BodyPartExamined..............:\", RefDs.BodyPartExamined) \n print(\"Image Position (Patient)...:\", RefDs.ImagePositionPatient)\n print(\"Image Orientation (Patient)...:\", RefDs.ImageOrientationPatient)\n print(\"Pixel Spacing.................:\", RefDs.PixelSpacing)\n print('Window Center.................:', RefDs.WindowCenter)\n print('Window Width..................:', RefDs.WindowWidth)\n print('Window Intercept..............:', RefDs.RescaleIntercept)\n\n \ninteract(explore_patients_metadata, patient_id= patient_ids, instance = (1,150))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Observation\nBy selecting different values of patient IDs, it is evident that, different patients have different parameters set up in their following properties. They are\n* Image Position\n* Pixel Spacing\n\nIn my opinion, preprocessing the images based on the image position and pixel spacing to make them uniform in appearance can be a very good preprocessing step. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Interactive Plot Visualizations\nLet's have an interactive visualization of the instances given for each of the patients. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define a function to visualize the data\ndef explore_dicoms(patient_id, instance):\n RefDs = pydicom.dcmread('../input/osic-pulmonary-fibrosis-progression/train/' + \n patient_id +\n '/' + \n str(instance) + '.dcm')\n plt.figure(figsize=(10, 5))\n \n plt.imshow(RefDs.pixel_array, cmap='gray');\n plt.title(f'P_ID: {patient_id}\\nInstance: {instance}')\n plt.axis('off')\ninteract(explore_dicoms, patient_id= patient_ids, instance = (1,40))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Observations\nIt seems that different patients have different parameters setup in their image capturing device, specially image position and pixel spacing. Therefore these changes should be taken in consideration when preprocessing. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# I am not an specialist in Pulmonary Fibrosis. If there is any mistake in my info, please do comment so that I can correct them. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}