{"cells":[{"metadata":{},"cell_type":"markdown","source":"Welcome to the notebook where in I explore data of the \"OSIC Pulmonary Fibrosis Progression\" competition. I will try to be as elaborate as possible.  \n\nLet's get started! \n\nThe first step towards approaching any problem is to understand the problem statement properly. \n\n# Problem statement \n\n**In this competition, you’ll predict a patient’s severity of decline in lung function based on a CT scan of their lungs. You’ll determine lung function based on output from a spirometer, which measures the volume of air inhaled and exhaled. The challenge is to use machine learning techniques to make a prediction with the image, metadata, and baseline FVC as input.**\n\n# Dig deeper into Pulmonary Fibrosis\n\n![](https://www.pulmonaryfibrosis.org/images/default-source/default-album/normal-and-impaired-gas-exchange.png?sfvrsn=c3b0918d_0)\n\nThe word **“Pulmonary”** means lung and the word **“fibrosis”** means scar tissue— similar to scars that you may have on your skin from an old injury or surgery. So, in its simplest sense, **pulmonary fibrosis (PF) means scarring in the lungs**. \n\nOver time, the scar tissue can destroy the normal lung and make it hard for oxygen to get into your blood. Low oxygen levels (and the stiff scar tissue itself) can cause you to feel short of breath, particularly when walking and exercising. **Pulmonary fibrosis** isn’t just one disease. It is a family of more than 200 different lung diseases that all look very much alike. The PF family of lung diseases falls into an even larger group of diseases called the interstitial lung diseases (also known as ILD), which includes all of the diseases that have inflammation and/or scarring in the lung. Some interstitial lung diseases don’t include scar tissue. When an interstitial lung disease does include scar tissue in the lung, we call it pulmonary fibrosis.\n\n## Symptoms \n\nSigns and symptoms of pulmonary fibrosis may include:\n\n1. Shortness of breath (dyspnea)\n2. A dry cough\n3. Fatigue\n4. Unexplained weight loss\n5. Aching muscles and joints\n6. Widening and rounding of the tips of the fingers or toes (clubbing)\n\nThe course of pulmonary fibrosis — and the severity of symptoms — can vary considerably from person to person. Some people become ill very quickly with severe disease. Others have moderate symptoms that worsen more slowly, over months or years.\n\n## Causes \n\nPulmonary fibrosis scars and thickens the tissue around and between the air sacs (alveoli) in your lungs. This makes it more difficult for oxygen to pass into your bloodstream. The damage can be caused by many different factors — including long-term exposure to certain toxins, certain medical conditions, radiation therapy and some medications.\n\n## Risk factors\n\n**Age**. Although pulmonary fibrosis has been diagnosed in children and infants, the disorder is much more likely to affect middle-aged and older adults.\n\n**Sex**. Idiopathic pulmonary fibrosis is more likely to affect men than women.\n\n**Smoking**. Far more smokers and former smokers develop pulmonary fibrosis than do people who have never smoked. Pulmonary fibrosis can occur in patients with emphysema.\n\n**Certain occupations**. You have an increased risk of developing pulmonary fibrosis if you work in mining, farming or construction or if you're exposed to pollutants known to damage your lungs.\n\n**Cancer treatments**. Having radiation treatments to your chest or using certain chemotherapy drugs can increase your risk of pulmonary fibrosis.\n\n**Genetic factors**. Some types of pulmonary fibrosis run in families, and genetic factors may be a component.\n\n## Why is this competition important?\n\nCurrent methods make fibrotic lung diseases difficult to treat, even with access to a chest CT scan. In addition, the wide range of varied prognoses create issues organizing clinical trials. Finally, patients suffer extreme anxiety—in addition to fibrosis-related symptoms—from the disease’s opaque path of progression.\n\nIf successful, patients and their families would better understand their prognosis when they are first diagnosed with this incurable lung disease. Improved severity detection would also positively impact treatment trial design and accelerate the clinical development of novel treatments.\n\n# Data description \n\nWe have seen the details of the disease and what competition expects us to do, let's see the description of the data and what we need to do about it. \n\nAs we understand from the problem statement of the competition, our job is to make prognosis easier for the patients. We are provided with a baseline chest CT scan and associated clinical information for a set of patients. A patient has an image acquired at time Week = 0 and has numerous follow up visits over the course of approximately 1-2 years, at which time their FVC is measured. \n\nLung function is assessed based on output from a spirometer, which measures the forced vital capacity (FVC), i.e. the volume of air exhaled.\n\n**Train:** We are provided with an anonymized, baseline CT scan and the entire history of FVC measurements.\n\n**Test:** We are provided with a baseline CT scan and only the initial FVC measurement. You are asked to predict the final three FVC measurements for each patient, as well as a confidence value in your prediction.\n\nLet's get started!","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n!pip install chart_studio\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n# import plotly.plotly as py\nimport chart_studio.plotly as py\nimport plotly.graph_objs as go\nfrom plotly.offline import iplot, init_notebook_mode\n\nimport cufflinks\ncufflinks.go_offline(connected=True)\ninit_notebook_mode(connected=True)\n\n\nsns.set()\nplt.rcParams[\"figure.figsize\"] = [8,12]\n\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"df_train=pd.read_csv(r\"/kaggle/input/osic-pulmonary-fibrosis-progression/train.csv\")\ndf_test=pd.read_csv(\"/kaggle/input/osic-pulmonary-fibrosis-progression/test.csv\")\ndf_train.head(170)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* So, there are 7 columns in our training dataset:\n\n1. **Patient id:** Will be useful for us to follow the week-by-week status update of an individual. As mentioned in the problem statement, patients have been followed on a week-by-week basis, so there could be a lot of repeatitive values according to the weekly update. We will also attempt to find unique ids in the column so that we know how many actual patients are we dealing with, in the data.\n\n2. **Weeks:** This denotes the week's number.\n\n3. **FVC:** This represents the total amount of air exhaled during the FEV test. Forced expiratory volume and forced vital capacity are lung function tests that are measured during spirometry.\n\n4. **Percent:** It is the computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics\n\n5. **Age** \n\n6. **Sex**\n\n7. **Smoking status:** This could be an important parameter in determining whether smoking status is relevant to the FVC or not. ","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"df_test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us have a look at the shape our training and testing dataset: ","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"print(\"The shape of our training dataset is:{}\".format(df_train.shape))\nprint(\"The shape of our testing dataset is:{}\".format(df_test.shape))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now it is time to see if our dataset has any missing value, we will check for both training and testing dataset:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"df_train.info()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"df_test.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Both the train and test dataset contain no missing values. Let us look at other statistical parameters of our training dataset, which is our quite important for us. ","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"df_train.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us how many unique values we have in our dataset:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"df_train['Patient'].nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, we have 176 unique patients ids. Rest of them are essentially the week-by-week record of these patients. Now we can analyze the data in two ways:\n\n1. We can use all the data\n2. We can use the data with uniqe ids.\n\nLet us first make a new dataframe with unique ids and also make a columns expressing frequency of each patient","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"df_unique = df_train.groupby([df_train.Patient,df_train.Age,df_train.Sex, df_train.SmokingStatus])['Patient'].count()\ndf_unique.index = df_unique.index.set_names(['Patient Id','Age','Sex','SmokingStatus'])\n\ndf_unique = df_unique.reset_index()\ndf_unique.rename(columns = {'Patient': 'Frequency'},inplace = True)\n\ndf_unique.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us find out the average appointment a patient takes to get his FVC measured:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_unique['Frequency'].mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that on an average, patients had around 9 appointments to get their FVC tested. Some had fewer than that, but most of them had close to 9 appointments. Let us explore other features in the dataset. \n\n## Data distribution by Sex (uniques values)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"print(df_unique['Sex'].value_counts())\npercent_sexwise=df_unique['Sex'].value_counts()/len(df_unique['Sex'])\nprint(percent_sexwise)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20,8))\nsns.countplot(ax=ax, x=\"Sex\", data=df_unique, palette=\"bone\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that more number of men suffer from pulmonary fibrosis than women. This could be because they are more exposed to environmental hazards because they work more in outdoor jobs. Women also tend to develop cardiovascular diseases around 7-12 years later than men. Cardiovascular diseases are also a potential factor behind development of pulmonary fibrosis. \n\n## Data distribution by Age (uniques values)","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"from matplotlib import colors\n\nfig, ax = plt.subplots(figsize=(20,8))\nN_points = 100000\nn_bins = 15\nN, bins, patches = plt.hist(df_unique['Age'], n_bins, alpha=0.5)\n\nfracs = N / N.max()\nnorm = colors.Normalize(fracs.min(), fracs.max())\nfor thisfrac, thispatch in zip(fracs, patches):\n    color = plt.cm.viridis(norm(thisfrac))\n    thispatch.set_facecolor(color)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most of the patients develop the symptoms of pulmonary fibrosis between 65 - 75 years of age. People start developing symptoms around 50 years of age and continue till 80 years of age. People below that age and above that age have lesser risk of developing symptoms. \n\n## Data distribution by Age according to Sex (uniques values)","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20,8))\ndf_train.groupby(['Age', 'Sex']).size().unstack().plot(kind='bar', stacked=True, ax=ax)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data distribution by Smoking Status (uniques values)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20,8))\nsns.countplot(x=\"SmokingStatus\", data=df_unique, palette=\"magma\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, most of the people have had smoking history before they developed symptoms. They could have dropped smoking after the symptoms perhaps, but it is very much evident that people with a smoking history have greater chances of contracting pulmonary fibriosis. One more thing could be the bias in smoking status with a bias towards the ex-smokers. \n\nWe will try to find that out!\n\n## Data distribution by Smoking Status according to Sex (uniques values)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20,8))\ndf_train.groupby(['SmokingStatus', 'Sex']).size().unstack().plot(kind='bar', stacked=True, ax=ax)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data distribution by FVC\n\n### Before moving further, let us understand more about FVC:\n\n![](http://)![](https://slideplayer.com/slide/10537676/36/images/8/Pulmonary+Fibrosis+Restrictive+Diseases+Pulmonary+Function+Tests.jpg)\n\n\n**Forced vital capacity (FVC)** is the amount of air that can be forcibly exhaled from your lungs after taking the deepest breath possible, as measured by spirometry. This test may help distinguish obstructive lung diseases, such as asthma and COPD, from restrictive lung diseases, such as pulmonary fibrosis and sarcoidosis.\n\nFVC can also help doctors assess the progression of lung disease and evaluate the effectiveness of treatment. An abnormal FVC value may be chronic, but sometimes the problem is reversible and the FVC can be corrected.\n\n### Interpreting Results\n\nYour total FVC volume can be compared with the standard FVC for your age, sex, height, and weight. Your FVC can also be compared with your own previous FVC values, if applicable, to determine whether your pulmonary condition is progressing or if your lung function is improving under treatment. **FVC also may be expressed as a percentage of the predicted FVC.** This is expressed as percent in our dataset. \n\n\nForced vital capacity will be reported in two ways:\n\n1. As an absolute value, reported as a number in liters (L)\n2. On a linear graph to chart the dynamics of your exhalation\n\n\nThe normal FVC range for an adult is between 3.0 and 5.0 L. This depends on a lot of factors like genetics, height, body weight and other factors. It has nowhere been explicitly mentioned, but I am assuming that FVC is given in ml. ","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20,8))\n\nN_points = 100000\nn_bins = 15\nN, bins, patches = plt.hist(df_train['FVC'], n_bins, alpha=0.5)\n\nfracs = N / N.max()\nnorm = colors.Normalize(fracs.min(), fracs.max())\nfor thisfrac, thispatch in zip(fracs, patches):\n    color = plt.cm.inferno(norm(thisfrac))\n    thispatch.set_facecolor(color)\nprint(df_train['FVC'].min())\nprint(df_train['FVC'].max())\nprint(df_train['FVC'].mean())\nprint(df_train['FVC'].median())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since this is real medical data, you will notice the relative timing of FVC measurements varies widely. The timing of the initial measurement relative to the CT scan and the duration to the forecasted time points may be different for each patient. This is considered part of the challenge of the competition.\n\nLet us undestand more on how our data is distributed: ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy.stats import kurtosis, skew\n\n\nprint(\"The mean of FVC data is: {}\".format(df_train['FVC'].mean()))\nprint(\"The median of FVC data is: {}\".format(df_train['FVC'].median()))\nprint(\"The standard deviation of FVC data is: {}\".format(df_train['FVC'].std()))\n\nprint( 'excess kurtosis of normal distribution (should be 0): {}'.format( kurtosis(df_train['FVC']) ))\nprint( 'skewness of normal distribution (should be 0): {}'.format( skew(df_train['FVC']) ))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We notice from the distribution that very few patients having very high FVC. The data is not normally distributed. We have a positive skew; Positive Skewness means when the tail on the right side of the distribution is longer or fatter. The mean and median will be greater than the mode. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20,15))\nsns.boxplot(x=\"Sex\", y=\"FVC\",hue=\"SmokingStatus\", data=df_train, palette=\"spring\", ax=ax)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Citing this [research paper](https://www.jstage.jst.go.jp/article/jpts/24/1/24_1_5/_pdf#:~:text=It%20was%20found%20that%20the,lung%20capacity%20in%20females%20than), women have 20-25% lower capacity than men, owing to their smaller lungs. \n\nHowever, I have one concern about the data shown above. As seen from the boxplot, men and women who currently smoke have higher FVC than ex-smokers and non-smokers which is contradictory to this [research paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3944281/#:~:text=They%20showed%20that%20smoking%20decreased,airway%20obstruction%20and%20small%20airway) which states that:\n\n**Some previous studies have demonstrated the effect of smoking on the pulmonary function of adults8,9,10). They showed that smoking decreased pulmonary function including forced vital capacity (FVC), forced expiratory volume in one second (FEV1), FEV1/FVC, and forced expiratory flow at 25–75% (FEF25–75%)9). Cigarette smoking causes deficits in both FEV1/FVC and FEF25–75 which indicate airway obstruction and small airway disease in adult smokers.**\n\nThese works suggest otherwise, which should be a cause of concern. I will read more and try to find the caauses, etc behind this. \n\n## Data distribution by Weeks \n\nThis will give us an information on weekly visits by patients:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20,8))\ndf_weeks=df_train.groupby(['Weeks']).count()\ndf_weeks.head(20)\nsns.distplot(df_weeks, color='g')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The distribution shows that most of the people have had their measurements done in 0-10 weeks. This indicates two things:\n\n1. Either patients are reluctant to visit hospitals to get their FVC mesured.\n2. Or, the health of patients deteriorated over the course of weeks and gradually death would have occured.\n\n## FVC vs percent \n\n\nLet us see what is the relationship between FVC and Percent:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20,15))\n\nx=df_train['FVC']\ny=df_train['Percent']\n\ncolors = df_train['FVC']  # 0 to 15 point radii\n\nplt.scatter(x, y, c=colors, alpha=0.5)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It was obvious and it is evident that FVC and Percent share linear relationship so in our analysis further, we need not use both of these variables. \n\n## Data distribution of few individuals\n\nWe will see data distribution of few individual patients to see how the pattern of FVC over the weeks. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_patient=df_train.groupby(['Patient'])\ndf_patient.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Patient 1 with id ID00010637202177584971671","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df = df_train[df_train['Patient'] == \"ID00010637202177584971671\"]\nprint(patient_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import plotly.express as px\n\nfig = px.line(patient_df, x=\"Weeks\", y=\"FVC\", title='Patient FVC over the weeks')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For a person with Pulmonary fibrosis, it is expected that the capacity of lungs will deteriorate with time. Will analyse more in the next versions.\n\n## Patient 2 with id ID00426637202313170790466","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df = df_train[df_train['Patient'] == \"ID00426637202313170790466\"]\nprint(patient_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import plotly.express as px\n\nfig = px.line(patient_df, x=\"Weeks\", y=\"FVC\", title='Patient FVC over the weeks')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The deterioration of FVC in patients who have never smoked have less steep fall in FVC as compared to the patient seen above who was an ex-smoker. Let us look at one patient who currently smokes: \n\n## Patient 3 with id ID00082637202201836229724","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_df = df_train[df_train['Patient'] == \"ID00082637202201836229724\"]\nprint(patient_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import plotly.express as px\n\nfig = px.line(patient_df, x=\"Weeks\", y=\"FVC\", title='Patient FVC over the weeks')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The detrioration is quite steep in this case. \n\nWe have seen the tabular dataset till now, it is time to explore the DICOM dataset. I tried to be very elaborate in tabular data, I will also try the same in the DICOM data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# DICOM \n\n**DICOM® — Digital Imaging and Communications in Medicine** — is the international standard for medical images and related information. It defines the formats for medical images that can be exchanged with the data and quality necessary for clinical use.\n\nDICOM® is implemented in almost every radiology, cardiology imaging, and radiotherapy device (X-ray, CT, MRI, ultrasound, etc.), and increasingly in devices in other medical domains such as ophthalmology and dentistry. With hundreds of thousands of medical imaging devices in use, DICOM® is one of the most widely deployed healthcare messaging Standards in the world.\n\n### What is the data related to DICOM?\n\nA DICOM data object consists of a number of attributes, including items such as name, ID, etc., and also one special attribute containing the image pixel data. A single DICOM object can have only one attribute containing pixel data. For many modalities, this corresponds to a single image. However, the attribute may contain multiple \"frames\", allowing storage of cine loops or other multi-frame data.\n\nLet us start by importing all the dependencies:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# common packages \nimport numpy as np \nimport os\nimport copy\nfrom math import *\nimport matplotlib.pyplot as plt\nfrom functools import reduce\n# reading in dicom files\nimport pydicom as dicom\nimport glob\n# skimage image processing packages\nfrom skimage import measure, morphology\nfrom skimage.morphology import ball, binary_closing\nfrom skimage.measure import label, regionprops\n# scipy linear algebra functions \nfrom scipy.linalg import norm\nimport scipy.ndimage\n# ipywidgets for some interactive plots\nfrom ipywidgets.widgets import * \nimport ipywidgets as widgets\n# plotly 3D interactive graphs \nimport plotly\nfrom plotly.graph_objs import *\nimport chart_studio.plotly as py\n# set plotly credentials here \n# this allows you to send results to your account plotly.tools.set_credentials_file(username=your_username, api_key=your_key)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"apply_resample = False","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_scan(path):\n    slices = [dicom.read_file(path + '/' + s) for s in os.listdir(path)]\n    slices.sort(key = lambda x: float(x.ImagePositionPatient[2]))\n    try:\n        slice_thickness = np.abs(slices[0].ImagePositionPatient[2] - slices[1].ImagePositionPatient[2])\n    except:\n        slice_thickness = np.abs(slices[0].SliceLocation - slices[1].SliceLocation)\n        \n    for s in slices:\n        s.SliceThickness = slice_thickness\n        \n    return slices","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_pixels_hu(slices):\n    image = np.stack([s.pixel_array for s in slices])\n    # Convert to int16 (from sometimes int16), \n    # should be possible as values should always be low enough (<32k)\n    image = image.astype(np.int16)\n\n    # Set outside-of-scan pixels to 0\n    # The intercept is usually -1024, so air is approximately 0\n    image[image == -2000] = 0\n    \n    # Convert to Hounsfield units (HU)\n    for slice_number in range(len(slices)):\n        \n        intercept = slices[slice_number].RescaleIntercept\n        slope = slices[slice_number].RescaleSlope\n        \n        if slope != 1:\n            image[slice_number] = slope * image[slice_number].astype(np.float64)\n            image[slice_number] = image[slice_number].astype(np.int16)\n            \n        image[slice_number] += np.int16(intercept)\n    \n    return np.array(image, dtype=np.int16)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def set_lungwin(img, hu=[-1200., 600.]):\n    lungwin = np.array(hu)\n    newimg = (img-lungwin[0]) / (lungwin[1]-lungwin[0])\n    newimg[newimg < 0] = 0\n    newimg[newimg > 1] = 1\n    newimg = (newimg * 255).astype('uint8')\n    return newimg","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scans = load_scan('/kaggle/input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/')\nscan_array = set_lungwin(get_pixels_hu(scans))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy.ndimage.interpolation import zoom\n\ndef resample(imgs, spacing, new_spacing):\n    new_shape = np.round(imgs.shape * spacing / new_spacing)\n    true_spacing = spacing * imgs.shape / new_shape\n    resize_factor = new_shape / imgs.shape\n    imgs = zoom(imgs, resize_factor, mode='nearest')\n    return imgs, true_spacing, new_shape\n\nspacing_z = (scans[-1].ImagePositionPatient[2] - scans[0].ImagePositionPatient[2]) / len(scans)\n\nif apply_resample:\n    scan_array_resample = resample(scan_array, np.array(np.array([spacing_z, *scans[0].PixelSpacing])), np.array([1.,1.,1.]))[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import imageio\nfrom IPython.display import Image\n\nimageio.mimsave(\"/tmp/gif.gif\", scan_array, duration=0.0001)\nImage(filename=\"/tmp/gif.gif\", format='png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Please upvote if you liked the content in the notebook and please drop in your comments so that I can improve further.\n\nThanks for reading!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}