{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\n**Open Source Imaging Consortium (OSIC) is a not-for-profit, co-operative effort between academia, industry and philanthropy. The group enables rapid advances in the fight against Idiopathic Pulmonary Fibrosis (IPF), fibrosing interstitial lung diseases (ILDs), and other respiratory diseases, including emphysematous conditions. Its mission is to bring together radiologists, clinicians and computational scientists from around the world to improve imaging-based treatments**\n\n**What is Pulmonary Fibrosis about?**\n\n**The word “pulmonary” means lung and the word “fibrosis” means scar tissue— similar to scars that you may have on your skin from an old injury or surgery. So, in its simplest sense, pulmonary fibrosis (PF) means scarring in the lungs.**\n\n**It is serious as over time, the scar tissue can destroy the normal lung and make it hard for oxygen to get into your blood. Low oxygen levels (and the stiff scar tissue itself) can cause you to feel short of breath, particularly when walking and exercising. Pulmonary fibrosis isn’t just one disease. It is a family of more than 200 different lung diseases that all look very much alike.**\n\n![](https://www.pulmonaryfibrosis.org/images/default-source/default-album/normal-and-impaired-gas-exchange.png?sfvrsn=c3b0918d_0)\n\n**For more info do visit https://www.pulmonaryfibrosis.org/life-with-pf/about-pf** \n\n# About the Competition\n\n![](https://www.kaggleusercontent.com/kf/40536606/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..4yf7NuyM78BCiuvU95LjCg.FRvwtJwLwfTAN4Jn01XO8g-cmK54Onr0h2vX9YV0fagrVPhJijWpRldRJCaXY3tKYfEQeFmukZ8Gftf0RVcn43oHReHZYQpvXEy0aKgsIvo_mMF4fJ0bK6KsaZPGIYu_mqv_dYs_jJi37dvJkqMI3DkZZu_MRHx48628X_tG9OGpmzCv9cscIoFF97r3FCOyNPcMOme8akhA5NXCpislOR4ELOqHSouIb0W5OccmjOF8Ur3abchozJMkwTuIYHzAZSlEs6ZnZsNjZpAaJuoMAR0MotpgnlT4ErK5RgMA21Ta1ZSYkJEzGoS3q6LHigr_zwtru06L_WAppP_0mUb20mKQ29KhlzbhZdd8Au8pucyQT9tBvigybd3V-nuB1fMjborZEiPUSzI57oGsq6gObtscx5VozG2iNdfJCYn2DFjpZ65Fkyq_WF16GhrKJW7a-8PTxVuzhjXs2B9_n8pm7-GxLNNVYQXUJjuITeMHwj0lGShv--3LLGhNWWAuLBILqflft7q7aA3GChMu_C-RfAj7LuFSswQhWD29RKnPsWmZJVS-Dkuwg4StrwTpn6McJ7LWlQjnbJ2SqMOu1NyI3lVIZ9TwZ07aIb-FxSV7timds2ImfSRSEuyoym-LCxVYVadB6eI-XD-5g7pZoJB-ULLW9oRKb7LXfkmu7HjQD68.uByHuDxikFpTmF-2vlZTBg/__results___files/__results___23_0.png)\n\n**In this competition, you’ll predict a patient’s severity of decline in lung function based on a CT scan of their lungs. You’ll determine lung function based on output from a spirometer, which measures the volume of air inhaled and exhaled. The challenge is to use machine learning techniques to make a prediction with the image, metadata, and baseline FVC as input.**\n\n**If successful, patients and their families would better understand their prognosis when they are first diagnosed with this incurable lung disease. Improved severity detection would also positively impact treatment trial design and accelerate the clinical development of novel treatments.**\n\n\n**Files**\n\n**This is a synchronous rerun code competition. The provided test set is a small representative set of files (copied from the training set) to demonstrate the format of the private test set. When you submit your notebook, Kaggle will rerun your code on the test set, which contains unseen images.**\n\n> **train.csv - the training set, contains full history of clinical information**\n\n> **test.csv - the test set, contains only the baseline measurement**\n\n> **train/ - contains the training patients' baseline CT scan in DICOM format**\n\n> **test/ - contains the test patients' baseline CT scan in DICOM format**\n\n> **sample_submission.csv - demonstrates the submission format**\n\n\n**Columns**\n\n\n> **Patient- a unique Id for each patient (also the name of the patient's DICOM folder)**\n\n> **Weeks- the relative number of weeks pre/post the baseline CT (may be negative)**\n\n> **FVC - the recorded lung capacity in ml**\n\n> **Percent- a computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics**\n\n>**Age**\n\n>**Sex**\n\n> **SmokingStatus**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**I will be using certain library called ONNX to convert pytorch based codes to tensorflow!!**","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nimport copy\nfrom datetime import timedelta, datetime\nimport imageio\nimport matplotlib.pyplot as plt\nfrom matplotlib import cm\nimport multiprocessing\nimport numpy as np\nimport os\nfrom pathlib import Path\nimport pydicom\nimport pytest\nimport scipy.ndimage as ndimage\nfrom scipy.ndimage.interpolation import zoom\nfrom skimage import measure, morphology, segmentation\nfrom skimage.transform import resize\nfrom time import time, sleep\nfrom tqdm import trange, tqdm\nimport tensorflow as tf\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.layers import *\nfrom tensorflow.data import Dataset\nimport torch\nfrom torch.utils.tensorboard import SummaryWriter\nfrom torchvision import transforms\nimport warnings\nimport seaborn as sns\nimport glob as glob\nimport imageio\nfrom IPython.display import Image\n\n#for masking\nfrom skimage.measure import label,regionprops\nfrom sklearn.cluster import KMeans\nfrom skimage.segmentation import clear_border\n\nimport onnx\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading Libraries","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\nprint('{} Rows and {} Columns in train data '.format(train_df.shape[0], train_df.shape[1]))\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\nprint('{} Rows and {} Columns in test data '.format(test_df.shape[0], test_df.shape[1]))\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.heatmap(train_df.corr(), annot=True, cmap=plt.cm.plasma)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**FVC and Percent Column have good correlation among themselves**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10))\nsns.countplot(x='Age', hue='SmokingStatus', data=train_df).set_title('Age wise smokers distrbution')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Let's check for Uniqueness in the training data**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('The Number of Unique Patients in training data are : {}'.format(len(train_df['Patient'].unique()), \"\\n\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Is there any Decline in Lung Function?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#https://www.kaggle.com/carlossouza/probabilistic-machine-learning-a-diff-approach/\n\ndef chart_data(patient_id, ax):\n    plot_data = train_df[train_df['Patient']==patient_id]\n    fig1 = plot_data['Weeks']\n    fig2 = plot_data['FVC']\n    ax.set_title(patient_id)\n    ax = sns.regplot(fig1, fig2, ax=ax, ci=None, line_kws={'color':'red'})\n    \n\nf, axes = plt.subplots(1, 3, figsize=(15, 5))\nchart_data('ID00007637202177411956430', axes[0])\nchart_data('ID00009637202177434476278', axes[1])\nchart_data('ID00426637202313170790466', axes[2]) #non-smoker plot","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**There is a clear evidence now as suggested from the plot that there is a decline in lung function of patients**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Here I will be referring to some of the great implementations from [Andrada's notebook](https://www.kaggle.com/andradaolteanu/pulmonary-fibrosis-competition-eda-dicom-prep)**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Select unique bio info for the patients\ndata = train_df.groupby(by=\"Patient\")[[\"Patient\", \"Age\", \"Sex\", \"SmokingStatus\"]].first().reset_index(drop=True)\n\n# Figure\nf, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize = (16, 6))\n\na = sns.distplot(data[\"Age\"], ax=ax1, hist=False, kde_kws=dict(lw=6, ls=\"--\"))\nb = sns.countplot(data[\"Sex\"], ax=ax2)\nc = sns.countplot(data[\"SmokingStatus\"], ax=ax3)\n\na.set_title(\"Patient Age Distribution\", fontsize=16)\nb.set_title(\"Sex Frequency\", fontsize=16)\nc.set_title(\"Smoking Status\", fontsize=16);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Let's visualize FVC and Percent columns as these had high correlation with each other**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Figure\nf, (ax1, ax2) = plt.subplots(1, 2, figsize = (16, 6))\n\na = sns.distplot(train_df[\"FVC\"], ax=ax1, hist=False, kde_kws=dict(lw=6, ls=\"--\"))\nb = sns.distplot(train_df[\"Percent\"], ax=ax2, hist=False, kde_kws=dict(lw=6, ls=\"-.\"))\n\na.set_title(\"FVC Distribution\", fontsize=16)\nb.set_title(\"Percent Distribution\", fontsize=16);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# DICOM Images Loading and Preprocessing","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Let's start with creating some helper functions to load the DICOM images from the dataset**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_path = '../input/osic-pulmonary-fibrosis-progression/train/'\n\noutput_path = '../input/output/'\ntrain_image_files = sorted(glob.glob(os.path.join(data_path, '*','*.dcm')))\npatients = os.listdir(data_path)\npatients.sort()\n\nprint('Some sample Patient ID''s :', len(train_image_files))\nprint(\"\\n\".join(train_image_files[:5]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_scan(path):\n    \"\"\"\n    Loads scans from a folder and into a list.\n    \n    Parameters: path (Folder path)\n    \n    Returns: slices (List of slices)\n    \"\"\"\n    \n    slices = [pydicom.read_file(path + '/' + s) for s in os.listdir(path)]\n    slices.sort(key = lambda x: int(x.InstanceNumber))\n    \n    try:\n        slice_thickness = np.abs(slices[0].ImagePositionPatient[2] - slices[1].ImagePositionPatient[2])\n    except:\n        slice_thickness = np.abs(slices[0].SliceLocation - slices[1].SliceLocation)\n        \n    for s in slices:\n        s.SliceThickness = slice_thickness\n        \n    return slices","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Credits to [Aadhav Vignesh](https://www.kaggle.com/aadhavvignesh/lung-segmentation-by-marker-controlled-watershed) for an awesome implementation**\n\n**Hounsfield Units\nThe unit of measurement in CT scans is the Hounsfield Unit (HU), which is a measure of radiodensity.**\n\n**Hounsfield units (HU) are a dimensionless unit universally used in computed tomography (CT) scanning to express CT numbers in a standardized and convenient form. Hounsfield units are obtained from a linear transformation of the measured attenuation coefficients.**\n\n![HU Table](http://patentimages.storage.googleapis.com/WO2005055806A2/imgf000011_0001.png)\n\n***HUs can be calculated from the pixel data with a DICOM Image using the following formula:***\n\n> ***HU = m ∗ P + b***\n\n***where,***\n\n> ***m  = RescaleSlope attribute of the DICOM image,***\n\n> ***b  = RescaleIntercept attribute of the DICOM image,***\n\n> ***P  = Pixel Array***","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**So Let's start with applying Houndsfield metrics to our images!!**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_pixels_hu(scans):\n    \"\"\"\n    Converts raw images to Hounsfield Units (HU).\n    \n    Parameters: scans (Raw images)\n    \n    Returns: image (NumPy array)\n    \"\"\"\n    \n    image = np.stack([s.pixel_array for s in scans])\n    image = image.astype(np.int16)\n\n    # Since the scanning equipment is cylindrical in nature and image output is square,\n    # we set the out-of-scan pixels to 0\n    image[image == -2000] = 0\n    \n    \n    # HU = m*P + b\n    intercept = scans[0].RescaleIntercept\n    slope = scans[0].RescaleSlope\n    \n    if slope != 1:\n        image = slope * image.astype(np.float64)\n        image = image.astype(np.int16)\n        \n    image += np.int16(intercept)\n    \n    return np.array(image, dtype=np.int16)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![](https://media.tenor.com/images/b4c2f5c658c1d3ade7e506ee7ffe3c5e/tenor.gif)\n\n\n**Now that we have created a helper function for converting it to HU units let's proceed ahead with Preprocessing**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_patient_scans = load_scan(data_path + patients[2])\ntest_patient_images = get_pixels_hu(test_patient_scans)\n\n#We'll be taking a random slice to perform segmentation:\n\nfor imgs in range(len(test_patient_images[0:5])):\n    f, (ax1, ax2, ax3) = plt.subplots(1, 3, sharey=True, figsize=(15,15))\n    ax1.imshow(test_patient_images[imgs], cmap=plt.cm.bone)\n    ax1.set_title(\"Original Slice\")\n    \n    ax2.imshow(test_patient_images[imgs], cmap=plt.cm.bone)\n    ax2.set_title(\"Original Slice\")\n    \n    ax3.imshow(test_patient_images[imgs], cmap=plt.cm.bone)\n    ax3.set_title(\"Original Slice\")\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Animated Scans","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def set_lungwin(img, hu=[-1200., 600.]):\n    lungwin = np.array(hu)\n    newimg = (img-lungwin[0]) / (lungwin[1]-lungwin[0])\n    newimg[newimg < 0] = 0\n    newimg[newimg > 1] = 1\n    newimg = (newimg * 255).astype('uint8')\n    return newimg\n\n\nscans = load_scan('../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/')\nscan_array = set_lungwin(get_pixels_hu(scans))\n\nimageio.mimsave(\"/tmp/gif.gif\", scan_array, duration=0.0001)\nImage(filename=\"/tmp/gif.gif\", format='png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Marker-controlled Watershed Transformation \n\n**We will be applying Marker-Controlled Watershed Transformation now**\n\n**Why?**\n\n**We do this because Watershed Transform is a really powerful segmentation algorithm, but has a drawback:**\n\n**Over Segmentation: Oversegmentation occurs because every regional minimum forms its own catchment basin. Here is an example where steel grains are over-segmented by watershed transformation**\n\n![](https://www.mathworks.com/company/newsletters/articles/the-watershed-transform-strategies-for-image-segmentation/_jcr_content/mainParsys/image_9.adapt.1200.high.gif/1542750812181.gif)\n\n![](https://www.mathworks.com/company/newsletters/articles/the-watershed-transform-strategies-for-image-segmentation/_jcr_content/mainParsys/image_10.adapt.1200.high.gif/1542750812206.gif)\n\n> Left: Steel Grains, Right: Oversegmented image as a result of using normal watershed transformation.\n\n**To overcome this drawback, we use a marker-controlled watershed transformation, where we manually create markers where we start the flooding process.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def generate_markers(image):\n    \"\"\"\n    Generates markers for a given image.\n    \n    Parameters: image\n    \n    Returns: Internal Marker, External Marker, Watershed Marker\n    \"\"\"\n    \n    #Creation of the internal Marker\n    marker_internal = image < -400\n    marker_internal = segmentation.clear_border(marker_internal)\n    marker_internal_labels = measure.label(marker_internal)\n    \n    areas = [r.area for r in measure.regionprops(marker_internal_labels)]\n    areas.sort()\n    \n    if len(areas) > 2:\n        for region in measure.regionprops(marker_internal_labels):\n            if region.area < areas[-2]:\n                for coordinates in region.coords:                \n                       marker_internal_labels[coordinates[0], coordinates[1]] = 0\n    \n    marker_internal = marker_internal_labels > 0\n    \n    # Creation of the External Marker\n    external_a = ndimage.binary_dilation(marker_internal, iterations=10)\n    external_b = ndimage.binary_dilation(marker_internal, iterations=55)\n    marker_external = external_b ^ external_a\n    \n    # Creation of the Watershed Marker\n    marker_watershed = np.zeros((512, 512), dtype=np.int)\n    marker_watershed += marker_internal * 255\n    marker_watershed += marker_external * 128\n    \n    return marker_internal, marker_external, marker_watershed","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Let's see the samples of Marker Watershed Transformation**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_patient_internal, test_patient_external, test_patient_watershed = generate_markers(test_patient_images[15])\n\nf, (ax1, ax2, ax3) = plt.subplots(1, 3, sharey=True, figsize=(15,15))\n\nax1.imshow(test_patient_internal, cmap='gray')\nax1.set_title(\"Internal Marker\")\nax1.axis('off')\n\nax2.imshow(test_patient_external, cmap='gray')\nax2.set_title(\"External Marker\")\nax2.axis('off')\n\nax3.imshow(test_patient_watershed, cmap='gray')\nax3.set_title(\"Watershed Marker\")\nax3.axis('off')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Mask DICOM images\n\n**Let's create a mask for some images of the Dataset, here the approach is morphological masking**\n\n**All Credits to [Welf Crozzo](https://www.kaggle.com/miklgr500/unsupervise-lung-detection)**\n\n**The determination of a mask for the lungs is the starting point in the algorithm for determining the volume of the lungs by CT images. The next step is the correct integration of all CT images to determine the volume of the lungs.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_image = pydicom.dcmread(train_image_files[7])\nimg = sample_image.pixel_array\n\nplt.imshow(img, cmap='gray')\nplt.title('Original Image')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Now we will create a Binary Mask Image with rescale intercept and slope and adjusting values below -400 HU**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"img = (img + sample_image.RescaleIntercept) / sample_image.RescaleSlope\nimg = img < -400 #HU unit range for lungs CT SCAN\n\nplt.imshow(img, cmap='gray')\nplt.title('Binary Mask Image')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Cleaning Border**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"img = clear_border(img)\nplt.imshow(img, cmap='gray')\nplt.title('Cleaned Border Image')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Labelling a small region of scan**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"img = label(img)\nplt.imshow(img, cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"areas = [r.area for r in regionprops(img)]\nareas.sort()\nif len(areas) > 2:\n    for region in regionprops(img):\n        if region.area < areas[-2]:\n            for coordinates in region.coords:                \n                img[coordinates[0], coordinates[1]] = 0\nimg = img > 0\nplt.imshow(img, cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Other Masks","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.raddq.com/dicom-processing-segmentation-visualization-in-python/\n\ndef make_lungmask(img, display=False):\n    row_size= img.shape[0]\n    col_size = img.shape[1]\n    \n    mean = np.mean(img)\n    std = np.std(img)\n    img = img-mean\n    img = img/std\n    \n    # Find the average pixel value near the lungs\n        # to renormalize washed out images\n    middle = img[int(col_size/5):int(col_size/5*4),int(row_size/5):int(row_size/5*4)] \n    mean = np.mean(middle)  \n    max = np.max(img)\n    min = np.min(img)\n    \n    # To improve threshold finding, I'm moving the \n    # underflow and overflow on the pixel spectrum\n    img[img==max]=mean\n    img[img==min]=mean\n    \n    # Using Kmeans to separate foreground (soft tissue / bone) and background (lung/air)\n    \n    kmeans = KMeans(n_clusters=2).fit(np.reshape(middle,[np.prod(middle.shape),1]))\n    centers = sorted(kmeans.cluster_centers_.flatten())\n    threshold = np.mean(centers)\n    thresh_img = np.where(img<threshold,1.0,0.0)  # threshold the image\n\n    # First erode away the finer elements, then dilate to include some of the pixels surrounding the lung.  \n    # We don't want to accidentally clip the lung.\n\n    eroded = morphology.erosion(thresh_img,np.ones([3,3]))\n    dilation = morphology.dilation(eroded,np.ones([8,8]))\n\n    labels = measure.label(dilation) # Different labels are displayed in different colors\n    label_vals = np.unique(labels)\n    regions = measure.regionprops(labels)\n    good_labels = []\n    for prop in regions:\n        B = prop.bbox\n        if B[2]-B[0]<row_size/10*9 and B[3]-B[1]<col_size/10*9 and B[0]>row_size/5 and B[2]<col_size/5*4:\n            good_labels.append(prop.label)\n    mask = np.ndarray([row_size,col_size],dtype=np.int8)\n    mask[:] = 0\n\n\n    #  After just the lungs are left, we do another large dilation\n    #  in order to fill in and out the lung mask \n    \n    for N in good_labels:\n        mask = mask + np.where(labels==N,1,0)\n    mask = morphology.dilation(mask,np.ones([10,10])) # one last dilation\n\n    if (display):\n        fig, ax = plt.subplots(3, 2, figsize=[12, 12])\n        ax[0, 0].set_title(\"Original\")\n        ax[0, 0].imshow(img, cmap='gray')\n        ax[0, 0].axis('off')\n        ax[0, 1].set_title(\"Threshold\")\n        ax[0, 1].imshow(thresh_img, cmap='gray')\n        ax[0, 1].axis('off')\n        ax[1, 0].set_title(\"After Erosion and Dilation\")\n        ax[1, 0].imshow(dilation, cmap='gray')\n        ax[1, 0].axis('off')\n        ax[1, 1].set_title(\"Color Labels\")\n        ax[1, 1].imshow(labels)\n        ax[1, 1].axis('off')\n        ax[2, 0].set_title(\"Final Mask\")\n        ax[2, 0].imshow(mask, cmap='gray')\n        ax[2, 0].axis('off')\n        ax[2, 1].set_title(\"Apply Mask on Original\")\n        ax[2, 1].imshow(mask*img, cmap='gray')\n        ax[2, 1].axis('off')\n        \n        plt.show()\n    return mask*img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Select a sample\npath = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/19.dcm\"\ndataset = pydicom.dcmread(path)\nimg = dataset.pixel_array\n\n# Masked image\nmask_img = make_lungmask(img, display=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import re\npatient_dir = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430\"\ndatasets = []\n\n# First Order the files in the dataset\nfiles = []\nfor dcm in list(os.listdir(patient_dir)):\n    files.append(dcm) \nfiles.sort(key=lambda f: int(re.sub('\\D', '', f)))\n\n# Read in the Dataset\nfor dcm in files:\n    path = patient_dir + \"/\" + dcm\n    datasets.append(pydicom.dcmread(path))\n    \nimgs = []\nfor data in datasets:\n    img = data.pixel_array\n    imgs.append(img)\n    \n    \n# Show masks\nfig=plt.figure(figsize=(16, 6))\ncolumns = 10\nrows = 3\n\nfor i in range(1, columns*rows +1):\n    img = make_lungmask(datasets[i-1].pixel_array)\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(img, cmap=\"gray\")\n    plt.title(i, fontsize = 9)\n    plt.axis('off');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Extract Metadata\n**Credits to [Rajasekhar](https://www.kaggle.com/trsekhar123/nb-to-extract-metadata-and-resize-images-train) for an easier implementation on this**\n\n**MetaData is a file that has a complete information of a data path and it's relevance to the project. So let's create one!!**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_observation_data(path):\n    '''Get information from the .dcm files.\n    path: complete path to the .dcm file'''\n\n    image_data = pydicom.read_file(path)\n\n    # Dictionary to store the information from the image\n    observation_data = {\n        \"FileNumber\" : path.split('/')[5],\n        \"Rows\" : image_data.Rows,\n        \"Columns\" : image_data.Columns,\n\n        \"PatientID\" : image_data.PatientID,\n        \"BodyPartExamined\" : image_data.BodyPartExamined,\n        \"SliceThickness\" : int(image_data.SliceThickness),\n        \"KVP\" : int(image_data.KVP),\n        \"DistanceSourceToDetector\" : int(image_data.DistanceSourceToDetector),\n        \"DistanceSourceToPatient\" : int(image_data.DistanceSourceToPatient),\n        \"GantryDetectorTilt\" : int(image_data.GantryDetectorTilt),\n        \"TableHeight\" : int(image_data.TableHeight),\n        \"RotationDirection\" : image_data.RotationDirection,\n        \"XRayTubeCurrent\" : int(image_data.XRayTubeCurrent),\n        \"GeneratorPower\" : int(image_data.GeneratorPower),\n        \"ConvolutionKernel\" : image_data.ConvolutionKernel,\n        \"PatientPosition\" : image_data.PatientPosition,\n\n        \"ImagePositionPatient\" : str(image_data.ImagePositionPatient),\n        \"ImageOrientationPatient\" : str(image_data.ImageOrientationPatient),\n        \"PhotometricInterpretation\" : image_data.PhotometricInterpretation,\n        \"ImageType\" : str(image_data.ImageType),\n        \"PixelSpacing\" : str(image_data.PixelSpacing),\n        \"WindowCenter\" : int(image_data.WindowCenter),\n        \"WindowWidth\" : int(image_data.WindowWidth),\n        \"Modality\" : image_data.Modality,\n        \"StudyInstanceUID\" : image_data.StudyInstanceUID,\n        \"PixelPaddingValue\" : image_data.PixelPaddingValue,\n        \"SamplesPerPixel\" : image_data.SamplesPerPixel,\n        \"SliceLocation\" : int(image_data.SliceLocation),\n        \"BitsAllocated\" : image_data.BitsAllocated,\n        \"BitsStored\" : image_data.BitsStored,\n        \"HighBit\" : image_data.HighBit,\n        \"PixelRepresentation\" : image_data.PixelRepresentation,\n        \"RescaleIntercept\" : int(image_data.RescaleIntercept),\n        \"RescaleSlope\" : int(image_data.RescaleSlope),\n        \"RescaleType\" : image_data.RescaleType\n    }\n    \n    return observation_data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Uncomment this if it is your first time in editing my kernel**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#meta_data_df = []\n#for filename in tqdm(train_image_files):\n    #try:\n       # meta_data_df.append(get_observation_data(filename))\n   # except Exception as e:\n      #  continue","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Takes 3-4 mins I think and does not require GPU!!**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Creating a Metadata file","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#meta_data_df = pd.DataFrame.from_dict(meta_data_df)\n#meta_data_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**So here we have captured the information pertaining to CT Scans performed on lungs, these can be very useful in conveying informations about a person's FVC and other parameters**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Exploring Metadata File","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dicom_df = pd.read_csv('../input/osic-dicom-image-features/metadata.csv')\ndicom_df.shape","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}