{"cells":[{"metadata":{},"cell_type":"markdown","source":"![OSIC Pulmonary Fibrosis](https://www.osicild.org/uploads/1/2/2/7/122798879/editor/kaggle-v01-clipped.png?1569346633)\n> Imagine one day, your breathing became consistently labored and shallow. Months later you were finally diagnosed with pulmonary fibrosis, a disorder with no known cause and no known cure, created by scarring of the lungs. If that happened to you, you would want to know your prognosis. That’s where a troubling disease becomes frightening for the patient: outcomes can range from long-term stability to rapid deterioration, but doctors aren’t easily able to tell where an individual may fall on that spectrum. Your help, and data science, may be able to aid in this prediction, which would dramatically help both patients and clinicians.\n\n> Current methods make fibrotic lung diseases difficult to treat, even with access to a chest CT scan. In addition, the wide range of varied prognoses create issues organizing clinical trials. Finally, patients suffer extreme anxiety—in addition to fibrosis-related symptoms—from the disease’s opaque path of progression.\n\n> Open Source Imaging Consortium (OSIC) is a not-for-profit, co-operative effort between academia, industry and philanthropy. The group enables rapid advances in the fight against Idiopathic Pulmonary Fibrosis (IPF), fibrosing interstitial lung diseases (ILDs), and other respiratory diseases, including emphysematous conditions. Its mission is to bring together radiologists, clinicians and computational scientists from around the world to improve imaging-based treatments.\n\n> In this competition, you’ll predict a patient’s severity of decline in lung function based on a CT scan of their lungs. You’ll determine lung function based on output from a spirometer, which measures the volume of air inhaled and exhaled. The challenge is to use machine learning techniques to make a prediction with the image, metadata, and baseline FVC as input.\n\n> If successful, patients and their families would better understand their prognosis when they are first diagnosed with this incurable lung disease. Improved severity detection would also positively impact treatment trial design and accelerate the clinical development of novel treatments.\n\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"scrolled":true},"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport re\nimport cv2\nimport matplotlib.pyplot as plt\nfrom matplotlib import rc\n\nimport pydicom\nimport plotly.graph_objs as go\nimport glob as glob\nimport imageio\n\n# Segmentation\nfrom skimage import morphology\nfrom scipy import ndimage\nfrom skimage import measure\nfrom skimage.transform import resize\nfrom sklearn.cluster import KMeans\nfrom plotly.tools import FigureFactory as FF\n\nimport warnings\nwarnings.simplefilter(action='ignore')\nsns.set(style=\"white\")\n\n%matplotlib inline\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/osic-pulmonary-fibrosis-progression/train.csv\")\ntest = pd.read_csv(\"/kaggle/input/osic-pulmonary-fibrosis-progression/test.csv\")\nsample = pd.read_csv(\"/kaggle/input/osic-pulmonary-fibrosis-progression/sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"\nprint(f\"No of patients are {train['Patient'].count()} with {train['Patient'].value_counts().shape[0]} unique patient ID\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"columns = list(train.columns)\nprint(f'The colums are {columns}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(7, 5))\nsns.despine(f)\nsns.distplot(train['Age'])\nplt.title(\"PATIENT AGE DISTRIBUTION\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"plt.figure(figsize=(8, 4))\nsns.countplot(x='SmokingStatus', data=train)\nplt.title(\"Smoking Status Patient\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"sns.scatterplot(x='FVC', y='Percent', hue='Sex', data=train, palette='PuOr')\nplt.title('Correlation between FVC and Percent')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"# FVC vs AGE\nsns.scatterplot(x='Age', y='FVC', hue='SmokingStatus', data=train, palette='Set1')\nplt.title('Correlation between FVC and Age')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"plt.title('Correlation between FVC and Age')\nsns.kdeplot(train['FVC'], train['Age'], cmap=\"Blues\", shade=True, shade_lowest=True )\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"heat_map = sns.heatmap(train.corr(method='pearson'), annot=True, fmt='.2f', linewidths=2)\nheat_map.set_xticklabels(heat_map.get_xticklabels(), rotation=45);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EXAMPLES OF PATIENT LUNGS"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"#Image pacient on Data\n\ndir = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/20.dcm\"\ndata = pydicom.dcmread(dir)\n\nprint(\"Patient id.......:\", data.PatientID, \"\\n\" +\n      \"Modality.........:\", data.Modality, \"\\n\" +\n      \"Rows.............:\", data.Rows, \"\\n\" +\n      \"Columns..........:\", data.Columns)\n\nplt.figure(figsize = (5, 5))\nplt.imshow(data.pixel_array, cmap=\"Blues_r\")\nplt.axis('off');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EXAMPLES OF PATIENT LUNGS"},{"metadata":{"trusted":true},"cell_type":"code","source":"dir = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430\"\ndata = []\n\n\nfiles = []\nfor dcm in list(os.listdir(dir)):\n    files.append(dcm) \nfiles.sort(key=lambda f: int(re.sub('\\D', '', f)))\n\n\nfor dcm in files:\n    path = dir + \"/\" + dcm\n    data.append(pydicom.dcmread(path))\n\n\nfig=plt.figure(figsize=(18, 6))\ncolumns = 10\nrows = 3\n\nfor i in range(1, columns*rows +1):\n    img = data[i-1].pixel_array\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(img, cmap=\"Blues_r\")\n    plt.title(i, fontsize = 12)\n    plt.axis('off');\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Create Mask of CT Scans Patient**\n**References** [Medical Images in Python](https://vincentblog.xyz/posts/medical-images-in-python-computed-tomography)"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndir = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/18.dcm\"\n\ndata = pydicom.dcmread(dir)\nprint(data)\nimage = data.pixel_array","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def transform_to_hu(data, image):\n    intercept = data.RescaleIntercept\n    slope = data.RescaleSlope\n    hu_image = image * slope + intercept\n\n    return hu_image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def window_image(image, window_center, window_width):\n    img_min = window_center - window_width // 2\n    img_max = window_center + window_width // 2\n    window_image = image.copy()\n    window_image[window_image < img_min] = img_min\n    window_image[window_image > img_max] = img_max\n    \n    return window_image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dir = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/16.dcm\"\ndata = pydicom.dcmread(dir)\n\ndef remove_noise(dir, display=False):\n    data = pydicom.dcmread(dir)\n    image = data.pixel_array\n    \n    hu_image = transform_to_hu(data, image)\n    brain_image = window_image(hu_image, 40, 80)\n\n    # morphology.dilation creates a segmentation of the image\n    # If one pixel is between the origin and the edge of a square of size\n    # 5x5, the pixel belongs to the same class\n    \n    # We can instead use a circule using: morphology.disk(2)\n    # In this case the pixel belongs to the same class if it's between the origin\n    # and the radius\n    \n    segmentation = morphology.dilation(brain_image, np.ones((5, 5)))\n    labels, label_nb = ndimage.label(segmentation)\n    \n    label_count = np.bincount(labels.ravel().astype(np.int))\n    # The size of label_count is the number of classes/segmentations found\n    \n    # We don't use the first class since it's the background\n    label_count[0] = 0\n    \n    # We create a mask with the class with more pixels\n    # In this case should be the brain\n    mask = labels == label_count.argmax()\n    \n    # Improve the brain mask\n    mask = morphology.dilation(mask, np.ones((5, 5)))\n    mask = ndimage.morphology.binary_fill_holes(mask)\n    mask = morphology.dilation(mask, np.ones((3, 3)))\n    \n    # Since the the pixels in the mask are zero's and one's\n    # We can multiple the original image to only keep the brain region\n    masked_image = mask * brain_image\n\n    if display:\n        plt.figure(figsize=(15, 2.5))\n        plt.subplot(141)\n        plt.imshow(brain_image)\n        plt.title('Original Image')\n        plt.axis('off')\n        \n        plt.subplot(142)\n        plt.imshow(mask)\n        plt.title('Mask')\n        plt.axis('off')\n\n        plt.subplot(143)\n        plt.imshow(masked_image)\n        plt.title('Final Image')\n        plt.axis('off')\n    \n    return masked_image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_ = remove_noise(dir, display=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dir = \"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/30.dcm\"\ndata = pydicom.dcmread(dir)\n\ndef crop_image(image, display=False):\n    # Create a mask with the background pixels\n    mask = image == 0\n\n    # Find the brain area\n    coords = np.array(np.nonzero(~mask))\n    top_left = np.min(coords, axis=1)\n    bottom_right = np.max(coords, axis=1)\n    \n    # Remove the background\n    croped_image = image[top_left[0]:bottom_right[0],\n                top_left[1]:bottom_right[1]]\n    \n    return croped_image\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final = remove_noise(dir)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(final)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final = crop_image(final)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(final)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}