{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<h1>RSNA Pneumonia Detection Challenge<h1>\nSociedade Beneficente de Senhoras - Hospital Sírio-Libanês - Brazil","metadata":{"_uuid":"4e52def28a73f1ca23e48d5b6495061bcf6ef966"}},{"cell_type":"markdown","source":"# **Contents**\n1. [Overview](#1.-Overview)\n1. [Data preparation](#2.-Data-preparation)\n1. [Segmentation training](#3.-Segmentation-training)\n1. [Results](#4.-Results)","metadata":{"_uuid":"828d2fadb694c84f41867478a9608b822949f6a7"}},{"cell_type":"markdown","source":"# 1. Overview\nThis notebook follows the work of [Kevin Mader](https://www.kaggle.com/kmader/training-u-net-on-tb-images-to-segment-lungs/notebook) for lung segmentation. Our motivation is to automatically identify lung opacities in chest x-rays for the [RSNA Pneumonia Detection Challenge](https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/leaderboard). \n\nMedical Image Segmentation is the process of automatic detection of boundaries within images. In this exercise, we train a convolutional neural network with [U-Net](https://arxiv.org/abs/1505.04597) architecture, which training strategy relies on the strong use of data augmentation to improve the efficiency of available annotated samples.\n\nThe training is done with two chest x-rays datasets: [Montgomery County and Shenzhen Hospital](https://ceb.nlm.nih.gov/repositories/tuberculosis-chest-x-ray-image-data-sets/). The Montgomery County dataset includes manually segmented lung masks, whereas Shenzhen Hospital dataset was manually segmented by [Stirenko et al](https://arxiv.org/abs/1803.01199). The lung segmentation masks were dilated to load lung boundary information within the training net and the images were resized to 512x512 pixels.","metadata":{"_uuid":"516c9b53d192813594987edec5c0552c4449607c"}},{"cell_type":"markdown","source":"# 2. Data preparation\nPrepare the input segmentation directory structure.","metadata":{"_uuid":"2392d90df7ff92794b4d5ef9c47c102b289fd724"}},{"cell_type":"code","source":"!mkdir /kaggle/working/segmentation\n!mkdir /kaggle/working/segmentation/test\n!mkdir /kaggle/working/segmentation/test_auf\n!mkdir /kaggle/working/segmentation/train\n!mkdir /kaggle/working/segmentation/train/augmentation\n!mkdir /kaggle/working/segmentation/train/image\n!mkdir /kaggle/working/segmentation/train/mask\n!mkdir /kaggle/working/segmentation/train/dilate","metadata":{"_uuid":"763240ce0dbdfb26510ded9b55b5bc94d400f74e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Import required Python libraries","metadata":{"_uuid":"e222153d13b6602792f447ed86401f3ee5d4965b"}},{"cell_type":"code","source":"import os\n\nimport numpy as np\nimport cv2\nimport matplotlib.pyplot as plt\n\nfrom keras.models import *\nfrom keras.layers import *\nfrom keras.optimizers import *\nfrom keras import backend as keras\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ModelCheckpoint, LearningRateScheduler\n\nfrom glob import glob\nfrom tqdm import tqdm","metadata":{"_uuid":"cb2b30651cf57582f7715325ea9519f04f4407d4","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Define appropriate constants for directory paths and training parameters","metadata":{"_uuid":"2f6aa09deb74f25f9937d16ef32ddc32a22d780c"}},{"cell_type":"code","source":"INPUT_DIR = os.path.join(\"..\", \"input\")\nMY_DIR = \"/kaggle/working\"\n\nSEGMENTATION_DIR = os.path.join(MY_DIR, \"segmentation\")\nSEGMENTATION_TEST_DIR = os.path.join(SEGMENTATION_DIR, \"test\")\nSEGMENTATION_TEST_DIR_AUF = os.path.join(SEGMENTATION_DIR, \"test_auf\")\nSEGMENTATION_TRAIN_DIR = os.path.join(SEGMENTATION_DIR, \"train\")\nSEGMENTATION_AUG_DIR = os.path.join(SEGMENTATION_TRAIN_DIR, \"augmentation\")\nSEGMENTATION_IMAGE_DIR = os.path.join(SEGMENTATION_TRAIN_DIR, \"image\")\nSEGMENTATION_MASK_DIR = os.path.join(SEGMENTATION_TRAIN_DIR, \"mask\")\nSEGMENTATION_DILATE_DIR = os.path.join(SEGMENTATION_TRAIN_DIR, \"dilate\")\nSEGMENTATION_SOURCE_DIR = os.path.join(INPUT_DIR, \\\n                                       \"pulmonary-chest-xray-abnormalities\")\nSEGMENTATION_SOURCE_DIR_AUF = os.path.join(INPUT_DIR, \\\n                                           \"covid19-radiography-database\")\n\nCANDEMIR_TRAIN_DIR = os.path.join(SEGMENTATION_SOURCE_DIR, \"lung-mask-candemir\")\nCANDEMIR_IMAGE_DIR = os.path.join(CANDEMIR_TRAIN_DIR, \"Model_Xrays\")\nCANDEMIR_MASK_DIR = os.paht.join(CANDEMIR_MASK_DIR, \"Masks\")\n\n\nSHENZHEN_TRAIN_DIR = os.path.join(SEGMENTATION_SOURCE_DIR, \"ChinaSet_AllFiles\", \\\n                                  \"ChinaSet_AllFiles\")\nSHENZHEN_IMAGE_DIR = os.path.join(SHENZHEN_TRAIN_DIR, \"CXR_png\")\nSHENZHEN_MASK_DIR = os.path.join(INPUT_DIR, \"shcxr-lung-mask\", \"mask\", \"mask\")\n\nMONTGOMERY_TRAIN_DIR = os.path.join(SEGMENTATION_SOURCE_DIR, \\\n                                    \"Montgomery\", \"MontgomerySet\")\nMONTGOMERY_IMAGE_DIR = os.path.join(MONTGOMERY_TRAIN_DIR, \"CXR_png\")\nMONTGOMERY_LEFT_MASK_DIR = os.path.join(MONTGOMERY_TRAIN_DIR, \\\n                                        \"ManualMask\", \"leftMask\")\nMONTGOMERY_RIGHT_MASK_DIR = os.path.join(MONTGOMERY_TRAIN_DIR, \\\n                                         \"ManualMask\", \"rightMask\")\n\nDILATE_KERNEL = np.ones((15, 15), np.uint8)\n\nBATCH_SIZE=2\n\n#Prod\nEPOCHS=56\n\n#Desv\n#EPOCHS=16","metadata":{"_uuid":"5307ff8180fe09ccb769e1e9809e33061451c5e1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"1. My Candemir dataset\n1. Resize images to 512x512 pixels\n1. Dilate masks to gain more information on the edge of lungs\n1. Split images into training and test datasets\n1. Write images to /segmentation directory","metadata":{"_uuid":"8a8adbf02b8a5c0b014b4e833e48d089e4f89035"}},{"cell_type":"code","source":"candemir_dir   = glob(os.path.join(CANDEMIR_IMAGE_DIR, '*.png'))\ncandemir_test  = candemir_dir[0:10]\ncandemir_train = candemir_dir[90:]\n\nfor image_file in tqdm(candemir_dir):\n    base_file = os.path.basename(image_file)\n    mask_file = os.path.join(CANDEMIR_MASK_DIR, base_file)\n    image = cv2.imread(image_file)\n    mask = cv2.imread(mask_file)\n    mask_dilate = cv2.dilate(mask, DILATE_KERNEL, iterations=1)\n\n    if (image_file in candemir_train):\n        cv2.imwrite(os.path.join(SEGMENTATION_IMAGE_DIR, base_file), \\\n                    image)\n        cv2.imwrite(os.path.join(SEGMENTATION_MASK_DIR, base_file), \\\n                    mask)\n        cv2.imwrite(os.path.join(SEGMENTATION_DILATE_DIR, base_file), \\\n                    mask_dilate)\n    else:\n        filename, fileext = os.path.splitext(base_file)\n        cv2.imwrite(os.path.join(SEGMENTATION_TEST_DIR, base_file), \\\n                    image)\n        cv2.imwrite(os.path.join(SEGMENTATION_TEST_DIR, \\\n                                 \"%s_mask%s\" % (filename, fileext)), mask)\n        cv2.imwrite(os.path.join(SEGMENTATION_TEST_DIR, \\\n                                 \"%s_dilate%s\" % (filename, fileext)), mask_dilate)\n    ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Combine left and right lung segmentation masks of Montgomery chest x-rays\nResize images to 512x512 pixels\nDilate masks to gain more information on the edge of lungs\nSplit images into training and test datasets\nWrite images to /segmentation directory","metadata":{}},{"cell_type":"code","source":"montgomery_left_mask_dir = glob(os.path.join(MONTGOMERY_LEFT_MASK_DIR, '*.png'))\nmontgomery_test = montgomery_left_mask_dir[0:10]\nmontgomery_train= montgomery_left_mask_dir[90:]\n\nfor left_image_file in tqdm(montgomery_left_mask_dir):\n    base_file = os.path.basename(left_image_file)\n    image_file = os.path.join(MONTGOMERY_IMAGE_DIR, base_file)\n    right_image_file = os.path.join(MONTGOMERY_RIGHT_MASK_DIR, base_file)\n\n    image = cv2.imread(image_file)\n    left_mask = cv2.imread(left_image_file, cv2.IMREAD_GRAYSCALE)\n    right_mask = cv2.imread(right_image_file, cv2.IMREAD_GRAYSCALE)\n    \n    image = cv2.resize(image, (512, 512))\n    left_mask = cv2.resize(left_mask, (512, 512))\n    right_mask = cv2.resize(right_mask, (512, 512))\n    \n    mask = np.maximum(left_mask, right_mask)\n    mask_dilate = cv2.dilate(mask, DILATE_KERNEL, iterations=1)\n    \n    if (left_image_file in montgomery_train):\n        cv2.imwrite(os.path.join(SEGMENTATION_IMAGE_DIR, base_file), \\\n                    image)\n        cv2.imwrite(os.path.join(SEGMENTATION_MASK_DIR, base_file), \\\n                    mask)\n        cv2.imwrite(os.path.join(SEGMENTATION_DILATE_DIR, base_file), \\\n                    mask_dilate)\n    else:\n        filename, fileext = os.path.splitext(base_file)\n        cv2.imwrite(os.path.join(SEGMENTATION_TEST_DIR, base_file), \\\n                    image)\n        cv2.imwrite(os.path.join(SEGMENTATION_TEST_DIR, \\\n                                 \"%s_mask%s\" % (filename, fileext)), mask)\n        cv2.imwrite(os.path.join(SEGMENTATION_TEST_DIR, \\\n                                 \"%s_dilate%s\" % (filename, fileext)), mask_dilate)","metadata":{"_uuid":"b8e995d39dc50ae09aaca9fca60d5f01a4afa493","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Define some useful functions to display images with segmentation as overlays","metadata":{"_uuid":"d6f4b42dcde137249d107e754efa7c25087224d1"}},{"cell_type":"code","source":"def add_colored_dilate(image, mask_image, dilate_image):\n    mask_image_gray = cv2.cvtColor(mask_image, cv2.COLOR_BGR2GRAY)\n    dilate_image_gray = cv2.cvtColor(dilate_image, cv2.COLOR_BGR2GRAY)\n    \n    mask = cv2.bitwise_and(mask_image, mask_image, mask=mask_image_gray)\n    dilate = cv2.bitwise_and(dilate_image, dilate_image, mask=dilate_image_gray)\n    \n    mask_coord = np.where(mask!=[0,0,0])\n    dilate_coord = np.where(dilate!=[0,0,0])\n\n    mask[mask_coord[0],mask_coord[1],:]=[255,0,0]\n    dilate[dilate_coord[0],dilate_coord[1],:] = [0,0,255]\n\n    ret = cv2.addWeighted(image, 0.7, dilate, 0.3, 0)\n    ret = cv2.addWeighted(ret, 0.7, mask, 0.3, 0)\n\n    return ret\n\ndef add_colored_mask(image, mask_image):\n    mask_image_gray = cv2.cvtColor(mask_image, cv2.COLOR_BGR2GRAY)\n    \n    mask = cv2.bitwise_and(mask_image, mask_image, mask=mask_image_gray)\n    \n    mask_coord = np.where(mask!=[0,0,0])\n\n    mask[mask_coord[0],mask_coord[1],:]=[255,0,0]\n\n    ret = cv2.addWeighted(image, 0.7, mask, 0.3, 0)\n\n    return ret\n\ndef diff_mask(ref_image, mask_image):\n    mask_image_gray = cv2.cvtColor(mask_image, cv2.COLOR_BGR2GRAY)\n    \n    mask = cv2.bitwise_and(mask_image, mask_image, mask=mask_image_gray)\n    \n    mask_coord = np.where(mask!=[0,0,0])\n\n    mask[mask_coord[0],mask_coord[1],:]=[255,0,0]\n\n    ret = cv2.addWeighted(ref_image, 0.7, mask, 0.3, 0)\n    return ret","metadata":{"_uuid":"46e63227a43918c9189e192402be274f5e1e4466","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Show some Montgomery chest x-rays and its lung segmentation masks from training and test dataset to verify the procedure above. In merged image it is possible to see the difference between the dilated mask in blue and the original mask in red.","metadata":{"_uuid":"a5eb09219a9cb07e2bd6bc855304ad5fb805415f"}},{"cell_type":"code","source":"base_file = os.path.basename(montgomery_train[0])\n\nimage_file = os.path.join(SEGMENTATION_IMAGE_DIR, base_file)\nmask_image_file = os.path.join(SEGMENTATION_MASK_DIR, base_file)\ndilate_image_file = os.path.join(SEGMENTATION_DILATE_DIR, base_file)\n\nimage = cv2.imread(image_file)\nmask_image = cv2.imread(mask_image_file)\ndilate_image = cv2.imread(dilate_image_file)\nmerged_image = add_colored_dilate(image, mask_image, dilate_image)\n                          \nfig, axs = plt.subplots(2, 4, figsize=(15, 8))\n\naxs[0, 0].set_title(\"X-Ray\")\naxs[0, 0].imshow(image)\n\naxs[0, 1].set_title(\"Mask\")\naxs[0, 1].imshow(mask_image)\n\naxs[0, 2].set_title(\"Dilate\")\naxs[0, 2].imshow(dilate_image)\n\naxs[0, 3].set_title(\"Merged\")\naxs[0, 3].imshow(merged_image)\n\nbase_file = os.path.basename(montgomery_test[0])\nfilename, fileext = os.path.splitext(base_file)\nimage_file = os.path.join(SEGMENTATION_TEST_DIR, base_file)\nmask_image_file = os.path.join(SEGMENTATION_TEST_DIR, \\\n                               \"%s_mask%s\" % (filename, fileext))\ndilate_image_file = os.path.join(SEGMENTATION_TEST_DIR, \\\n                                 \"%s_dilate%s\" % (filename, fileext))\n\nimage = cv2.imread(image_file)\nmask_image = cv2.imread(mask_image_file)\ndilate_image = cv2.imread(dilate_image_file)\nmerged_image = add_colored_dilate(image, mask_image, dilate_image)\n\naxs[1, 0].set_title(\"X-Ray\")\naxs[1, 0].imshow(image)\n\naxs[1, 1].set_title(\"Mask\")\naxs[1, 1].imshow(mask_image)\n\naxs[1, 2].set_title(\"Dilate\")\naxs[1, 2].imshow(dilate_image)\n\naxs[1, 3].set_title(\"Merged\")\naxs[1, 3].imshow(merged_image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"1. Resize Shenzhen Hospital chest x-ray images to 512x512 pixels\n1. Dilate masks to gain more information on the edge of lungs\n1. Split images into training and test datasets\n1. Write images to /segmentation directory","metadata":{"_uuid":"961855a0ef2989b675ad037a1c674acf53701396"}},{"cell_type":"code","source":"shenzhen_mask_dir = glob(os.path.join(SHENZHEN_MASK_DIR, '*.png'))\nshenzhen_test = shenzhen_mask_dir[0:1]\nshenzhen_train= shenzhen_mask_dir[99:]\n\nfor mask_file in tqdm(shenzhen_mask_dir):\n    base_file = os.path.basename(mask_file).replace(\"_mask\", \"\")\n    image_file = os.path.join(SHENZHEN_IMAGE_DIR, base_file)\n\n    image = cv2.imread(image_file)\n    mask = cv2.imread(mask_file, cv2.IMREAD_GRAYSCALE)\n        \n    image = cv2.resize(image, (512, 512))\n    mask = cv2.resize(mask, (512, 512))\n    mask_dilate = cv2.dilate(mask, DILATE_KERNEL, iterations=1)\n    \n    if (mask_file in shenzhen_train):\n        cv2.imwrite(os.path.join(SEGMENTATION_IMAGE_DIR, base_file), \\\n                    image)\n        cv2.imwrite(os.path.join(SEGMENTATION_MASK_DIR, base_file), \\\n                    mask)\n        cv2.imwrite(os.path.join(SEGMENTATION_DILATE_DIR, base_file), \\\n                    mask_dilate)\n    else:\n        filename, fileext = os.path.splitext(base_file)\n\n        cv2.imwrite(os.path.join(SEGMENTATION_TEST_DIR, base_file), \\\n                    image)\n        cv2.imwrite(os.path.join(SEGMENTATION_TEST_DIR, \\\n                                 \"%s_mask%s\" % (filename, fileext)), mask)\n        cv2.imwrite(os.path.join(SEGMENTATION_TEST_DIR, \\\n                                 \"%s_dilate%s\" % (filename, fileext)), mask_dilate)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Show some Shenzhen Hospital chest x-rays and its lung segmentation masks from training and test dataset to verify the procedure above. In merged image it is possible to see the difference between the dilated mask in blue and the original mask in red.","metadata":{"_uuid":"9e6e9dbb064305877847351015aba026582f7bb3"}},{"cell_type":"code","source":"base_file = os.path.basename(shenzhen_train[0].replace(\"_mask\", \"\"))\n\nimage_file = os.path.join(SEGMENTATION_IMAGE_DIR, base_file)\nmask_image_file = os.path.join(SEGMENTATION_MASK_DIR, base_file)\ndilate_image_file = os.path.join(SEGMENTATION_DILATE_DIR, base_file)\n\nimage = cv2.imread(image_file)\nmask_image = cv2.imread(mask_image_file)\ndilate_image = cv2.imread(dilate_image_file)\nmerged_image = add_colored_dilate(image, mask_image, dilate_image)\n                          \nfig, axs = plt.subplots(2, 4, figsize=(15, 8))\n\naxs[0, 0].set_title(\"X-Ray\")\naxs[0, 0].imshow(image)\n\naxs[0, 1].set_title(\"Mask\")\naxs[0, 1].imshow(mask_image)\n\naxs[0, 2].set_title(\"Dilate\")\naxs[0, 2].imshow(dilate_image)\n\naxs[0, 3].set_title(\"Merged\")\naxs[0, 3].imshow(merged_image)\n\nbase_file = os.path.basename(shenzhen_test[0].replace(\"_mask\", \"\"))\nimage_file = os.path.join(SEGMENTATION_TEST_DIR, base_file)\nfilename, fileext = os.path.splitext(base_file)\nmask_image_file = os.path.join(SEGMENTATION_TEST_DIR, \\\n                               \"%s_mask%s\" % (filename, fileext))\n\nfilename, fileext = os.path.splitext(base_file)\nimage_file = os.path.join(SEGMENTATION_TEST_DIR, base_file)\nmask_image_file = os.path.join(SEGMENTATION_TEST_DIR, \\\n                               \"%s_mask%s\" % (filename, fileext))\ndilate_image_file = os.path.join(SEGMENTATION_TEST_DIR, \\\n                                 \"%s_dilate%s\" % (filename, fileext))\n\nimage = cv2.imread(image_file)\nmask_image = cv2.imread(mask_image_file)\ndilate_image = cv2.imread(dilate_image_file)\nmerged_image = add_colored_dilate(image, mask_image, dilate_image)\n\naxs[1, 0].set_title(\"X-Ray\")\naxs[1, 0].imshow(image)\n\naxs[1, 1].set_title(\"Mask\")\naxs[1, 1].imshow(mask_image)\n\naxs[1, 2].set_title(\"Dilate\")\naxs[1, 2].imshow(dilate_image)\n\naxs[1, 3].set_title(\"Merged\")\naxs[1, 3].imshow(merged_image)","metadata":{"_uuid":"e9923cd36413cc086db684d3a076cbba9b479778","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Print the count of images and segmentation lung masks available to test and train the model","metadata":{"_uuid":"67e2abed6ba57198cd442763925af44b493569ec"}},{"cell_type":"code","source":"train_files = glob(os.path.join(SEGMENTATION_IMAGE_DIR, \"*.png\"))\ntest_files = glob(os.path.join(SEGMENTATION_TEST_DIR, \"*.png\"))\nmask_files = glob(os.path.join(SEGMENTATION_MASK_DIR, \"*.png\"))\ndilate_files = glob(os.path.join(SEGMENTATION_DILATE_DIR, \"*.png\"))\n\n(len(train_files), \\\n len(test_files), \\\n len(mask_files), \\\n len(dilate_files))","metadata":{"_uuid":"e5381719ef7e49e6823c1a9417e7eff7cfb1572e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Segmentation training\n\nReferences: https://github.com/zhixuhao/unet/, https://github.com/jocicmarko/ultrasound-nerve-segmentation","metadata":{"_uuid":"774eec88428a8d1899247401acfef369e20abd2e"}},{"cell_type":"markdown","source":"Data augmentation helper function for training the net","metadata":{"_uuid":"ab2175b11034175361ed8595ef8715afd76fd04a"}},{"cell_type":"code","source":"# From: https://github.com/zhixuhao/unet/blob/master/data.py\ndef train_generator(batch_size, train_path, image_folder, mask_folder, aug_dict,\n        image_color_mode=\"grayscale\",\n        mask_color_mode=\"grayscale\",\n        image_save_prefix=\"image\",\n        mask_save_prefix=\"mask\",\n        save_to_dir=None,\n        target_size=(256,256),\n        seed=1):\n    '''\n    can generate image and mask at the same time use the same seed for\n    image_datagen and mask_datagen to ensure the transformation for image\n    and mask is the same if you want to visualize the results of generator,\n    set save_to_dir = \"your path\"\n    '''\n    image_datagen = ImageDataGenerator(**aug_dict)\n    mask_datagen = ImageDataGenerator(**aug_dict)\n    \n    image_generator = image_datagen.flow_from_directory(\n        train_path,\n        classes = [image_folder],\n        class_mode = None,\n        color_mode = image_color_mode,\n        target_size = target_size,\n        batch_size = batch_size,\n        save_to_dir = save_to_dir,\n        save_prefix  = image_save_prefix,\n        seed = seed)\n\n    mask_generator = mask_datagen.flow_from_directory(\n        train_path,\n        classes = [mask_folder],\n        class_mode = None,\n        color_mode = mask_color_mode,\n        target_size = target_size,\n        batch_size = batch_size,\n        save_to_dir = save_to_dir,\n        save_prefix  = mask_save_prefix,\n        seed = seed)\n\n    train_gen = zip(image_generator, mask_generator)\n    \n    for (img, mask) in train_gen:\n        img, mask = adjust_data(img, mask)\n        yield (img,mask)\n\ndef adjust_data(img,mask):\n    img = img / 255\n    mask = mask / 255\n    mask[mask > 0.5] = 1\n    mask[mask <= 0.5] = 0\n    \n    return (img, mask)","metadata":{"_uuid":"2e51bb0ff0d463ff5fc26af0d7bc0f3df745bdaa","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"U-net architecture","metadata":{"_uuid":"11ad24573000715e344f314f9766867d8b228d45"}},{"cell_type":"code","source":"# From: https://github.com/jocicmarko/ultrasound-nerve-segmentation/blob/master/train.py\ndef dice_coef(y_true, y_pred):\n    y_true_f = keras.flatten(y_true)\n    y_pred_f = keras.flatten(y_pred)\n    intersection = keras.sum(y_true_f * y_pred_f)\n    return (2. * intersection + 1) / (keras.sum(y_true_f) + keras.sum(y_pred_f) + 1)\n\ndef dice_coef_loss(y_true, y_pred):\n    return -dice_coef(y_true, y_pred)\n\ndef unet(input_size=(256,256,1)):\n    inputs = Input(input_size)\n    \n    conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(inputs)\n    conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv1)\n    pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)\n\n    conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(pool1)\n    conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv2)\n    pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)\n\n    conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(pool2)\n    conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv3)\n    pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)\n\n    conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(pool3)\n    conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv4)\n    pool4 = MaxPooling2D(pool_size=(2, 2))(conv4)\n\n    conv5 = Conv2D(512, (3, 3), activation='relu', padding='same')(pool4)\n    conv5 = Conv2D(512, (3, 3), activation='relu', padding='same')(conv5)\n\n    up6 = concatenate([Conv2DTranspose(256, (2, 2), strides=(2, 2), padding='same')(conv5), conv4], axis=3)\n    conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(up6)\n    conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv6)\n\n    up7 = concatenate([Conv2DTranspose(128, (2, 2), strides=(2, 2), padding='same')(conv6), conv3], axis=3)\n    conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(up7)\n    conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv7)\n\n    up8 = concatenate([Conv2DTranspose(64, (2, 2), strides=(2, 2), padding='same')(conv7), conv2], axis=3)\n    conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(up8)\n    conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv8)\n\n    up9 = concatenate([Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same')(conv8), conv1], axis=3)\n    conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(up9)\n    conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv9)\n\n    conv10 = Conv2D(1, (1, 1), activation='sigmoid')(conv9)\n\n    return Model(inputs=[inputs], outputs=[conv10])","metadata":{"_uuid":"fd23bd3ff7c4ef8186043cfcc1a161b7b90ae893","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Helper functions to load test chest x-ray images","metadata":{"_uuid":"a5af02b028aec42c9b1e9f9312114febe0df602c"}},{"cell_type":"code","source":"# From: https://github.com/zhixuhao/unet/blob/master/data.py\ndef test_load_image(test_file, target_size=(256,256)):\n    img = cv2.imread(test_file, cv2.IMREAD_GRAYSCALE)\n    img = img / 255\n    img = cv2.resize(img, target_size)\n    img = np.reshape(img, img.shape + (1,))\n    img = np.reshape(img,(1,) + img.shape)\n    return img\n\ndef test_generator(test_files, target_size=(256,256)):\n    for test_file in test_files:\n        yield test_load_image(test_file, target_size)\n        \ndef save_result(save_path, npyfile, test_files):\n    for i, item in enumerate(npyfile):\n        result_file = test_files[i]\n        img = (item[:, :, 0] * 255.).astype(np.uint8)\n\n        filename, fileext = os.path.splitext(os.path.basename(result_file))\n\n        result_file = os.path.join(save_path, \"%s_predict%s\" % (filename, fileext))\n\n        cv2.imwrite(result_file, img)","metadata":{"_uuid":"00c16a2cd8fe3b47cddb7c4a7c1de31b5bae481c","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Select test and validation files","metadata":{"_uuid":"c92b6dc486eefe890feec5fbd760cb766a3612f1"}},{"cell_type":"code","source":"def add_suffix(base_file, suffix):\n    filename, fileext = os.path.splitext(base_file)\n    return \"%s_%s%s\" % (filename, suffix, fileext)\n\ntest_files = [test_file for test_file in glob(os.path.join(SEGMENTATION_TEST_DIR, \"*.png\")) \\\n              if (\"_mask\" not in test_file \\\n                  and \"_dilate\" not in test_file \\\n                  and \"_predict\" not in test_file)]\n\nvalidation_data = (test_load_image(test_files[0], target_size=(512, 512)),\n                    test_load_image(add_suffix(test_files[0], \"dilate\"), target_size=(512, 512)))\n\nlen(test_files), len(validation_data)","metadata":{"_uuid":"09df4552b2a1c6935849214f4f439d2c581d8c16","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_files_auf = glob('../input/covid19-radiography-database/COVID-19_Radiography_Dataset/COVID/*.png')\nlen(test_files_auf)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Prepare the U-Net model and train the model. It will take a while...","metadata":{"_uuid":"f48364da322a5fd84178cac099510598e8c67f77"}},{"cell_type":"code","source":"train_generator_args = dict(rotation_range=0.2,\n                            width_shift_range=0.05,\n                            height_shift_range=0.05,\n                            shear_range=0.05,\n                            zoom_range=0.05,\n                            horizontal_flip=True,\n                            fill_mode='nearest')\n\ntrain_gen = train_generator(BATCH_SIZE,\n                            SEGMENTATION_TRAIN_DIR,\n                            'image',\n                            'dilate', \n                            train_generator_args,\n                            target_size=(512,512),\n                            save_to_dir=os.path.abspath(SEGMENTATION_AUG_DIR))\n\nmodel = unet(input_size=(512,512,1))\nmodel.compile(optimizer=Adam(lr=1e-5), loss=dice_coef_loss, \\\n                  metrics=[dice_coef, 'binary_accuracy'])\nmodel.summary()\n\nmodel_checkpoint = ModelCheckpoint('unet_lung_seg.hdf5', \n                                   monitor='loss', \n                                   verbose=1, \n                                   save_best_only=True)\n\nhistory = model.fit_generator(train_gen,\n                              steps_per_epoch=len(train_files) / BATCH_SIZE, \n                              epochs=EPOCHS, \n                              callbacks=[model_checkpoint],\n                              validation_data = validation_data)","metadata":{"_uuid":"9893f1eedda1d1fe36cfd042cc28144dd9d6b1f0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Show some results from model fitting history","metadata":{"_uuid":"7fa6d4810998c47cca1d54005c5beb873cf3da8a"}},{"cell_type":"code","source":"fig, axs = plt.subplots(1, 2, figsize = (15, 4))\n\ntraining_loss = history.history['loss']\nvalidation_loss = history.history['val_loss']\n\ntraining_accuracy = history.history['binary_accuracy']\nvalidation_accuracy = history.history['val_binary_accuracy']\n\nepoch_count = range(1, len(training_loss) + 1)\n\naxs[0].plot(epoch_count, training_loss, 'r--')\naxs[0].plot(epoch_count, validation_loss, 'b-')\naxs[0].legend(['Training Loss', 'Validation Loss'])\n\naxs[1].plot(epoch_count, training_accuracy, 'r--')\naxs[1].plot(epoch_count, validation_accuracy, 'b-')\naxs[1].legend(['Training Accuracy', 'Validation Accuracy'])","metadata":{"_uuid":"925c24f71ba1c8e3697312ff7315e5e9f8059a0b","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Make lung segmentation predictions","metadata":{"_uuid":"7ac5b6dbe1dd70125529ffe46d748ddc2c4cc2a3"}},{"cell_type":"code","source":"test_gen = test_generator(test_files, target_size=(512,512))\nresults = model.predict_generator(test_gen, len(test_files), verbose=1)\nsave_result(SEGMENTATION_TEST_DIR, results, test_files)","metadata":{"_uuid":"15ad649aa0d5e94f42a78b58c8d58cd96d82a42d","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_gen_auf = test_generator(test_files_auf, target_size=(512,512))\nresults_auf = model.predict_generator(test_gen_auf, len(test_files_auf), verbose=1)\nsave_result(SEGMENTATION_TEST_DIR_AUF, results_auf, test_files_auf)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"glob(os.path.join(SEGMENTATION_TEST_DIR_AUF,\"*.png\"))\n","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Results\n\nBelow, we see some results from our work, presented as Predicted, Gold Standard (manually segmented) and the difference between segmentations.\n\nThe next step will be the selection of lungs area on RSNA images dataset and the generation of a lungs-only image dataset.","metadata":{"_uuid":"067ce01e33264729319d4c3105e99e30e5a19163"}},{"cell_type":"code","source":"#image = cv2.imread(\"../input/segmentation/test_auf/COVID-2312_predict.png\")\nglob(os.path.join(SEGMENTATION_TEST_DIR,\"*.png\"))","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image = cv2.imread(\"/kaggle/working/segmentation/test/CHNCXR_0635_1.png\")\npredict_image = cv2.imread(\"/kaggle/working/segmentation/test//CHNCXR_0635_1_predict.png\")\nmask_image = cv2.imread(\"/kaggle/working/segmentation/test/CHNCXR_0635_1_dilate.png\")\n\nfig, axs = plt.subplots(4, 3, figsize=(16, 16))\n\naxs[0, 0].set_title(\"Predicted\")\naxs[0, 0].imshow(add_colored_mask(image, mask_image)) #predict_image))\naxs[0, 1].set_title(\"Gold Std.\")\naxs[0, 1].imshow(add_colored_mask(image, mask_image))\naxs[0, 2].set_title(\"Diff.\")\naxs[0, 2].imshow(diff_mask(mask_image, predict_image))\n\nimage = cv2.imread(\"/kaggle/working/segmentation/test/MCUCXR_0084_0.png\")\npredict_image = cv2.imread(\"/kaggle/working/segmentation/test/MCUCXR_0084_0_predict.png\")\nmask_image = cv2.imread(\"/kaggle/working/segmentation/test/MCUCXR_0084_0_dilate.png\")\n\naxs[1, 0].set_title(\"Predicted\")\naxs[1, 0].imshow(add_colored_mask(image, predict_image))\naxs[1, 1].set_title(\"Gold Std.\")\naxs[1, 1].imshow(add_colored_mask(image, mask_image))\naxs[1, 2].set_title(\"Diff.\")\naxs[1, 2].imshow(diff_mask(mask_image, predict_image))\n\nimage = cv2.imread(\"/kaggle/working/segmentation/test/MCUCXR_0058_0.png\")\npredict_image = cv2.imread(\"/kaggle/working/segmentation/test/MCUCXR_0058_0_predict.png\")\nmask_image = cv2.imread(\"/kaggle/working/segmentation/test/MCUCXR_0058_0_dilate.png\")\n\naxs[2, 0].set_title(\"Predicted\")\naxs[2, 0].imshow(add_colored_mask(image, predict_image))\naxs[2, 1].set_title(\"Gold Std.\")\naxs[2, 1].imshow(add_colored_mask(image, mask_image))\naxs[2, 2].set_title(\"Diff.\")\naxs[2, 2].imshow(diff_mask(mask_image, predict_image))\n\nimage = cv2.imread(\"/kaggle/working/segmentation/test/CHNCXR_0234_0.png\")\npredict_image = cv2.imread(\"/kaggle/working/segmentation/test/CHNCXR_0234_0_predict.png\")\nmask_image = cv2.imread(\"/kaggle/working/segmentation/test/CHNCXR_0234_0_dilate.png\")\n\naxs[3, 0].set_title(\"Predicted\")\naxs[3, 0].imshow(add_colored_mask(image, predict_image))\naxs[3, 1].set_title(\"Gold Std.\")\naxs[3, 1].imshow(add_colored_mask(image, mask_image))\naxs[3, 2].set_title(\"Diff.\")\naxs[3, 2].imshow(diff_mask(mask_image, predict_image))","metadata":{"_uuid":"3b1f04bd591e2dcfa01bfc239a52d574a75ca42b","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#!tar zcf results.tgz --directory=../input/segmentation/test .\n!tar zcf mascaras_covid.tgz --directory=/kaggle/working/segmentation/test_auf .\n#glob(\"./segmentation/test_auf/*.*\")\n","metadata":{"_uuid":"fc6cc9565c3f0df279dd1a79ba91c0b61d9e29fd","trusted":true},"execution_count":null,"outputs":[]}]}