{"cells":[{"metadata":{},"cell_type":"markdown","source":"![](https://pbs.twimg.com/profile_banners/1102969378109304832/1598977165/1080x360)"},{"metadata":{},"cell_type":"markdown","source":"# <div align = 'center'> Simple Keras Pipeline </div>"},{"metadata":{},"cell_type":"markdown","source":"# Table of contents <a id='0.1'></a>\n\n1. [Introduction](#1)\n2. [Import Packages](#2)\n3. [Loading Data](#3)\n4. [Model](#4)\n5. [Training](#5)\n6. [Inference](#6)\n7. [Reference](#7)"},{"metadata":{},"cell_type":"markdown","source":"**Version Notes**:-\n* Version 8 : Used [Divam Gupta's implementation](https://github.com/divamgupta/image-segmentation-keras) of segmentation models in Keras.\n* Version 10 : \n   * Used [Qubvel's implementation](https://github.com/qubvel/segmentation_models) of segmentation models in Keras.\n   * Made the NB, completely offline.\n   * Added successful code for inferene.\n* Version 11 : Fixed two bugs. In the previous version, submission file has blank 'predicted' column, solved that by fixing these bugs.\n* Version 12 : Added Group K-Fold validation and augmentations.\n* Version 14 : Added checkpoint callback.\n* Version 15 : Using serexnet50, 60 epochs per fold."},{"metadata":{},"cell_type":"markdown","source":"# 1. <a id='1'>Introduction</a>\n\n**This notebook will show you how to**:\n* Install qubvel's '**segmentation_models**' library offline.\n* Do inference in the Keras Pipeline without having OOM (Out Of Memory) problem."},{"metadata":{},"cell_type":"markdown","source":"Let's start with installing segmentation_models libary offline."},{"metadata":{"trusted":true},"cell_type":"code","source":"# First, installing the dependencies\n!pip install -U ../input/kerasapplications/Keras_Applications-1.0.8-py3-none-any.whl\n!pip install ../input/qubvel/efficientnet-1.0.0-py3-none-any.whl\n!pip install ../input/qubvel/image_classifiers-1.0.0-py3-none-any.whl\n\n# Now, installing segmentation_models (short for 'sm')\n!pip install ../input/qubvel-segmentation-model-keras-v101/segmentation_models-master\n\n# sm can work with both Keras and Tensorflow.\n# By default, it look for keras.\n# But, with Keras, it's giving error during the import. \n# So, we will be using Tensorflow as the backend for sm.\n%env SM_FRAMEWORK=tf.keras","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. <a id='2'>Import Packages</a>\n[Table of contents](#0.1)"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os, gc\nimport numpy as np \nimport pandas as pd \nimport cv2\nimport glob\nimport numba\nimport pathlib\nfrom tqdm.notebook import tqdm\nimport tifffile as tiff\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import GroupKFold\n\nfrom albumentations import *\n\n# Segmentation\nimport segmentation_models as sm\n\n# For reading tiff images in parts \nimport rasterio\nfrom rasterio.windows import Window\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. <a id='3'>Loading Data</a>\n[Table of contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"Utilities (Hidden)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def get_settings(batch_size = 32, \n                 encoder = 'resnet34',\n                 epochs = 5):\n    \n    return {'BATCH_SIZE' : batch_size,\n           'ENCODER' : encoder,\n           'EPOCHS' : epochs}\n\n\ndef get_path(pathof):\n    '''\n    Returns mapping function based on the input argument\n    \n    Arguments:\n            pathof : value can be either 'image' or 'mask'\n    '''\n    \n    if pathof=='image':\n        return lambda fname: tf.strings.join([path_train, fname], \n                                             separator = '/')\n    \n    elif pathof=='mask':\n        return lambda fname: tf.strings.join([path_masks, fname], \n                                              separator = '/') \n\ndef get_normalised_tensor(path):\n    '''\n    Reads the image and scale it to [0, 1]\n    \n    Arguments:\n            path : path of the image to be read\n            \n    Returns:\n            Returns normalized image tensor\n    '''   \n    \n    im = tf.io.read_file(path)\n    im = tf.io.decode_png(im)\n    im = tf.image.convert_image_dtype(im, \n                                      tf.float32)\n    return im\n\ndef get_tensor(path): \n    '''\n    Reads the image or mask\n    \n    Arguments:\n            path : path of the image or mask to be read\n            \n    Returns:\n            Returns image or mask tensor\n    '''\n    \n    mask = tf.io.read_file(path)\n    mask = tf.io.decode_png(mask)\n    \n    return mask\n\n\n\n# This is how we can add albumentation transformation in tf.data pipeline\ndef augment_data(image, mask):\n    '''\n    This is a mapping function on a zipped dataset.\n    \n    It takes tf tensors, do numpy based albumentation transformations \n    and returns augmented tf tensors.\n    \n    Reference: https://albumentations.ai/docs/examples/tensorflow-example/\n    '''\n    \n    def _fn(image, mask):\n        sample = transforms(image = image, \n                            mask = mask)\n        aug_img = sample['image']\n        aug_msk = sample['mask']\n        \n        aug_img = tf.cast(aug_img/255.0, tf.float32)\n        \n        return aug_img, aug_msk\n    \n    \n    # tf.numpy_function: Wraps a python function and uses it as a TensorFlow op.\n    # [tf docs]\n    aug_img, aug_msk = tf.numpy_function(func = _fn, \n                                         inp = [image, mask],\n                                         Tout = [tf.float32, tf.uint8])\n    \n    return aug_img, aug_msk\n\n\ndef dice_coeff(y_true, y_pred, epsilon=1.):\n    \n    '''\n    Calculates dice coefficient\n\n    Arguments: \n            y_true : tensor of ground truth values.\n            y_pred : tensor of predicted values.\n            epsilon : constant to avoid divide by 0 errors.\n    \n    Returns:\n            dice_coefficient\n    '''\n    \n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = K.sum(y_true_f * y_pred_f)\n    score = (2. * intersection + epsilon) / (K.sum(y_true_f) + K.sum(y_pred_f) + epsilon)\n    return score\n\n\n# https://www.kaggle.com/leighplt/pytorch-fcn-resnet50\ndef make_grid(shape, window=256, min_overlap=32):\n    \"\"\"\n        Return Array of size (N,4), where N - number of tiles,\n        2nd axis represente slices: x1,x2,y1,y2 \n    \"\"\"\n    x, y = shape\n    nx = x // (window - min_overlap) + 1\n    x1 = np.linspace(0, x, num=nx, endpoint=False, dtype=np.int64)\n    x1[-1] = x - window\n    x2 = (x1 + window).clip(0, x)\n    ny = y // (window - min_overlap) + 1\n    y1 = np.linspace(0, y, num=ny, endpoint=False, dtype=np.int64)\n    y1[-1] = y - window\n    y2 = (y1 + window).clip(0, y)\n    slices = np.zeros((nx,ny, 4), dtype=np.int64)\n    \n    for i in range(nx):\n        for j in range(ny):\n            slices[i,j] = x1[i], x2[i], y1[j], y2[j]    \n    return slices.reshape(nx*ny,4)\n\n@numba.njit()\ndef rle_numba(pixels):\n    size = len(pixels)\n    points = []\n    if pixels[0] == 1: points.append(0)\n    flag = True\n    for i in range(1, size):\n        if pixels[i] != pixels[i-1]:\n            if flag:\n                points.append(i+1)\n                flag = False\n            else:\n                points.append(i+1 - points[-1])\n                flag = True\n    if pixels[-1] == 1: points.append(size-points[-1]+1)    \n    return points\n\ndef rle_numba_encode(image):\n    pixels = image.flatten(order = 'F')\n    points = rle_numba(pixels)\n    return ' '.join(str(x) for x in points)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"settings = get_settings(batch_size = 32,\n                        epochs = 60,\n                        encoder = 'seresnext50')\n\nDATA_ORIG_PATH = '../input/hubmap-kidney-segmentation'\nDATA_PATH = '../input/hubmap-256x256'\n\n# Using iafoss's data\npath_train = os.path.join(DATA_PATH, 'train')\npath_masks = os.path.join(DATA_PATH, 'masks')\n\npath_submission_file = os.path.join(DATA_ORIG_PATH, 'sample_submission.csv')\n\nprint(f'No. of training images: {len(os.listdir(path_train))}')\nprint(f'No. of masks: {len(os.listdir(path_masks))} \\n')\n\n\nprint(settings)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Images and its corresponding masks are saved with the same filename.\nfnames = np.array(os.listdir(path_train))\n\n# Each 256 size png images (tiles) belongs to one particular tiff image.\n# Here, we are calling this tiff image a group.\n# 'groups' list contains tiff ids corresponding to each png images.\ngroups = [fname[:9] for fname in fnames]\n\n\n# Creates folds by putting all the png images(tiles) of one tiff image in \n# one particular fold only. It will avoid the possible data leakage.\ngroup_kfold = GroupKFold(n_splits = 4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 'BUFFER_SIZE' is used by shuffle() method\n# Concept is pretty simple.\n# Please refer to this: \n# https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shuffle\nBUFFER_SIZE = 1000\n\n# We know this, right?\nBATCH_SIZE = settings['BATCH_SIZE']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. <a id='4'>Model</a>\n[Table of contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# We will be using pretrained resnet34 as our encoder.\n# Putting it at the location where tf model will look for.\n\n!mkdir -p ~/.keras/models\n\nif settings['ENCODER']=='resnet34':\n    !cp ../input/keras-pretrained-imagenet-weights/resnet34_imagenet_1000_no_top.h5 ~/.keras/models/resnet34_imagenet_1000_no_top.h5\nelif settings['ENCODER']=='resnet50':\n    !cp ../input/keras-pretrained-imagenet-weights/resnet50_imagenet_1000_no_top.h5 ~/.keras/models/resnet50_imagenet_1000_no_top.h5\nelif settings['ENCODER']=='seresnext50':\n    !cp ../input/keras-pretrained-imagenet-weights/seresnext50_imagenet_1000_no_top.h5 ~/.keras/models/seresnext50_imagenet_1000_no_top.h5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. <a id='5'>Training</a>\n[Table of contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/iafoss/hubmap-pytorch-fast-ai-starter\ntransforms = Compose([\n            HorizontalFlip(),\n            VerticalFlip(),\n            RandomRotate90(),\n            ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=15, p=0.9, \n                             border_mode=cv2.BORDER_REFLECT),\n            OneOf([\n                OpticalDistortion(p=0.3),\n                GridDistortion(p=.1),\n                IAAPiecewiseAffine(p=0.3),\n            ], p=0.3),\n            OneOf([\n                HueSaturationValue(10,15,10),\n                CLAHE(clip_limit=2),\n                RandomBrightnessContrast(),            \n            ], p=0.3),\n        ], p = 1.0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"for fold, (t_idx, v_idx) in enumerate(group_kfold.split(fnames, \n                                                        groups = groups)):\n    \n    print(f'Fold: {fold+1}')\n    \n    t_fnames_ds = tf.data.Dataset.from_tensor_slices(fnames[t_idx])\n    v_fnames_ds = tf.data.Dataset.from_tensor_slices(fnames[v_idx])\n\n    t_img_ds = t_fnames_ds.map(get_path('image')).map(get_tensor)\n    t_msk_ds = t_fnames_ds.map(get_path('mask')).map(get_tensor)\n    \n    v_img_ds = v_fnames_ds.map(get_path('image')).map(get_normalised_tensor)\n    v_msk_ds = v_fnames_ds.map(get_path('mask')).map(get_tensor)\n    \n\n    train_ds = tf.data.Dataset.zip((t_img_ds, t_msk_ds))\n    train_ds = train_ds.map(augment_data)  \n    \n    val_ds = tf.data.Dataset.zip((v_img_ds, v_msk_ds))\n\n    del t_fnames_ds, v_fnames_ds, t_img_ds, t_msk_ds, v_img_ds, v_msk_ds\n    gc.collect()\n\n    train_ds = train_ds.shuffle(BUFFER_SIZE)\\\n                       .batch(BATCH_SIZE)\\\n                       .repeat()\n    \n    val_ds = val_ds.batch(BATCH_SIZE)\\\n                   .repeat()\n    \n    \n    model = sm.Unet(settings['ENCODER'], \n                encoder_weights='imagenet')\n    \n    \n    \n    model.compile(optimizer = 'adam',\n                  loss = tf.keras.losses.BinaryCrossentropy(),\n                  metrics = [dice_coeff])\n    \n    checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(f'./saved_models/fold_model_{fold+1}.pb',\n                                                            save_best_only = True)\n    \n    EPOCHS = settings['EPOCHS']\n\n    TOTAL_TRAIN_SAMPLES = len(t_idx)\n    TOTAL_VAL_SAMPLES = len(t_idx)\n    \n    '''\n    We have to mention 'steps_per_epoch', because we are repeating\n    our datset for infinite times, so model will have no idea when to end \n    1 epoch, steps_per_epoch will tell the model, when to end.\n    '''\n    STEPS_PER_EPOCH = TOTAL_TRAIN_SAMPLES//BATCH_SIZE\n    \n    VALIDATION_STEPS = TOTAL_VAL_SAMPLES//BATCH_SIZE\n\n    history = model.fit(train_ds, \n                        epochs = EPOCHS,\n                        steps_per_epoch = STEPS_PER_EPOCH,\n                        validation_data = val_ds,\n                        validation_steps = VALIDATION_STEPS,\n                        callbacks = [checkpoint_callback])\n    \n#     trained_fold_models.append(model)\n    \n    del train_ds, val_ds, model\n    gc.collect()\n    \n    # Not running the last iteration\n    if fold==0:\n        break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6. <a id='6'>Inference</a>\n[Table of contents](#0.1)\n\nI have adopted leigh's code from his [notebook](https://www.kaggle.com/leighplt/pytorch-fcn-resnet50) which is based on PyTorch and modified it for our Keras pipeline.\n\nHe has used rasterio library which allows us to read and load only some part of the big image into the RAM, which solves OOM problem."},{"metadata":{"trusted":true},"cell_type":"code","source":"identity = rasterio.Affine(1, 0, 0, 0, 1, 0)\n\n# WINDOW is the size of the tile to be read by rasterio\nWINDOW = 1024\n\n# Tiles will have some overlap\nMIN_OVERLAP = 32\n\n# Tiles will be resized to NEW_SIZE, which is the size of the image\n# on which, we have trained our model.\nNEW_SIZE = 256","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ./saved_models","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fold_models = []\n\nfor fold_model_path in glob.glob(os.path.join('./saved_models/*.pb')):\n    \n    fold_models.append(tf.keras.models.load_model(fold_model_path, custom_objects={'dice_coeff': dice_coeff}))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"p = pathlib.Path(DATA_ORIG_PATH)\nsubm = {}\n\nfor i, filename in tqdm(enumerate(p.glob('test/*.tiff')), \n                        total = len(list(p.glob('test/*.tiff')))):\n    \n    print(f'{i+1} Predicting {filename.stem}')\n    \n    dataset = rasterio.open(filename.as_posix(), transform = identity)\n    slices = make_grid(dataset.shape, window=WINDOW, min_overlap=MIN_OVERLAP)\n    preds = np.zeros(dataset.shape, dtype=np.uint8)\n    \n    for (x1,x2,y1,y2) in slices:\n        image = dataset.read([1,2,3],\n                    window=Window.from_slices((x1,x2),(y1,y2)))\n        image = np.moveaxis(image, 0, -1)\n        \n        image = tf.image.convert_image_dtype(image, \n                                 tf.float32)\n        image = cv2.resize(image.numpy(), (NEW_SIZE, NEW_SIZE))\n        image = np.expand_dims(image, 0)\n        \n        pred = None\n        \n        for fold_model in fold_models:\n            if pred is None:\n                pred = np.squeeze(fold_model.predict(image))\n            else:\n                pred += np.squeeze(fold_model.predict(image))\n        \n        pred = pred/len(fold_models)\n        \n        pred = cv2.resize(pred, (WINDOW, WINDOW))\n        preds[x1:x2,y1:y2] = (pred > 0.5).astype(np.uint8)\n            \n    subm[i] = {'id':filename.stem, 'predicted': rle_numba_encode(preds)}\n    del preds\n    gc.collect();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame.from_dict(subm, orient='index')\nsubmission.to_csv('submission.csv', index=False)\n\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If you liked it, then please upvote it. :)"},{"metadata":{},"cell_type":"markdown","source":"# 7. <a id='7'>References</a>\n[Table of contents](#0.1)\n* https://www.tensorflow.org/api_docs/python/tf/data/Dataset \n* https://github.com/qubvel/segmentation_models\n* https://www.kaggle.com/leighplt/pytorch-fcn-resnet50"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}