{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<center><img src='https://miro.medium.com/max/875/1*2hZsom9OR2luUM1nGnOQQg.png'></center>","metadata":{}},{"cell_type":"markdown","source":"[source](https://medium.com/analytics-vidhya/tpu-training-made-easy-with-colab-3b73b920878f)\n\n# <center style=\"font-family:Segoe UI; font-size: 1.4em\">🧠TRAINING WITH TF-KERAS ON TPU USING TFRECORDS🧠</center>","metadata":{}},{"cell_type":"markdown","source":"# Table of contents <a id='0.1'></a>\n1. [Version Note](#1)\n2. [Introduction](#2)\n3. [Import Packages](#3)\n4. [Configuration](#4)\n  * 4.1 [Hardware Configuration (TPU/GPU/CPU)](#4.1)\n  * 4.1 [Weights and Biases Configuration](#4.2)\n5. [Utility](#5)\n  * 5.1 [Augmentation Utilities](#5.1)\n  * 5.2 [Data Utilities](#5.2)\n6. [Data Preprocessing](#6)\n  * 6.1 [Load Data](#6.1)\n  * 6.2 [Data Augmentation](#6.2)\n6. [Metric: Dice Coefficient](#7)\n7. [Loss Function](#8)\n8. [Model](#9)\n9. [Callbacks](#10)\n10. [Training](#11)\n11. [Reference](#12)","metadata":{}},{"cell_type":"markdown","source":"# 1. <a id='1'>Version Notes</a>\n[Table of contents](#0.1)\n\n* Version 1\n   * Using tfrecords to train the model 512x512.\n   * Logging results using Weights and Biases.\n   * Training on 5 folds (Kfold).\n   * Dice loss.\n   * Adam Optimizer with LR = 5e-4.\n   * Advance augmentations using TPU.\n* Version 3\n   * Dice loss.\n   * Adam Optimizer with LR = 5e-4.\n   * Training for 5 folds.\n* Version 4\n   * Training on remaining folds. (3)\n* Version 5\n   * Training on remaining folds. (4-5)\n* Version 6:\n   * Training on updated data and augs.\n   * Updated augmentations.\n   * Training for 8 folds. \n   * Using Reduce on plateau.\n* Version 7:\n   * Using bce_jaccard_loss.\n* Version 8:\n   * Using cosine annealing scheduler\n* Version 9:\n   * Training fold 0 for 80 epochs.\n* Version 10:\n   * Added central crop and pixel level augmentations.","metadata":{}},{"cell_type":"markdown","source":"# 2. <a id='2'>Introduction</a>\n[Table of contents](#0.1)\n\n<span style=\"font-family: Segoe UI; font-size: 1.2em;\">The objective of this notebook is to train model using TPU and tfrecords. We will use the [tf.data](https://www.tensorflow.org/api_docs/python/tf/data) API in order to build our data pipeline. I am using data provided by [Wojtek Rosa](https://www.kaggle.com/wrrosa) in [this](https://www.kaggle.com/wrrosa/hubmap-tf-with-tpu-efficientunet-512x512-train) notebook. For extensive EDA please check [here](https://www.kaggle.com/kool777/hubmap-extensive-eda).</span>\n\n<span style=\"font-family: Segoe UI; font-size: 1.2em;\">I am using Weights and Biases to log my result. I have included some step just in case is anyone wants to start with it.</span>","metadata":{}},{"cell_type":"markdown","source":"# 3. <a id='3'>Import Packages</a>\n[Table of contents](#0.1)\n\nFirst we need to install [Pavel Yakubovskiy](https://github.com/qubvel) amazing library [segmentation_models](https://github.com/qubvel/segmentation_models). This will make experimentation real quick for us. We are also going to install [Weights & Biases](https://www.wandb.com/) in order to log our results. This will help us to interactively visualize our training results.","metadata":{}},{"cell_type":"code","source":"!pip install -q --upgrade wandb\n!pip install git+https://github.com/qubvel/segmentation_models","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Segmentation models seems to have problem with latest tf and keras you need to run below code cell to fix the error.","metadata":{}},{"cell_type":"code","source":"%env SM_FRAMEWORK=tf.keras","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# basic\nimport numba\nimport warnings\nimport time, math\nimport numpy as np\nimport pandas as pd\nfrom glob import glob\nimport sys, os, gc, cv2, re\nfrom pathlib import Path\nfrom functools import partial\nfrom tqdm.notebook import tqdm\nfrom kaggle_datasets import KaggleDatasets\n\n# image preprocessing \nimport rasterio\nimport tifffile as tiff\nfrom rasterio.windows import Window\nfrom IPython.display import Image\n\n# visulization\nimport matplotlib.pyplot as plt\n\n# deep learning\nimport tensorflow as tf\nimport segmentation_models as sm\nfrom tensorflow.keras.layers import *\nfrom tensorflow.keras import backend as K\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow_addons.optimizers import Lookahead\nfrom tensorflow.keras.models import Model, load_model\nfrom tensorflow.keras.utils import get_custom_objects\nfrom tensorflow.keras.losses import binary_crossentropy\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau, Callback, LearningRateScheduler\n\n# cross validation\nfrom sklearn.model_selection import KFold\n\n# logging\nimport wandb\nfrom wandb.keras import WandbCallback\nfrom kaggle_secrets import UserSecretsClient\n\nwarnings.filterwarnings('ignore')\n\nprint(f'Tensorflow Version: {tf.__version__}')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ROOT = '../input/hubmap-kidney-segmentation/'\nos.listdir(ROOT)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here I am using **Kaggle's Add-ons** to hide my secret wandb login key. You can create your secret key by first initializing your wandb project after creating your account on Weights and Biases. Don't worry you will see ahead how. For amazing step-by-step guide please refer [here](https://www.kaggle.com/imeintanis/cnn-track-your-experiments-weights-biases/notebook).\n    \n* Inside your notebook workspace on top header you will see options Click on **Add-ons**.\n\n* Now click on **secrets**.\n\n* When you'll run this line *wandb.init(project=\"project-folder-on-W&B\", name= 'project_name')* in code cell in upcoming section ahead in this notebook. After running this line you will see a link in output. You have to click on it, copy the key and paste it in the **Value** section inside **secret**.\n\n* See the below image to get some idea.","metadata":{}},{"cell_type":"code","source":"Image('../input/lyftl5googlecolab/wandb.png')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Now copy the code as given in the image or check the cell code cell below to use the login key. This way you can hide your credentials.","metadata":{}},{"cell_type":"code","source":"user_secrets = UserSecretsClient()\nsecret_value = user_secrets.get_secret(\"WANDB_KEY\")","metadata":{"_kg_hide-output":true,"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Run the below cell to append your wandb API key.","metadata":{}},{"cell_type":"code","source":"!wandb login $secret_value","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. <a id='4'>Configuration</a>\n## 4.1 <a id='4.1'>Hardware Configuration (TPU/GPU/CPU)</a>\n[Table of contents](#0.1)\n \nHere we are configurinhg are hardware accelerator which in this case is TPU.","metadata":{}},{"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Connecting to tpu...')\n    print('device running at:', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    print('Initializing TPU...')\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\n    print(\"TPU initialized\")\nelse:\n    print('Using deafualt strategy...')\n    strategy = tf.distribute.get_strategy()\n\nREPLICAS = strategy.num_replicas_in_sync\nprint(f\"REPLICAS:  {REPLICAS}\")\n\n# dynamically tune the data to be processed in TPU\nAUTOTUNE = tf.data.experimental.AUTOTUNE","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we are defining are hyperparameters which we will track using Weights and Biases.","metadata":{}},{"cell_type":"markdown","source":"## 4.2 <a id='4.2'>Weights and Biases Configuration</a>\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"code","source":"Params = dict(\n    DEVICE = 'tpu',\n    RUN = '5_(1)',                 # Successful version number to be tracked by W&B\n    SEED = 0,                      # seed for reproducibility\n    MIXED_PRECISION = True,        # enable/disable mixed precision\n    BATCH_SIZE = 16 * REPLICAS,    # batch size\n    IMAGE_DIM = 512,               # image dimension\n    ARCHITECTURE = 'Unet',         # model architecture\n    ENCODER = 'efficientnetb4',    # segmentation encoder\n    WEIGHT = 'imagenet',           # imagenet weights\n    VERBOSE = 1,                   # interactive/silent training\n    DISPLAY_PLOT = True,           # display plot at end of each fold training\n    EPOCHS = 85,                   # epoch\n    LR = 5e-4,                     # learning rate\n    FOLDS = 8,                     # number of folds\n)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DIM = Params['IMAGE_DIM']      # image dimension\nRUN = Params['RUN']            # successful wandb run number\nARCHI = Params['ARCHITECTURE'] # architecture\nMODEL = Params['ENCODER']      # segmentation encoder ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally I have initialized my run. Please keep in mind that each run is single execution of the training script. After running below cell you will get some links in output including link to your project page which you need to create first inside your W&B profile.\n\nHere you can see -\n* projects -- your project directory at your W&B profile.\n* name -- name of every run (single training script/notebook execution). You can keep to default depends on your choice.\n* config -- save all your hyperparameters in a config object.","metadata":{}},{"cell_type":"code","source":"wandb.init(project=\"hubmap-hacking-the-kidney\",\n           name= f'TPU-Tfrecord-{DIM}-{ARCHI}-{MODEL}-V{RUN}',\n           config=Params)\n\nconfig = wandb.config","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"config.keys()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. <a id='5'>Utility</a>\n\nWe will make some utility function which we will use in our data pipeline. We will try tensorflow best practices to optimize the data pipeline. We will also use [tf.image](https://www.tensorflow.org/api_docs/python/tf/image) API for data augmentation with TPU. We will perform augmentations using GPU/TPU using tf.data API. Please refer to [Data Augmentation](#6.2) section for more information.\n\n## 5.1 <a id='5.1'>Augmentation Utilities</a>\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"markdown","source":"## Coarse Dropout","metadata":{}},{"cell_type":"code","source":"def make_mask(num_holes,side_length,rows, cols, num_channels):\n    \n    \"\"\"Builds the mask for all sprinkles.\"\"\"\n    \n    row_range = tf.tile(tf.range(rows)[..., tf.newaxis], [1, num_holes])\n    col_range = tf.tile(tf.range(cols)[..., tf.newaxis], [1, num_holes])\n    r_idx = tf.random.uniform([num_holes], minval=0, maxval=rows-1,\n                              dtype=tf.int32)\n    c_idx = tf.random.uniform([num_holes], minval=0, maxval=cols-1,\n                              dtype=tf.int32)\n    r1 = tf.clip_by_value(r_idx - side_length // 2, 0, rows)\n    r2 = tf.clip_by_value(r_idx + side_length // 2, 0, rows)\n    c1 = tf.clip_by_value(c_idx - side_length // 2, 0, cols)\n    c2 = tf.clip_by_value(c_idx + side_length // 2, 0, cols)\n    row_mask = (row_range > r1) & (row_range < r2)\n    col_mask = (col_range > c1) & (col_range < c2)\n\n    # Combine masks into one layer and duplicate over channels.\n    mask = row_mask[:, tf.newaxis] & col_mask\n    mask = tf.reduce_any(mask, axis=-1)\n    mask = mask[..., tf.newaxis]\n    mask = tf.tile(mask, [1, 1, num_channels])\n    return mask\n    \ndef sprinkles(image): \n    num_holes = 5\n    side_length = int(.1 * 512)\n    mode = 'normal'\n    PROBABILITY = 1\n    \n    RandProb = tf.cast( tf.random.uniform([],0,1) < PROBABILITY, tf.int32)\n    if (RandProb == 0)|(num_holes == 0): return image\n    \n    img_shape = tf.shape(image)\n    if mode is 'normal':\n        rejected = tf.zeros_like(image)\n    elif mode is 'salt_pepper':\n        num_holes = num_holes // 2\n        rejected_high = tf.ones_like(image)\n        rejected_low = tf.zeros_like(image)\n    elif mode is 'gaussian':\n        rejected = tf.random.normal(img_shape, dtype=tf.float32)\n    else:\n        raise ValueError(f'Unknown mode \"{mode}\" given.')\n        \n    rows = img_shape[0]\n    cols = img_shape[1]\n    num_channels = img_shape[-1]\n    if mode is 'salt_pepper':\n        mask1 = make_mask(num_holes,side_length,rows, cols, num_channels)\n        mask2 = make_mask(num_holes,side_length,rows, cols, num_channels)\n        filtered_image = tf.where(mask1, rejected_high, image)\n        filtered_image = tf.where(mask2, rejected_low, filtered_image)\n    else:\n        mask = make_mask(num_holes,side_length,rows, cols, num_channels)\n        filtered_image = tf.where(mask, rejected, image)\n    return filtered_image","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Shear Transformation\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"code","source":"def transform_shear(image, height, shear, mask=False):\n    \n    '''\n    shear augmentation on image\n    and mask.\n    --------------------------------\n    \n    Arguments:\n    image -- input image\n    mask -- input mask\n    \n    Return:\n    image -- augmented image \n    mask -- augmented mask\n    '''\n    \n    DIM = height\n    XDIM = DIM%2 #fix for size 331\n    \n    shear = shear * tf.random.uniform([1],dtype='float32')\n    shear = math.pi * shear / 180.\n        \n    # SHEAR MATRIX\n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)\n    shear_matrix = tf.reshape(tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3])    \n\n    # LIST DESTINATION PIXEL INDICES\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(shear_matrix,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES \n    idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image, tf.transpose(idx3))\n        \n    if mask:\n        return tf.reshape(d, [DIM,DIM,1])\n    \n    return tf.reshape(d, [DIM,DIM,3])","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Shift Transform\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"code","source":"def transform_shift(image, height, h_shift, w_shift, mask=False):\n    \n    '''\n    shift augmentation on image\n    and mask.\n    --------------------------------\n    \n    Arguments:\n    image -- input image\n    mask -- input mask\n    \n    Return:\n    image -- augmented image \n    mask -- augmented mask\n    '''\n    \n    DIM = height\n    XDIM = DIM%2 #fix for size 331\n    \n    height_shift = h_shift * tf.random.uniform([1],dtype='float32') \n    width_shift = w_shift * tf.random.uniform([1],dtype='float32') \n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n        \n    # SHIFT MATRIX\n    shift_matrix = tf.reshape(tf.concat([one,zero,height_shift, zero,one,width_shift, zero,zero,one],axis=0),[3,3])\n\n    # LIST DESTINATION PIXEL INDICES\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(shift_matrix,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES \n    idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image, tf.transpose(idx3))\n        \n    if mask:\n        return tf.reshape(d, [DIM,DIM,1])\n    \n    return tf.reshape(d, [DIM,DIM,3])","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Rotation Transform\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"code","source":"def transform_rotation(image, height, rotation, mask=False):\n    \n    '''\n    rotation augmentation on image\n    and mask.\n    --------------------------------\n    \n    Arguments:\n    image -- input image\n    mask -- input mask\n    \n    Return:\n    image -- augmented image \n    mask -- augmented mask\n    '''\n    \n    DIM = height\n    XDIM = DIM%2 #fix for size 331\n    rotation = rotation * tf.random.uniform([1],dtype='float32')\n    \n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    \n    # ROTATION MATRIX\n    c1 = tf.math.cos(rotation)\n    s1 = tf.math.sin(rotation)\n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    rotation_matrix = tf.reshape(tf.concat([c1,s1,zero, -s1,c1,zero, zero,zero,one],axis=0),[3,3])\n\n    # LIST DESTINATION PIXEL INDICES\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(rotation_matrix,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES \n    idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image, tf.transpose(idx3))\n    \n    if mask:\n        return tf.reshape(d, [DIM,DIM,1])\n    \n    return tf.reshape(d, [DIM,DIM,3])","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def augmentations(image, mask):\n    \n    '''\n    Apply different augmentations on \n    image and mask.\n    --------------------------------\n    \n    Arguments:\n    image -- input image\n    mask -- input mask\n    \n    Return:\n    image -- augmented image \n    mask -- augmented mask\n    '''\n    \n    spatial = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    transpose = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    shear = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    shift = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    central_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    drop_coarse = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    pixel = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    \n    # SPATIAL-LEVEL TRANSFORMATIONS\n    ## FLIP LEFT-RIGHT\n    if spatial >= .6:\n        image = tf.image.flip_left_right(image)\n        mask = tf.image.flip_left_right(mask)\n    \n    ## FLIP UP-DOWN\n    if spatial >= .3:   \n        image = tf.image.flip_up_down(image)\n        mask = tf.image.flip_up_down(mask)\n        \n    ## ROTATIONS\n    if rotate > .75:\n        image = tf.image.rot90(image, k=3) # rotate 270º\n        mask = tf.image.rot90(mask, k=3) # rotate 270º\n    elif rotate > .5:\n        image = tf.image.rot90(image, k=2) # rotate 180º\n        mask = tf.image.rot90(mask, k=2) # rotate 180º\n    elif rotate > .25:\n        image = tf.image.rot90(image, k=1) # rotate 90º\n        mask = tf.image.rot90(mask, k=1) # rotate 90º\n        \n    ## TRANSPOSE\n    if transpose >= .3:\n        image = tf.image.transpose(image)\n        mask = tf.image.transpose(mask)\n    \n#     ## SHEAR \n#     if shear >= .3:\n#         image = transform_shear(image, height=IMAGE_DIM, shear=20.)\n#         mask = transform_shear(mask, height=IMAGE_DIM, shear=20., mask=True)\n    \n#     ## SHIFT\n#     if shift >= .3:\n#         image = transform_shift(image, height=IMAGE_DIM, h_shift=15., w_shift=15.)\n#         mask = transform_shift(mask, height=IMAGE_DIM, h_shift=15., w_shift=15., mask=True)\n\n    ## CROP\n    if central_crop >= .35:\n        image = tf.image.central_crop(image, 0.7)\n        image = tf.image.resize(image, size=[DIM, DIM])\n        mask = tf.image.central_crop(mask, 0.7)\n        mask = tf.image.resize(mask, size=[DIM, DIM])\n        \n    ## COARSE-DROPOUT\n    if drop_coarse >= .4:\n        image = sprinkles(image)\n        mask = sprinkles(mask)\n    \n    # PIXEL-LEVEL TRANSFORMATION\n    if pixel >= .2:\n        \n        if pixel >= .7:\n            image = tf.image.adjust_brightness(image, 0.4)\n        elif pixel >= .5:\n            image = tf.image.adjust_contrast(image, 0.4)\n        elif pixel >= .4:\n            image = tf.image.random_saturation(image, 0.7, 1.3)\n        \n    return image, mask","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.2 <a id='5.2'>Data Utilities</a>\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"code","source":"def decode_image(image):\n    \n    '''\n    decode and normalize image.\n    --------------------------------\n    \n    Arguments:\n    image -- input image (str)\n    \n    Return:\n    image -- normalized image\n    '''\n    image = tf.io.decode_raw(image, out_type = np.dtype('uint8'))\n    image = tf.image.convert_image_dtype(image, tf.float32) \n    image = tf.reshape(image, (config.IMAGE_DIM, config.IMAGE_DIM, 3))              \n    return image\n\ndef decode_mask(mask):\n    \n    '''\n    decode and normalize mask.\n    --------------------------------\n    \n    Arguments:\n    mask -- input mask (str)\n    \n    Return:\n    mask -- normalized mask\n    '''\n\n    mask = tf.io.decode_raw(mask, out_type = 'bool')\n    mask = tf.cast(mask, tf.float32)                        \n    mask = tf.reshape(mask, (config.IMAGE_DIM, config.IMAGE_DIM, 1))                \n    return mask\n\ndef read_labeled_tfrecord(example):\n    \n    '''\n    prepare image and mask\n    from tfrecord\n    -----------------------\n    \n    Arguments:\n    example -- tfrecord file.\n    \n    Return:\n    image -- normalized image\n    mask -- normalized mask\n    '''\n    tfrecord_format = {\n        'image': tf.io.FixedLenFeature([], tf.string),\n        'mask': tf.io.FixedLenFeature([], tf.string)\n    }\n    \n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    mask = decode_mask(example['mask'])\n    return image, mask\n\ndef generate_data(filenames, ordered=False, repeat=False, shuffle=False, augment=False):\n    \n    '''\n    generate batches of tf.Dataset\n    object\n    --------------------------------\n    \n    Arguments:\n    tiff -- tf.data.Dataset object (tf.Tensor)\n    mask -- tf.data.Dataset object (tf.Tensor)\n    batch_size -- batches of image, mask pair\n    shuffle -- shuffle data \n    augment -- apply augmentations\n    \n    Return:\n    ds - tf.data.Dataset dataset \n    '''\n    \n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False \n        \n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTOTUNE) \n    dataset = dataset.with_options(ignore_order) \n    dataset = dataset.map(read_labeled_tfrecord, num_parallel_calls=AUTOTUNE) \n    \n    if augment:\n        dataset = dataset.map(augmentations, num_parallel_calls=AUTOTUNE)\n        \n    if repeat:\n        dataset = dataset.repeat() \n        \n    if shuffle:\n        dataset = dataset.shuffle(1000)\n        \n    dataset = dataset.batch(config.BATCH_SIZE, drop_remainder=True)\n    dataset = dataset.prefetch(AUTOTUNE)\n    return dataset\n\ndef plot(image, mask):\n    \n    '''\n    plot image and mask\n    ---------------------\n    \n    Arguments:\n    image -- tiff image \n    mask -- segmentation mask\n    \n    Returns:\n    matplotlib plot\n    '''\n    plt.figure(figsize=(15, 15))\n\n    # Image\n    plt.subplot(1, 3, 1)\n    plt.imshow(image)\n    plt.title(\"Image\", fontsize=16)\n\n    # Mask\n    plt.subplot(1, 3, 2)\n    plt.imshow(np.squeeze(mask))\n    plt.title(\"Image Mask\", fontsize=16)\n\n    # Image + Mask\n    plt.subplot(1, 3, 3)\n    plt.imshow(image)\n    plt.imshow(np.squeeze(mask), alpha=0.5)\n    plt.title(\"Image + Mask\", fontsize=16);\n\ndef count_data_items(filenames):\n    \n    '''\n    count total number of\n    files in tfrecords\n    --------------------\n    \n    Arguments:\n    filenames -- tfrecord filename\n    \n    Returns:\n    np.sum(n) -- total number of files in tfrecords.\n    '''\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. <a id='6'>Data Preprocessing🔬</a>\n## 6.1 <a id='6.1'>Load Data</a>\n[Table of contents](#0.1)\n\nWe will write our data pipeline using [tf.data](https://www.tensorflow.org/tutorials/load_data/images#using_tfdata_for_finer_control) API. Please check the [utility](#3) section for implemented utilities.","metadata":{}},{"cell_type":"code","source":"# appending GCS PATH\nGCS_PATH = KaggleDatasets().get_gcs_path('hubmap-tfrecords-1024-512')\n\n# 512x512 tfrecords\nDATA = tf.io.gfile.glob(str(GCS_PATH + '/train/*.tfrec'))\nDATA2 = tf.io.gfile.glob(str(GCS_PATH + '/train2/*.tfrec'))\nALL_DATA = DATA + DATA2\n\n# train csv file\ntrain = pd.read_csv(os.path.join(ROOT, 'train.csv'))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'Total number of files: {count_data_items(DATA)+count_data_items(DATA2)}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ds = generate_data(DATA, ordered=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for image, mask in train_ds.take(1):\n    image_batch, mask_batch = image, mask\n    print(\"Image shape: \", image_batch.numpy().shape)\n    print(\"Mask shape: \", mask_batch.numpy().shape)\n    \ndel train_ds","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,16))\nfor i,(img,mask) in enumerate(zip(image_batch[:64], mask_batch[:64])):\n    plt.subplot(8,8,i+1)\n    plt.imshow(img,vmin=0,vmax=255)\n    plt.imshow(np.squeeze(mask), alpha=0.4)\n    plt.axis('off')\n    plt.subplots_adjust(wspace=None, hspace=None)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6.2 <a id='6.2'>Data Augmentation</a>\n[Table of contents](#0.1)\n\nWe will perform augmentations using GPU/TPU using tf.data API. For more information check this amazing notebook by [Chris Deotte](https://www.kaggle.com/cdeotte) and [Dimitre Oliveira](https://www.kaggle.com/dimitreoliveira). The notebooks can be found [here](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96#Data-Augmentation-using-GPU/TPU-for-Maximum-Speed!) and [here](https://www.kaggle.com/dimitreoliveira/flower-with-tpus-advanced-augmentations#Advanced-augmentations).","metadata":{}},{"cell_type":"code","source":"train_ds = generate_data(DATA, ordered=True, augment=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for image, mask in train_ds.take(1):\n    image_batch, mask_batch = image, mask\n    print(\"Image shape: \", image_batch.numpy().shape)\n    print(\"Mask shape: \", mask_batch.numpy().shape)\n    \ndel train_ds","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,16))\nfor i,(img,mask) in enumerate(zip(image_batch[:64], mask_batch[:64])):\n    plt.subplot(8,8,i+1)\n    plt.imshow(img,vmin=0,vmax=255)\n    plt.imshow(np.squeeze(mask), alpha=0.4)\n    plt.axis('off')\n    plt.subplots_adjust(wspace=None, hspace=None)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 7. <a id='7'>Evaluation Metric</a>\n[Table of contents](#0.1)\n \n\n## 7.1 <a id='7.1'>Intersection-Over-Union (IoU)</a>\n\n<span style=\"font-family: Segoe UI; font-size: 1.2em;\">The Intersection-Over-Union (IoU), also known as the Jaccard Index, is one of the most commonly used metrics in semantic segmentation.</span>\n\n<center><img src=\"https://miro.medium.com/max/375/0*kraYHnYpoJOhaMzq.png\"></center>\n<br>\n\n<span style=\"font-family: Segoe UI; font-size: 1.2em;\">IoU is the area of overlap between the predicted and the ground truth segmentation divided by the area of union between the predicted and the ground truth segmentation respectively. The IoU ranges from **0 (imperfect match) to 1 (perfect match)**.\n\n<span style=\"font-family: Segoe UI; font-size: 1.2em;\">The IoU or Jaccard Index is given as follows - </span>\n\n$$\\text{J}(A, B) = \\frac{|A \\cap B|}{|A| + |B|}.$$\n\nHere,\n* A = predicted mask.\n* B = ground truth mask.\n    \n<span style=\"font-family: Segoe UI; font-size: 1.2em;\">The IoU is usually used during evaluation and can not be used as loss function while training the model. The choice of loss while training is Dice Loss.</span>","metadata":{}},{"cell_type":"code","source":"def IoUCoefficient(y_true, y_pred, epsilon = 1.):\n\n    '''\n    Dice Coefficient in Tensorflow\n    ------------------------------\n\n    Arguments: \n    y_true (Tensorflow tensor) -- tensor of ground truth values.\n    y_pred (Tensorflow tensor) -- tensor of predicted values.\n    epsilon -- constant to avoid divide by 0 errors.\n\n    Returns:\n    IoU_coefficient\n    '''\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = K.sum(K.abs(y_true_f * y_pred_f))\n    union = K.sum(y_true_f)  + K.sum(y_pred_f) - intersection\n    return (K.mean((intersection + epsilon) / (union + epsilon)))","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 7.2 <a id='7.2'>Dice Coefficient🎲</a>\n[Table of contents](#0.1)\n\n\n<span style=\"font-family: Segoe UI; font-size: 1.2em;\">Dice similarity coefficient is ideal for segmentation tasks. It is measure of how well two contours overlap. The dice index ranges from **0 (imperfect match) to 1 (perfect match)**.</span>\n\n<center><img src=\"https://miro.medium.com/max/536/1*yUd5ckecHjWZf6hGrdlwzA.png\"></center>\n\n<span style=\"font-family: Segoe UI; font-size: 1.2em;\">The dice coefficient is givern as follows -</span>\n\n$$\\text{DSC}(A, B) = \\frac{2 \\times |A \\cap B|}{|A| + |B|}.$$\n\nHere,\n* A = predicted mask.\n* B = ground truth mask.\n\n$$\\text{DSC}(f, x, y) = \\frac{2 \\times \\sum_{i, j} f(x)_{ij} \\times y_{ij} + \\epsilon}{\\sum_{i,j} f(x)_{ij} + \\sum_{i, j} y_{ij} + \\epsilon}$$\n\nHere,\n* x = input image.\n* f(x) = predicted output mask by model.\n* y = ground truth mask.\n* epsilon = small number to avoid divide by zero.","metadata":{}},{"cell_type":"code","source":"# dice coefficient\ndef diceCoefficient(y_true, y_pred, epsilon = 1e-10):\n\n    '''\n    Dice Coefficient in Tensorflow\n    ------------------------------\n\n    Arguments: \n    y_true (Tensorflow tensor) -- tensor of ground truth values.\n    y_pred (Tensorflow tensor) -- tensor of predicted values.\n    epsilon -- constant to avoid divide by 0 errors.\n\n    Returns:\n    dice_coefficient\n    '''\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = K.sum(y_true_f * y_pred_f)\n    return (2. * intersection + epsilon) / (K.sum(y_true_f) + K.sum(y_pred_f) + epsilon)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 8. <a id='8'>Loss Functions\t🎲</a>\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"markdown","source":"## Dice Loss","metadata":{}},{"cell_type":"code","source":"# with strategy.scope():\ndef dice_loss(y_true, y_pred):\n    loss = 1 - diceCoefficient(y_true, y_pred)\n    return loss\n\nget_custom_objects().update({\"dice\": dice_loss})","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Tversky Loss","metadata":{}},{"cell_type":"code","source":"# tversky loss\ndef tversky(y_true, y_pred, alpha=0.7, beta=0.3, smooth=1):\n\n\n    '''\n    Tversky in Keras\n    ------------------------------\n\n    Arguments: \n    y_true (Tensorflow tensor) -- tensor of ground truth values.\n    y_pred (Tensorflow tensor) -- tensor of predicted values.\n    smooth -- constant to avoid divide by 0 errors.\n    alpha -- constant to control penalties for false positives. \n    beta -- constant to control penalties for false negatives.\n\n    Returns:\n    tversky loss\n    '''\n\n    y_true_pos = K.flatten(y_true)\n    y_pred_pos = K.flatten(y_pred)\n    true_pos = K.sum(y_true_pos * y_pred_pos)\n    false_neg = K.sum(y_true_pos * (1 - y_pred_pos))\n    false_pos = K.sum((1 - y_true_pos) * y_pred_pos)\n    return (true_pos + smooth) / (true_pos + alpha * false_neg + beta * false_pos + smooth)\n\n# tversky loss\ndef tversky_loss(y_true, y_pred):\n    return 1 - tversky(y_true, y_pred)\n\nget_custom_objects().update({\"tversky\": tversky_loss})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Focal Tversky Loss","metadata":{}},{"cell_type":"code","source":"# focal tversky loss\ndef focal_tversky_loss(y_true, y_pred, gamma=0.75):\n    tv = tversky(y_true, y_pred)\n    return K.pow((1 - tv), gamma)\n\nget_custom_objects().update({\"focal_tversky\": focal_tversky_loss})","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Lovasz Loss","metadata":{}},{"cell_type":"code","source":"# \"\"\"\n# Lovasz-Softmax and Jaccard hinge loss in Tensorflow\n# Maxim Berman 2018 ESAT-PSI KU Leuven (MIT License)\n# \"\"\"\ndef lovasz_loss(y_true, y_pred):\n    y_true, y_pred = K.cast(K.squeeze(y_true, -1), 'int32'), K.cast(K.squeeze(y_pred, -1), 'float32')\n    logits = K.log(y_pred / (1. - y_pred))\n    loss = lovasz_hinge(logits, y_true, per_image=True, ignore=None)\n    return loss\n\n\ndef lovasz_grad(gt_sorted):\n    \"\"\"\n    Computes gradient of the Lovasz extension w.r.t sorted errors\n    See Alg. 1 in paper\n    \"\"\"\n    gts = tf.reduce_sum(gt_sorted)\n    intersection = gts - tf.cumsum(gt_sorted)\n    union = gts + tf.cumsum(1. - gt_sorted)\n    jaccard = 1. - intersection / union\n    jaccard = tf.concat((jaccard[0:1], jaccard[1:] - jaccard[:-1]), 0)\n    return jaccard\n\n\ndef lovasz_hinge(logits, labels, per_image=True, ignore=None):\n    \"\"\"\n    Binary Lovasz hinge loss\n      logits: [B, H, W] Variable, logits at each pixel (between -\\infty and +\\infty)\n      labels: [B, H, W] Tensor, binary ground truth masks (0 or 1)\n      per_image: compute the loss per image instead of per batch\n      ignore: void class id\n    \"\"\"\n    if per_image:\n        def treat_image(log_lab):\n            log, lab = log_lab\n            log, lab = tf.expand_dims(log, 0), tf.expand_dims(lab, 0)\n            log, lab = flatten_binary_scores(log, lab, ignore)\n            return lovasz_hinge_flat(log, lab)\n\n        losses = tf.map_fn(treat_image, (logits, labels), dtype=tf.float32)\n\n        # Fixed python3\n        losses.set_shape((None,))\n\n        loss = tf.reduce_mean(losses)\n    else:\n        loss = lovasz_hinge_flat(*flatten_binary_scores(logits, labels, ignore))\n    return loss\n\n\ndef lovasz_hinge_flat(logits, labels):\n    \"\"\"\n    Binary Lovasz hinge loss\n      logits: [P] Variable, logits at each prediction (between -\\infty and +\\infty)\n      labels: [P] Tensor, binary ground truth labels (0 or 1)\n      ignore: label to ignore\n    \"\"\"\n\n    def compute_loss():\n        labelsf = tf.cast(labels, logits.dtype)\n        signs = 2. * labelsf - 1.\n        errors = 1. - logits * tf.stop_gradient(signs)\n        errors_sorted, perm = tf.nn.top_k(errors, k=tf.shape(errors)[0], name=\"descending_sort\")\n        gt_sorted = tf.gather(labelsf, perm)\n        grad = lovasz_grad(gt_sorted)\n        # loss = tf.tensordot(tf.nn.relu(errors_sorted), tf.stop_gradient(grad), 1, name=\"loss_non_void\")\n        # ELU + 1\n        loss = tf.tensordot(tf.nn.elu(errors_sorted) + 1., tf.stop_gradient(grad), 1, name=\"loss_non_void\")\n        return loss\n\n    # deal with the void prediction case (only void pixels)\n    loss = tf.cond(tf.equal(tf.shape(logits)[0], 0),\n                   lambda: tf.reduce_sum(logits) * 0.,\n                   compute_loss,\n#                    strict=True,\n                   name=\"loss\"\n                   )\n    return loss\n\ndef flatten_binary_scores(scores, labels, ignore=None):\n    \"\"\"\n    Flattens predictions in the batch (binary case)\n    Remove labels equal to 'ignore'\n    \"\"\"\n    scores = tf.reshape(scores, (-1,))\n    labels = tf.reshape(labels, (-1,))\n    if ignore is None:\n        return scores, labels\n    valid = tf.not_equal(labels, ignore)\n    vscores = tf.boolean_mask(scores, valid, name='valid_scores')\n    vlabels = tf.boolean_mask(labels, valid, name='valid_labels')\n    return vscores, vlabels\n\n# lovasz loss\ndef symmetric_lovasz(y_true, y_pred):\n    return 0.5*(lovasz_hinge(y_pred, y_true) + lovasz_hinge(-y_pred, 1.0 - y_true))\n\nget_custom_objects().update({\"lovasz\": symmetric_lovasz})","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## BCE Dice Loss","metadata":{}},{"cell_type":"code","source":"def bce_dice_loss(y_true, y_pred):\n    loss = 0.5*binary_crossentropy(y_true, y_pred) + 0.5*dice_loss(y_true, y_pred)\n    return loss\n\nget_custom_objects().update({\"bce_dice\": bce_dice_loss})","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## BCE Jaccard Loss","metadata":{}},{"cell_type":"code","source":"from segmentation_models.losses import bce_jaccard_loss","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## BCE","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.losses import BinaryCrossentropy","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 9. <a id='9'>Model🚀</a>\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"code","source":"# model = sm.Unet(config.ENCODER, encoder_weights=None)\n# model.summary()","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# del model\n# gc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 10. <a id='10'>Callbacks<a>  \n    \n[Table of contents](#0.1)","metadata":{}},{"cell_type":"markdown","source":"## Learning Rate Schedular","metadata":{}},{"cell_type":"code","source":"###############################\n#OneCycleLearningRateSchedular#\n###############################\n\nLR_START = 0.00001\nLR_MAX = 0.00005\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = .8\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n\nlr_step = LearningRateScheduler(lrfn, verbose=config.VERBOSE)\n\n##########################\n#CosineAnnealingScheduler#\n##########################\n\nclass CosineAnnealingScheduler(Callback):\n    \"\"\"Cosine annealing scheduler.\n    \"\"\"\n\n    def __init__(self, T_max, eta_max, eta_min=0, verbose=1):\n        super(CosineAnnealingScheduler, self).__init__()\n        self.T_max = T_max\n        self.eta_max = eta_max\n        self.eta_min = eta_min\n        self.verbose = verbose\n\n    def on_epoch_begin(self, epoch, logs=None):\n        if not hasattr(self.model.optimizer, 'lr'):\n            raise ValueError('Optimizer must have a \"lr\" attribute.')\n        lr = self.eta_min + (self.eta_max - self.eta_min) * (1 + math.cos(math.pi * epoch / self.T_max)) / 2\n        K.set_value(self.model.optimizer.lr, lr)\n        print('\\nEpoch %05d: CosineAnnealingScheduler setting learning ''rate to %s.' % (epoch + 1, lr))\n\n    def on_epoch_end(self, epoch, logs=None):\n        logs = logs or {}\n        logs['lr'] = K.get_value(self.model.optimizer.lr)\n\n\ncosine_annealer = CosineAnnealingScheduler(T_max=30, eta_max=0.0005, eta_min=0.0001, verbose=config.VERBOSE)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Early Stopping","metadata":{}},{"cell_type":"code","source":"early_stop = EarlyStopping(monitor='val_diceCoefficient', mode = 'max',\n                                              patience=10, restore_best_weights=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Reduce On Plateau","metadata":{}},{"cell_type":"code","source":"reduce = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=8, min_lr=0.00001)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## WandbCallback","metadata":{}},{"cell_type":"code","source":"# wandb = WandbCallback(monitor='val_diceCoefficient')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 11. <a id='11'>Training </a>\n[Table of contents](#0.1)\n\nCheck version notes for more information. You can also visualize the training results in notebook. We are training on 4 folds using advance augmentations. Inference notebook will be published soon. Check [here](https://wandb.ai/kool7/hubmap-hacking-the-kidney) for interactive results.","metadata":{}},{"cell_type":"code","source":"fold = KFold(n_splits=config.FOLDS, shuffle=True, random_state=config.SEED)\nfor fold,(idxT,idxV) in enumerate(fold.split(ALL_DATA)):\n    \n    if fold == 2:\n        \n        print('#'*16); print(f'#### FOLD {fold+1} ####'); print('#'*16)\n        print(f'Image Dim: {config.IMAGE_DIM}, Batch Size: {config.BATCH_SIZE}, Epochs: {config.EPOCHS}')\n\n        # CREATE TRAIN AND VALIDATION SUBSETS\n        TRAINING_FILENAMES = [ALL_DATA[fi] for fi in idxT]\n        VALIDATION_FILENAMES = [ALL_DATA[fi] for fi in idxV]\n        STEPS_PER_EPOCH = count_data_items(TRAINING_FILENAMES) // config.BATCH_SIZE\n\n        # BUILD MODEL\n        print('initializing model...')\n        K.clear_session()\n        with strategy.scope():   \n\n            model = sm.Unet(config.ENCODER, encoder_weights=config.WEIGHT)\n\n            model.compile(optimizer = Adam(lr = config.LR),\n                          loss = bce_jaccard_loss,\n                          metrics=[diceCoefficient])\n\n        # CALLBACKS\n        checkpoint = ModelCheckpoint(f'/kaggle/working/hubmap-tf-keras-{config.DEVICE}-fold-%i.h5'%fold,\n                                     verbose=config.VERBOSE,\n                                     monitor='val_diceCoefficient',\n                                     mode='max',\n                                     save_best_only=True)\n\n        print('Training Model...')\n        history = model.fit(\n            generate_data(TRAINING_FILENAMES, ordered=False, repeat=True, shuffle=True, augment=True),\n            epochs = config.EPOCHS,\n            steps_per_epoch = STEPS_PER_EPOCH,\n            callbacks = [checkpoint, cosine_annealer],\n            validation_data = generate_data(VALIDATION_FILENAMES, ordered=True),\n            verbose=config.VERBOSE\n        )\n\n        del model\n        gc.collect()\n\n        # PLOT TRAINING\n        # https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\n        if config.DISPLAY_PLOT:\n\n            plt.figure(figsize=(15,5))\n            TOTAL = np.arange(len(history.history['diceCoefficient']))\n            plt.plot(TOTAL, history.history['diceCoefficient'], '-o', label='Train diceCoefficient', color='#ff7f0e')\n            plt.plot(TOTAL, history.history['val_diceCoefficient'], '-o', label='Val diceCoefficient', color='#1f77b4')\n\n            x = np.argmax( history.history['val_diceCoefficient'] ); y = np.max( history.history['val_diceCoefficient'] )\n            xdist = plt.xlim()[1] - plt.xlim()[0]; ydist = plt.ylim()[1] - plt.ylim()[0]\n            plt.scatter(x,y,s=200,color='#1f77b4'); plt.text(x-0.03*xdist,y-0.13*ydist,'max diceCoefficient\\n%.2f'%y,size=14)\n\n            plt.ylabel('diceCoefficient',size=14); plt.xlabel('Epoch',size=14)\n            plt.legend(loc=2)\n\n            plt2 = plt.gca().twinx()\n\n            plt2.plot(TOTAL,history.history['loss'],'-o',label='Train Loss',color='#2ca02c')\n            plt2.plot(TOTAL,history.history['val_loss'],'-o',label='Val Loss',color='#d62728')\n\n            x = np.argmin( history.history['val_loss'] ); y = np.min( history.history['val_loss'] )\n            ydist = plt.ylim()[1] - plt.ylim()[0]\n            plt.scatter(x,y,s=200,color='#d62728'); plt.text(x-0.03*xdist,y+0.05*ydist,'min loss',size=14)\n\n            plt.ylabel('Loss',size=14)\n            plt.legend(loc=3)\n            plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 12. <a id='12'>Reference</a>\n[Table of contents](#0.1)\n* https://www.tensorflow.org/tutorials/load_data/images#using_tfdata_for_finer_control\n* https://www.tensorflow.org/api_docs/python/tf\n* [dice loss](https://www.kaggle.com/marcosnovaes/hubmap-3-unet-models-with-keras-cpu-gpu/notebook)\n* [Plot from chris deotte's notebook](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords)\n* https://www.kaggle.com/wrrosa/hubmap-tf-with-tpu-efficientunet-512x512-train/output","metadata":{}}]}