"},{"metadata":{},"cell_type":"markdown","source":"# Table of contents \n\n1. [Version Notes](#1)\n1. [Summary](#2)\n2. [Introduction](#3)\n3. [Import Packages](#4)\n4. [Utility](#5)\n * 5.1 [Augmentation Utilities](#5.1)\n * 5.2 [Data Preprocessing Utilities](#5.2)\n * 5.3 [Training Utilities](#5.3)\n5. [Data Overview](#6)\n * 6.1 [Train Data](#6.1)\n * 6.2 [HuBMAP-Metadata](#6.2)\n * 6.3 [Image + Segmentation Mask](#6.3)\n * 6.3.1 [Train Tiff Images](#6.3.1)\n * 6.3.2 [Test Tiff Images](#6.3.2)\n6. [EDA](#7)\n * 7.1 [Individual Features](#7.1)\n * 7.2 [Multiple Features](#7.2)\n * 7.2.1 [Image Shape Distribution](#7.2.1)\n * 7.2.2 [Metadata Heatmap](#7.2.2)\n7. [Pandas MetaData Profiling](#8)\n8. [Tensorflow-Keras Training Pipeline](#9)\n * 8.1 [Configuration](#9.1)\n * 8.1.1 [Hardware Configuration (TPU/GPU/CPU)](#9.1.1)\n * 8.1.2 [Weights and Biases Configuration](#9.1.2)\n * 8.2 [Data Preprocessing](#9.2)\n * 8.3 [Visualize Train Examples](#9.3)\n * 8.4 [Metric: Dice Coefficient](#9.4)\n * 8.5 [Loss](#9.5)\n * 8.6 [Model: Unet](#9.6)\n * 8.7 [Callbacks](#9.7)\n * 8.8 [Training](#9.8)\n9. [Reference](#10)"},{"metadata":{},"cell_type":"markdown","source":"# 1. Version Notes\n[Table of contents](#0.1)\n\n* Version 29: Compelete EDA. \n* Version 31: Added training part using Keras-Tensorflow for training withour tfrecords on TPU. Using 4 folds. \n* Version 33: Added augmentations. Using tf.image for augmentation inside `decode_image_and_mask` function. \n* Version 37\n * Added logging abilities using weights and biases.\n * Using qubvel/segmentation models for training.\n * Added advance augmentations.\n * Training using FCN.\n * Using LR Scheduler for faster training. \n* Version 40: \n * Updated narration.\n * Using efficientnetb4 encoder.\n * Cosine Annealing Scheduler.\n * Added Coarse Dropout\n* Version 41:\n * Added BCE Dice Loss"},{"metadata":{},"cell_type":"markdown","source":"# 2. Summary\n[Table of contents](#0.1)\n\n* We are given kidney images (.tiff) (high res) and their corresponding masks either as RLE encoded or unencoded JSON files using which we need to develop segmentation model that identify glomeruli in the PAS stained microscopy data.\n* We have 8 train tiff images and 5 test tiff images.\n* **Private test set** is larger as compared to **public test set**.\n* A **Glomerulus** a tiny ball-shaped structure composed of capillary blood vessels actively involved in the filtration of the blood to form urine. The glomerulus is one of the key structures that make up the nephron, the functional unit of the kidney. [source](https://www.medicinenet.com/glomerulus/definition.htm)\n* **Functional Tissue Unit** is defined as a “three-dimensional block of cells centered around a capillary, such that each cell in this block is within diffusion distance from any other cell in the same block”. \n* Images are huge tiff files we need to subsequent data preprocessing. Magnificient notebook [here](https://www.kaggle.com/iafoss/256x256-images) will help."},{"metadata":{},"cell_type":"markdown","source":"# 3. Introduction📔\n[Table of contents](#0.1)\n\nWelcome to this new Kaggle competition. The [Human BioMolecular Atlas Program (HuBMAP)](https://hubmapconsortium.org/) is sponsored by The [National Institutes of Health (NIH)](https://www.nih.gov/). The primary task of HuBMAP is to catalyze the development of a framework for mapping the human body at a level of **glomeruli functional tissue units** for the first time in history. Hoping to become one of the world’s largest collaborative biological projects, HuBMAP aims to be an open map of the human body at the cellular level. **This competition, “Hacking the Kidney,\" starts by mapping the human kidney at single cell resolution.**\n\n\"**Your challenge is to detect functional tissue units (FTUs) across different tissue preparation pipelines.**\"\n\nSuccessful submissions will construct the tools, resources, and cell atlases needed to determine how the relationships between cells can affect the health of an individual.\n\n## What is HuBMAP?\n\nThe focus of HuBMAP is understanding the intrinsic intra-, inter-, and extra- cellular biomolecular distribution in human tissue. HuBMAP will focus on fresh, fixed, or frozen healthy human tissue using in situ and dissociative techniques that have high-spatial resolution.\n\nThe Human BioMolecular Atlas Program is a consortium composed of diverse research teams funded by the [Common Fund at the National Institutes of Health](https://commonfund.nih.gov/HuBMAP) . HuBMAP values secure, open sharing, and collaboration with other consortia and the wider research community.\n\nHuBMAP is developing the tools to create an open, global atlas of the human body at the cellular level. These tools and maps will be openly available, to accelerate understanding of the relationships between cell and tissue organization and function and human health."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from IPython.display import HTML\n\nHTML('
')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## What is FTU?\n\nAn FTU is defined as a “three-dimensional block of cells centered around a capillary, such that each cell in this block is within diffusion distance from any other cell in the same block” (de Bono, 2013). \n\nThe glomerulus (plural glomeruli) is a network of small blood vessels (capillaries) known as a tuft, located at the beginning of a nephron in the kidney. The tuft is structurally supported by the mesangium (the space between the blood vessels), composed of intraglomerular mesangial cells. The blood is filtered across the capillary walls of this tuft through the glomerular filtration barrier, which yields its filtrate of water and soluble substances to a cup-like sac known as Bowman's capsule. \n\n \n\n
\n\n\n
\n\n \n\n## Competition Goal (Brief)\n\n* The goal of this competition is the implementation of a successful and robust glomeruli FTU detector. Develop segmentation algorithms that identify **\"Glomerulus\"** in the PAS stained microscopy data. Detect functional tissue units (FTUs) across different tissue preparation pipelines.\n\n* For each image we are given annotations in separate JSON file and also the annotations are RLE encoded in train.csv.\n\n* We are segmenting **glomeruli FTU** in each image.\n\n* Since this is segmentation task our evaluation metric is Dice Coefficient. The Dice coefficient can be used to compare the pixel-wise agreement between a predicted segmentation and its corresponding ground truth.\n\n## About Competition Data\n\nThe data is huge **(24.5 GB)**. The HuBMAP data used in this hackathon includes 11 fresh frozen and 9 Formalin Fixed Paraffin Embedded (FFPE) PAS kidney images. Stained microscopy employs histological stains such as H&E or PAS to improve resolution and contrast for visualization of anatomical structures such as tubules or glomeruli. Glomeruli FTU annotations exist for all 20 tissue samples. Some of these will be shared for training, and others will be used to judge submissions.\n\n* The dataset is comprised of very large (>500MB - 5GB) TIFF files. \n* **\"The training set\"** has 8, and the public test set has 5 tiff files respectively. \n* **\"The private test set\"** is larger than the public test set.\n* The training set includes annotations in both RLE-encoded and unencoded (JSON) forms. The annotations denote segmentations of glomeruli.\n\n* **Both the training and public test sets also include anatomical structure segmentations. They are intended to help you identify the various parts of the tissue.**\n\nWe are provided with following files:\n\n* For each of the 11 training images we have been provided with a JSON file. Each JSON file has:\n * A type (Feature) and object type id (PathAnnotationObject). Note that these fields are the same between all files and do not offer signal.\n * A geometry containing a Polygon with coordinates for the feature's enclosing volume\n * Additional properties, including the name and color of the feature in the image.\n * The IsLocked field is the same across file types (locked for glomerulus, unlocked for anatomical structure) and is not signal-bearing.\n\n* train.csv contains the unique IDs for each image, as well as an RLE-encoded representation of the mask for the objects in the image. See the evaluation tab for details of the RLE encoding scheme. Note that we are also given annotations in JSON file for each image.\n\n* HuBMAP-20-dataset_information.csv contains additional information (including anonymized patient data) about each image.\n\n## What is RLE?\n\nRun-length encoding (RLE) is a form of lossless data compression in which runs of data (sequences in which the same data value occurs in many consecutive data elements) are stored as a single data value and count, rather than as the original run.\n\n## What we are prediciting?\n\nParticipants will develop segmentation algorithm that identify **\"glomeruli \"** in the PAS stained microscopy data. Detect functional tissue units (FTUs) across different tissue preparation pipelines. Participants are welcome to use other external data and/or pre-trained machine learning models in support of FTU segmentation. \n\n**We need to segment glomeruli in very large resolution Kidney images and annotations which are availabel as RLE encoded and as well as a JSON format.**\n\n## Evaluation Metric: Dice Coefficient\n\nDice Coefficient is common in case our task involve **segmentation**. The Dice coefficient can be used to compare the pixel-wise agreement between a predicted segmentation and its corresponding ground truth. the Dice similarity coefficient for two sets X and Y is defined as:\n\n$$\\text{DC}(X, Y) = \\frac{2 \\times |X \\cap Y|}{|X| + |Y|}.$$\n\nwhere X is the predicted set of pixels and Y is the ground truth."},{"metadata":{},"cell_type":"markdown","source":"# 4. Import Packages📚\n[Table of contents](#0.1)"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!pip install wandb -q --upgrade\n!pip install git+https://github.com/qubvel/segmentation_models","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"%env SM_FRAMEWORK=tf.keras","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# basic\nimport os\nimport cv2\nimport sys, gc\nimport warnings\nimport time, math\nimport numpy as np\nimport pandas as pd\nfrom glob import glob\nfrom pathlib import Path\nimport pandas_profiling as pp\nfrom tqdm.notebook import tqdm\n\n# visualize\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n \n# image preprocessing \nimport rasterio\nimport tifffile as tiff\nfrom rasterio.windows import Window\nfrom IPython.display import Image\n\n# kaggle datasets\nfrom kaggle_datasets import KaggleDatasets\n\n# deep learning\nimport tensorflow as tf\nimport segmentation_models as sm\nfrom tensorflow.keras.layers import *\nfrom tensorflow.keras import backend as K\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Model, load_model\nfrom tensorflow.keras.utils import get_custom_objects\nfrom tensorflow.keras.losses import binary_crossentropy\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau, Callback, LearningRateScheduler\n\n# cross validation\nfrom sklearn.model_selection import KFold\n\n# logging\nimport wandb\nfrom wandb.keras import WandbCallback\nfrom kaggle_secrets import UserSecretsClient\n\n%matplotlib inline\nwarnings.filterwarnings('ignore')\nprint(f'Wandb Version: {wandb.__version__}')\nprint(f'Seaborn Version: {sns.__version__}')\nprint(f'Tensorflow Version: {tf.__version__}')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# directory\nprint('Competition Data/Files')\nROOT = '../input/hubmap-kidney-segmentation/'\nos.listdir(ROOT)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Utility\n[Table of contents](#0.1)\n\nWe will make some utility function which we will use in our data pipeline. We will try tensorflow best practices to optimize the data pipeline. We will also use tf.image API for data augmentation with TPU. We will perform augmentations using GPU/TPU using tf.data API."},{"metadata":{},"cell_type":"markdown","source":"## 5.1 Augmentation Utilities\n[Table of contents](#0.1)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def make_mask(num_holes,side_length,rows, cols, num_channels):\n \n \"\"\"Builds the mask for all sprinkles.\"\"\"\n \n row_range = tf.tile(tf.range(rows)[..., tf.newaxis], [1, num_holes])\n col_range = tf.tile(tf.range(cols)[..., tf.newaxis], [1, num_holes])\n r_idx = tf.random.uniform([num_holes], minval=0, maxval=rows-1,\n dtype=tf.int32)\n c_idx = tf.random.uniform([num_holes], minval=0, maxval=cols-1,\n dtype=tf.int32)\n r1 = tf.clip_by_value(r_idx - side_length // 2, 0, rows)\n r2 = tf.clip_by_value(r_idx + side_length // 2, 0, rows)\n c1 = tf.clip_by_value(c_idx - side_length // 2, 0, cols)\n c2 = tf.clip_by_value(c_idx + side_length // 2, 0, cols)\n row_mask = (row_range > r1) & (row_range < r2)\n col_mask = (col_range > c1) & (col_range < c2)\n\n # Combine masks into one layer and duplicate over channels.\n mask = row_mask[:, tf.newaxis] & col_mask\n mask = tf.reduce_any(mask, axis=-1)\n mask = mask[..., tf.newaxis]\n mask = tf.tile(mask, [1, 1, num_channels])\n return mask\n \ndef sprinkles(image): \n num_holes = 20\n side_length = 15\n mode = 'normal'\n PROBABILITY = 1\n \n RandProb = tf.cast( tf.random.uniform([],0,1) < PROBABILITY, tf.int32)\n if (RandProb == 0)|(num_holes == 0): return image\n \n img_shape = tf.shape(image)\n if mode is 'normal':\n rejected = tf.zeros_like(image)\n elif mode is 'salt_pepper':\n num_holes = num_holes // 2\n rejected_high = tf.ones_like(image)\n rejected_low = tf.zeros_like(image)\n elif mode is 'gaussian':\n rejected = tf.random.normal(img_shape, dtype=tf.float32)\n else:\n raise ValueError(f'Unknown mode \"{mode}\" given.')\n \n rows = img_shape[0]\n cols = img_shape[1]\n num_channels = img_shape[-1]\n if mode is 'salt_pepper':\n mask1 = make_mask(num_holes,side_length,rows, cols, num_channels)\n mask2 = make_mask(num_holes,side_length,rows, cols, num_channels)\n filtered_image = tf.where(mask1, rejected_high, image)\n filtered_image = tf.where(mask2, rejected_low, filtered_image)\n else:\n mask = make_mask(num_holes,side_length,rows, cols, num_channels)\n filtered_image = tf.where(mask, rejected, image)\n return filtered_image\n\ndef transform_shear(image, height, shear, mask=False):\n \n '''\n shear augmentation on image\n and mask.\n --------------------------------\n \n Arguments:\n image -- input image\n mask -- input mask\n \n Return:\n image -- augmented image \n mask -- augmented mask\n '''\n \n DIM = height\n XDIM = DIM%2 #fix for size 331\n \n shear = shear * tf.random.uniform([1],dtype='float32')\n shear = math.pi * shear / 180.\n \n # SHEAR MATRIX\n one = tf.constant([1],dtype='float32')\n zero = tf.constant([0],dtype='float32')\n c2 = tf.math.cos(shear)\n s2 = tf.math.sin(shear)\n shear_matrix = tf.reshape(tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3]) \n\n # LIST DESTINATION PIXEL INDICES\n x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n z = tf.ones([DIM*DIM],dtype='int32')\n idx = tf.stack( [x,y,z] )\n \n # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n idx2 = K.dot(shear_matrix,tf.cast(idx,dtype='float32'))\n idx2 = K.cast(idx2,dtype='int32')\n idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n \n # FIND ORIGIN PIXEL VALUES \n idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n d = tf.gather_nd(image, tf.transpose(idx3))\n \n if mask:\n return tf.reshape(d, [DIM,DIM,1])\n \n return tf.reshape(d, [DIM,DIM,3])\n\ndef transform_shift(image, height, h_shift, w_shift, mask=False):\n \n '''\n shift augmentation on image\n and mask.\n --------------------------------\n \n Arguments:\n image -- input image\n mask -- input mask\n \n Return:\n image -- augmented image \n mask -- augmented mask\n '''\n \n DIM = height\n XDIM = DIM%2 #fix for size 331\n \n height_shift = h_shift * tf.random.uniform([1],dtype='float32') \n width_shift = w_shift * tf.random.uniform([1],dtype='float32') \n one = tf.constant([1],dtype='float32')\n zero = tf.constant([0],dtype='float32')\n \n # SHIFT MATRIX\n shift_matrix = tf.reshape(tf.concat([one,zero,height_shift, zero,one,width_shift, zero,zero,one],axis=0),[3,3])\n\n # LIST DESTINATION PIXEL INDICES\n x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n z = tf.ones([DIM*DIM],dtype='int32')\n idx = tf.stack( [x,y,z] )\n \n # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n idx2 = K.dot(shift_matrix,tf.cast(idx,dtype='float32'))\n idx2 = K.cast(idx2,dtype='int32')\n idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n \n # FIND ORIGIN PIXEL VALUES \n idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n d = tf.gather_nd(image, tf.transpose(idx3))\n \n if mask:\n return tf.reshape(d, [DIM,DIM,1])\n \n return tf.reshape(d, [DIM,DIM,3])\n\ndef augmentations(image, mask):\n \n '''\n Apply different augmentations on \n image and mask.\n --------------------------------\n \n Arguments:\n image -- input image\n mask -- input mask\n \n Return:\n image -- augmented image \n mask -- augmented mask\n '''\n \n spatial = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n shear = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n shift = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n pixel = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n drop_coarse = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n \n # SPATIAL-LEVEL TRANSFORMATIONS\n ## FLIP LEFT-RIGHT\n if spatial >= .2:\n image = tf.image.flip_left_right(image)\n mask = tf.image.flip_left_right(mask)\n \n ## FLIP UP-DOWN\n if spatial >= .3: \n image = tf.image.flip_up_down(image)\n mask = tf.image.flip_up_down(mask)\n \n ## ROTATIONS\n if rotate > .75:\n image = tf.image.rot90(image, k=3) # rotate 270º\n mask = tf.image.rot90(mask, k=3) # rotate 270º\n elif rotate > .5:\n image = tf.image.rot90(image, k=2) # rotate 180º\n mask = tf.image.rot90(mask, k=2) # rotate 180º\n elif rotate > .25:\n image = tf.image.rot90(image, k=1) # rotate 90º\n mask = tf.image.rot90(mask, k=1) # rotate 90º\n \n ## SHEAR \n if shear >= .3:\n image = transform_shear(image, height=config.IMAGE_DIM, shear=20.)\n mask = transform_shear(mask, height=config.IMAGE_DIM, shear=20., mask=True)\n \n# ## SHIFT\n# if shift >= .3:\n# image = transform_shift(image, height=config.IMAGE_DIM, h_shift=15., w_shift=15.)\n# mask = transform_shift(mask, height=config.IMAGE_DIM, h_shift=15., w_shift=15., mask=True)\n\n ## COARSE-DROPOUT\n if drop_coarse >= .3:\n image = sprinkles(image)\n mask = sprinkles(mask)\n \n # PIXEL-LEVEL TRANSFORMATION\n if pixel >= .2:\n \n if pixel >= .7:\n image = tf.image.random_brightness(image, .2)\n elif pixel >= .6:\n image = tf.image.random_hue(image, .2)\n elif pixel >= .5:\n image = tf.image.random_contrast(image, 0.8, 1.2)\n elif pixel >= .4:\n image = tf.image.random_saturation(image, 0.7, 1.3)\n \n return image, mask","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.2 Data Preprocessing Utilities\n[Table of contents](#0.1)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/paulorzp/rle-functions-run-lenght-encode-decode\ndef mask2rle(img):\n \n '''\n img: numpy array, 1 - mask, 0 - background\n Returns run length as string formated\n '''\n pixels= img.T.flatten()\n pixels = np.concatenate([[0], pixels, [0]])\n runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n runs[1::2] -= runs[::2]\n return ' '.join(str(x) for x in runs)\n \ndef rle2mask(mask_rle, shape):\n \n '''\n mask_rle: run-length as string formated (start length)\n shape: (width,height) of array to return \n Returns numpy array, 1 - mask, 0 - background\n\n '''\n s = mask_rle.split()\n starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n starts -= 1\n ends = starts + lengths\n img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n for lo, hi in zip(starts, ends):\n img[lo:hi] = 1\n return img.reshape(shape).T\n\ndef read_tiff(image, encoding_index, resize=None):\n \n '''\n read tiff images and mask.\n ----------------------------\n \n Arguments:\n image -- tiff image\n encoding_index -- corresponding tiff file encoding index.\n \n Returns:\n tiff_image -- tiff image\n tiff_mask -- segmentation mask\n '''\n \n tiff_image = tiff.imread(os.path.join(ROOT, f'train/{image}.tiff'))\n \n if len(tiff_image.shape) == 5:\n tiff_image = np.transpose(tiff_image.squeeze(), (1,2,0))\n \n tiff_mask = rle2mask(train['encoding'][encoding_index],\n (tiff_image.shape[1], tiff_image.shape[0]))\n \n print(f'Image Shape: {tiff_image.shape}')\n print(f'Mask Shape: {tiff_mask.shape}')\n \n if resize:\n rescaled = (tiff_image.shape[1] // resize, tiff_image.shape[0] // resize)\n tiff_image = cv2.resize(tiff_image, rescaled)\n tiff_mask = cv2.resize(tiff_mask, rescaled)\n\n return tiff_image, tiff_mask\n\ndef read_test_tiff(image, resize=None):\n \n '''\n read tiff images.\n ----------------------------\n \n Arguments:\n image -- tiff image\n \n Returns:\n tiff_image -- tiff image\n tiff_mask -- segmentation mask\n '''\n \n tiff_image = tiff.imread(os.path.join(ROOT, f'test/{image}.tiff'))\n \n if len(tiff_image.shape) == 5:\n tiff_image = np.transpose(tiff_image.squeeze(), (1,2,0))\n \n if resize:\n rescaled = (tiff_image.shape[1] // resize, tiff_image.shape[0] // resize)\n tiff_image = cv2.resize(tiff_image, rescaled)\n\n return tiff_image\n\ndef plot(image, mask):\n \n '''\n plot image and mask\n ---------------------\n \n Arguments:\n image -- tiff image \n mask -- segmentation mask\n \n Returns:\n matplotlib plot\n '''\n plt.figure(figsize=(15, 15))\n\n # Image\n plt.subplot(1, 3, 1)\n plt.imshow(image)\n plt.title(\"Image\", fontsize=16)\n\n # Mask\n plt.subplot(1, 3, 2)\n plt.imshow(mask)\n plt.title(\"Image Mask\", fontsize=16)\n\n # Image + Mask\n plt.subplot(1, 3, 3)\n plt.imshow(image)\n plt.imshow(mask, alpha=0.5, cmap='plasma')\n plt.title(\"Image + Mask\", fontsize=16);\n\ndef plot_subset(image, mask, start_rh, end_rh, start_cw, end_cw):\n \n '''\n plot image and mask\n ---------------------\n \n Arguments:\n image -- tiff image \n mask -- segmentation mask\n start_rh -- height start\n end_rh -- height end\n start_cw -- width start \n end_cw -- width end\n \n Returns:\n matplotlib plot\n '''\n\n # Figure size\n plt.figure(figsize=(15, 15))\n\n # subset image and mask\n subset_image = image[start_rh:end_rh, start_cw:end_cw, :]\n subset_mask = mask[start_rh:end_rh, start_cw:end_cw]\n\n # Image\n plt.subplot(1, 3, 1)\n plt.imshow(subset_image)\n plt.title(\"Zoomed Image\", fontsize=16)\n\n # Mask\n plt.subplot(1, 3, 2)\n plt.imshow(subset_mask)\n plt.title(\"Zoomed Mask\", fontsize=16)\n\n # Image + Mask\n plt.subplot(1, 3, 3)\n plt.imshow(subset_image)\n plt.imshow(subset_mask, alpha=0.5, cmap='plasma')\n plt.title(\"Zoomed Image + Mask\", fontsize=16);\n \ndef countplot(column, plot_type='multiple', gridstyle='whitegrid', gs=None,\n palette='Accent', xlab=None, ylab=None, title=None, fontsize=12):\n \n '''\n Make countplots\n -----------------\n \n Arguments:\n column -- column with categorical values\n plot_type -- multiple grid ('multiple/single')\n gridstyle -- seaborn gridstyle\n gs -- gridspec (if using subplots)\n palette -- color palette\n xlab -- x-axis label\n ylab -- y-axis label\n title -- plot title\n fontsize -- fontsize\n \n Returns:\n sns.countplot()\n '''\n if plot_type=='multiple':\n with sns.axes_style(gridstyle):\n ax = f.add_subplot(gs)\n aa = sns.countplot(column, palette=palette)\n for p in ax.patches:\n height = p.get_height()\n aa.text(p.get_x()+p.get_width()/2.,\n height,\n '{:1.2f}%'.format(height/len(column)*100),\n ha=\"center\", fontsize=fontsize)\n plt.xlabel(xlab,fontsize=fontsize)\n plt.ylabel(ylab,fontsize=fontsize)\n plt.title(title)\n \n elif plot_type=='single':\n with sns.axes_style(\"whitegrid\"):\n aa = sns.countplot(column, palette=palette)\n for p in aa.patches:\n height = p.get_height()\n aa.text(p.get_x()+p.get_width()/2.,\n height,\n '{:1.2f}%'.format(height/len(column)*100),\n ha=\"center\", fontsize=fontsize)\n plt.xlabel(xlab,fontsize=fontsize)\n plt.ylabel(ylab,fontsize=fontsize)\n plt.title(title)\n \ndef distplot(column, gridstyle='whitegrid', gs=None, stats=False, \n color='yellow', xlab=None, ylab=None, title=None, fontsize=12):\n \n '''\n Make distplots\n -----------------\n \n Arguments:\n column -- column with categorical values\n gridstyle -- seaborn gridstyle\n gs -- gridspec (if using subplots)\n stats -- mean, median, mode.\n color -- matplotlib color\n xlab -- x-axis label\n title -- plot title\n fontsize -- fontsize\n \n Returns:\n sns.distplot()\n '''\n with sns.axes_style(gridstyle):\n if gs:\n ax = f.add_subplot(gs)\n \n aa = sns.distplot(column, color=color)\n \n if stats:\n mean = column.mean()\n median = column.median()\n mode = column.mode()[0] \n ax.axvline(int(mean), color='r', linestyle='--')\n ax.axvline(int(median), color='g', linestyle='-')\n ax.axvline(mode, color='b', linestyle='-')\n plt.legend({'Mean':mean,'Median':median,'Mode':mode})\n \n plt.xlabel(xlab,fontsize=fontsize)\n plt.title(title)\n \ndef decode_image_and_mask(image, mask, augment=True):\n \n '''\n decode image and mask in order to\n feed data to TPU.\n --------------------------------\n \n Arguments:\n image -- patches of huge tiff file in png format.\n mask -- patches of mask in png format.\n augment -- apply augmentations on images and masks.\n \n Return:\n image \n mask\n '''\n \n # load raw data as string\n image = tf.io.read_file(image)\n mask = tf.io.read_file(mask)\n \n image = tf.io.decode_png(image, channels=3) # convert compressed string to 3D uint8 tensor\n mask = tf.io.decode_png(mask) # convert compressed string to uinst8 tensor\n \n if augment:\n \n if tf.random.uniform(()) > 0.5:\n image = tf.image.flip_left_right(image)\n mask = tf.image.flip_left_right(mask)\n \n if tf.random.uniform(()) > 0.4:\n image = tf.image.flip_up_down(image)\n mask = tf.image.flip_up_down(mask)\n \n if tf.random.uniform(()) > 0.5:\n image = tf.image.rot90(image, k=1)\n mask = tf.image.rot90(mask, k=1)\n \n if tf.random.uniform(()) > 0.45:\n image = tf.image.random_saturation(image, 0.7, 1.3)\n \n if tf.random.uniform(()) > 0.45:\n image = tf.image.random_contrast(image, 0.8, 1.2)\n \n image = tf.image.convert_image_dtype(image, tf.float32) # convert to floats in the [0,1] range\n mask = tf.cast(mask, tf.float32) # convert to floats 1. and 0.\n\n image = tf.reshape(image, [*IMAGE_DIM, 3]) # reshaping image tensor\n mask = tf.reshape(mask, [*IMAGE_DIM]) # reshaping mask tensor\n \n return image, mask\n\ndef generate_data(tiff, masks, batch_size=16, shuffle=True):\n \n '''\n generate batches of tf.Dataset\n object\n --------------------------------\n \n Arguments:\n tiff -- tf.data.Dataset object (tf.Tensor)\n mask -- tf.data.Dataset object (tf.Tensor)\n batch_size -- batches of image, mask pair\n shuffle -- generate train if True or validation data if False\n \n Return:\n ds - tf.data.Dataset dataset \n '''\n \n \n ds = Dataset.zip((tiff, masks)) # create dataset by zipping (image, mask) into pair\n ds = ds.map(decode_image_and_mask, num_parallel_calls=AUTOTUNE) # decode raw data coming from GCS bucket to valid image, mask pair \n ds = ds.cache() # cache dataset preprocessing work that doesn't fit in memory\n ds = ds.repeat() \n \n # shuffle while training else set to False\n if shuffle:\n ds = ds.shuffle(buffer_size=1000)\n \n ds = ds.batch(batch_size) # generate batches of data\n ds = ds.prefetch(buffer_size=AUTOTUNE) # fetch dataset while model is training\n return ds","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.3 Training Utilities\n[Table of contents](#0.1)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def decode_image_and_mask(image, mask):\n \n '''\n decode and normalize image and\n mask.\n --------------------------------\n \n Arguments:\n image -- input image (str)\n mask -- input mask (str)\n \n Return:\n image -- normalized image\n mask -- normalized mask\n '''\n \n # load raw data as string\n image = tf.io.read_file(image)\n mask = tf.io.read_file(mask)\n \n image = tf.io.decode_png(image, channels=3) # convert compressed string to 3D uint8 tensor\n mask = tf.io.decode_png(mask) # convert compressed string to uinst8 tensor\n \n image = tf.image.convert_image_dtype(image, tf.float32) # convert to floats in the [0,1] range\n mask = tf.cast(mask, tf.float32) # convert to floats 1. and 0.\n\n image = tf.reshape(image, (config.IMAGE_DIM, config.IMAGE_DIM, 3)) # reshaping image tensor\n mask = tf.reshape(mask, (config.IMAGE_DIM, config.IMAGE_DIM, 1)) # reshaping mask tensor\n \n return image, mask\n\ndef generate_data(tiff, masks, batch_size=16, shuffle=True, augment=False):\n \n '''\n generate batches of tf.Dataset\n object\n --------------------------------\n \n Arguments:\n tiff -- tf.data.Dataset object (tf.Tensor)\n mask -- tf.data.Dataset object (tf.Tensor)\n batch_size -- batches of image, mask pair\n shuffle -- shuffle data \n augment -- apply augmentations\n \n Return:\n ds - tf.data.Dataset dataset \n '''\n \n # create dataset by zipping (image, mask) into pair\n ds = tf.data.Dataset.zip((tiff, masks))\n \n # decode raw data coming from GCS bucket to valid image, mask pair\n ds = ds.map(decode_image_and_mask, num_parallel_calls=AUTOTUNE) \n \n # apply advance augmentations\n if augment:\n ds = ds.map(augmentations ,num_parallel_calls=AUTOTUNE)\n \n # cache dataset preprocessing work that doesn't fit in memory\n ds = ds.cache()\n \n # repeat forever\n ds = ds.repeat() \n \n # shuffle while training else set to False\n if shuffle:\n ds = ds.shuffle(buffer_size=1000)\n \n ds = ds.batch(config.BATCH_SIZE, drop_remainder=True) # generate batches of data\n ds = ds.prefetch(buffer_size=AUTOTUNE) # fetch dataset while model is training\n return ds","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6. Data Overview🔍\n[Table of contents](#0.1)\n\nWe can see that we have **train and test folders with .tiff images and annotations in JSON**. We have **train.csv** and **HuBMAP-20-dataset_information.csv** containing image, masks information and metadata respectively.\n\n## 6.1 Train Data\n\nWe will start getting glimpse of our train data first. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train = pd.read_csv(os.path.join(ROOT, 'train.csv'))\ntrain","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Observations**\n\n**We have following features in train.csv-**\n* id - unique id for each image.\n* encoding - RLE encoded representation of the glomeruli mask in the image. Run-length encoding (RLE) is a form of lossless data compression in which runs of data (sequences in which the same data value occurs in many consecutive data elements) are stored as a single data value and count."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'We have {train.shape[0]} rows and {train.shape[1]} columns in our train.csv.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train Dataframe Information\n\nWe have two columns in train.csv. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Missing Values\n\nWe have no missing values."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'Missing values in train.csv in each columns:\\n{train.isnull().sum()}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Unique Values\n\nOnly 8 images are there, all unique. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('Unique Values in each column of train.csv')\nprint('##########################################')\nfor col in train:\n print(f'{col}: {train[col].nunique()}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.2 HuBMAP-Metadata\n[Table of contents](#0.1)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"metadata = pd.read_csv(os.path.join(ROOT, 'HuBMAP-20-dataset_information.csv'))\nmetadata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Observations**\n\n* image_file - Unique image id.\n* width_pixels - pixel width.\n* height_pixels - pixel height.\n* anatomical_structures_segmention_file\t- JSON file corresponding per image.\n* glomerulus_segmentation_file - segmentations of glomeruli per image.\n* patient_number - As the name suggest patient id.\n* race - which race a patient belong.\n* ethnicity - which ethnic group patient belong.\n* sex - gender\n* age - patient's age\n* weight_kilograms - weight of patient (in Kg).\n* height_centimeters - height of patient (in cm).\n* bmi_kg/m^2 - Body-Mass index\n* laterality - Side of kideny (left / right).\n* percent_cortex - The outer part of the kidney is called the **cortex**.\n* percent_medulla - The inner part is the kidney is called the **medulla**."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'We have {metadata.shape[0]} rows and {metadata.shape[1]} columns in our metadata.csv.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Metadata Information\n\nFirst we will convert convert columns with category into categorical type."},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"for col in ['race', 'ethnicity', 'sex', 'laterality']:\n metadata[col] = metadata[col].astype('category')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"metadata.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Missing Values"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'Missing values in metadata.csv in each columns:\\n{metadata.isnull().sum()}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Unique Values"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('Unique Values in each column of metadata.csv')\nprint('##########################################')\nfor col in metadata:\n print(f'{col}: {metadata[col].nunique()}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.3 Image + Segmentation Mask\n### 6.3.1 Train Tiff Images\n[Table of contents](#0.1)\n\nThere are 8 tiff images available for training. "},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"print('We have following train images.')\nfor index, name in enumerate(train.id.values):\n print(name)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"image, mask = read_tiff('2f6ecfcdf', 0)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot(image, mask)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_subset(image, mask, 5000, 10000, 10000, 15000)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"image, mask = read_tiff('aaa6a05cc', 1)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot(image, mask)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_subset(image, mask, 10000, 12500, 2000, 4000)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"image, mask = read_tiff('cb2d976f4', 2, 3)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot(image, mask)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_subset(image, mask, 2000, 5000, 1000, 4000)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"image, mask = read_tiff('0486052bb', 3, 3)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot(image, mask)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_subset(image, mask, 1000, 4000, 1000, 4000)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7. EDA\n## 7.1 Individual Features\n[Table of contents](#0.1)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\ndistplot(metadata['patient_number'], gs=gs[0,0], stats=True, xlab='Patient Number', title='Patient Number Distribution',\n color = 'darkseagreen')\ndistplot(metadata['bmi_kg/m^2'], gs=gs[0,1], stats=True, xlab='BMI Index (kg/m^2)', title='BMI Index Distribution',\n color = 'teal')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Observations**\n* Patient Id ranges from 64000 to 67000. \n* BMI Index ranges from 22 to 36 with mean of **27 kg/m^2**."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\ndistplot(metadata['weight_kilograms'], gs=gs[0,0], stats=True, xlab='Weight (kgs)', title='Patient Weight Distribution')\ndistplot(metadata['height_centimeters'], gs=gs[0,1], stats=True, xlab='Height (cms)', title='Patient Height Distribution',\n color = 'purple')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Observations**\n* Both patient's **Weight & Height** seems to be normally distributed. \n* Mean Weight and Height is **81 kg and 170 cm** respectively.\n* Most patients have height between **160 cm to 175 cm.**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\ndistplot(metadata['width_pixels'], gs=gs[0,0], stats=True, xlab='Image Width (px)', title='Image Weight Distribution',\n color = 'cyan')\ndistplot(metadata['height_pixels'], gs=gs[0,1], stats=True, xlab='Image Height (px)', title='Image Height Distribution',\n color = 'green')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Observations**\n* Image width distribution ranges from 10000 to 50000. Mean and median at 32000. \n* Image height distribution ranges from 15000 to 38000. Mean and median at 28000 and 30500 respectively."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\ndistplot(metadata['percent_cortex'], gs=gs[0,0], stats=True, xlab='Cortex (%)', title='Cortex Distribution',\n color = 'coral')\ndistplot(metadata['percent_medulla'], gs=gs[0,1], stats=True, xlab='Medulla (%)', title='Medulla Distribution',\n color = 'red')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Observations**\n* Cortex and Medulla distribution is opposite of each other with different ranges . \n* Cortes % ranges from 53 and 80 percent.\n* Medulla % ranges fron 20 to 45 percent."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 3)\n\ncountplot(metadata['race'], gs=gs[0,0], xlab='race', ylab='count', title='Race Distribution')\ncountplot(metadata['sex'], gs=gs[0,1], palette='Blues', xlab='sex', ylab='count', title='Gender Distribution')\ncountplot(metadata['laterality'], gs=gs[0,2], palette='CMRmap', xlab='laterality', ylab='count',\n title='Laterality Distribution')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Observations**\n* **White** and **Black or African American** are 69% and 31% respectively.\n* There are **54% males and 46% females**.\n* We can also see some pattern in third column which is reverse of **Gender Distribution**. **54% and 46% laterality (Left/Right Kidney) are Right and Left respectively.**"},{"metadata":{},"cell_type":"markdown","source":"## 7.2 Multiple Features\n### 7.2.1 Image Shape Distribution\n[Table of contents](#0.1)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\nwith sns.axes_style(\"whitegrid\"):\n ax = f.add_subplot(gs[0, 0])\n sns.scatterplot(metadata['width_pixels'], metadata['height_pixels'], hue=metadata['age'])\n plt.xlabel('width',fontsize=12)\n plt.ylabel('height',fontsize=12)\n plt.title('Image Resolution Distribution Vs Age', fontsize=14)\n \nwith sns.axes_style(\"whitegrid\"):\n ax = f.add_subplot(gs[0, 1])\n sns.scatterplot(metadata['width_pixels'], metadata['height_pixels'], hue=metadata['race'], size=metadata['sex'])\n plt.xlabel('width',fontsize=12)\n plt.ylabel('height',fontsize=12)\n plt.title('Image Resolution Distribution Vs Race/Sex', fontsize=14)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Observations**\n* Most of the **White** patients are **Female**.\n* It seems all **Black and African American** patients are male.\n* Patients between age 30-60 are more prominant. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\nwith sns.axes_style(\"whitegrid\"):\n ax = f.add_subplot(gs[0, 0])\n sns.scatterplot(metadata['percent_cortex'], metadata['percent_medulla'], hue=metadata['race'],\n style=metadata['race'], palette='hot')\n plt.xlabel('Cortex (%)',fontsize=12)\n plt.ylabel('Medulla (%)',fontsize=12)\n plt.title('Cortex & Medulla Vs Race', fontsize=14)\n \nwith sns.axes_style(\"whitegrid\"):\n ax = f.add_subplot(gs[0, 1])\n sns.scatterplot(metadata['percent_cortex'], metadata['percent_medulla'], hue=metadata['sex'],\n size=metadata['sex'], palette='cividis')\n plt.xlabel('Cortex (%)',fontsize=12)\n plt.ylabel('Medulla (%)',fontsize=12)\n plt.title('Percent Cortex Vs Percent Medulla ', fontsize=14)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Observations**\n* We see negative relationship b/w **perent_cortex and percent_medulla**.\n* Distribution of **perent_cortex and percent_medulla** on the basis of **race** seems equal for both races i.e **White and Black or African American**.\n* Distribution of **perent_cortex and percent_medulla** on the basis of **sex** seems equal too for both **male and female**. "},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 7.2.2 Metadata Heatmap\n[Table of contents](#0.1)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 10))\n\nmask = np.triu(np.ones_like(metadata.corr(), dtype=bool))\n\nwith sns.axes_style(\"white\"):\n sns.heatmap(metadata.corr(), mask=mask, square=True, cmap = 'YlOrBr', annot=True);\n plt.title(\"Meta Data Feature Correlation\", fontsize=14)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 8. Pandas MetaData Profiling\n[Table of contents](#0.1)"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"metadata_profile = pp.ProfileReport(metadata)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"metadata_profile","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TRAINING PART IN THIS NOTEBOOK IS FOR THE PURPOSE OF HELPING PEOPLE WITH TENSORFLOW-KERAS TRAINING. YOU CAN TRY USING SEVERAL DIFFERENT ARCHITECTURES AVAILBLE IN SEGEMNTATION MODELS LIBRARY."},{"metadata":{},"cell_type":"markdown","source":"# 9. Tensorflow-Keras Training Pipeline\n[Table of contents](#0.1)\n\nIn this section we will see how to **train Unet model using tensorflow-keras on TPU if the data is not in \"Tfrecord format\"**. We will use the [tf.data](https://www.tensorflow.org/api_docs/python/tf/data) API in order to build our data pipeline. You can also train model using tfrecords. Many great notebooks have been published for the same by [Marcos Novaes](https://www.kaggle.com/marcosnovaes) and [Geir Drange](https://www.kaggle.com/mistag).\n\nI am using Weights and Biases to log my result. I have included some step just in case is anyone wants to start with it.\n\nFirst we need to install [Pavel Yakubovskiy](https://github.com/qubvel) amazing library [segmentation_models](https://github.com/qubvel/segmentation_models). This will make experimentation real quick for us. We are also going to install [Weights & Biases](https://www.wandb.com/) in order to log our results. This will help us to interactively visualize our training results."},{"metadata":{},"cell_type":"markdown","source":"
"},{"metadata":{},"cell_type":"markdown","source":"## 9.1 Configuration🔨\n### 9.1.1 Hardware Configuration (TPU/GPU/CPU)🔨\n[Table of contents](#0.1)\n\nWe will configure hardware accelerator here. For more information on how **TPU works on Kaggle** please go [here](https://www.kaggle.com/docs/tpu)."},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n # detect and initialize tpu\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Connecting to tpu...')\n print('device running at:', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n print('Initializing TPU...')\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n # instantiate a distribution strategy\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\n print(\"TPU initialized\")\nelse:\n print('Using deafualt strategy...')\n strategy = tf.distribute.get_strategy()\n\nREPLICAS = strategy.num_replicas_in_sync\nprint(f\"REPLICAS: {REPLICAS}\")\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 9.1.2 Weights and Biases Configuration\n[Table of contents](#0.1)\n\nHere I am using **Kaggle's Add-ons** to hide my secret wandb login key. You can create your secret key by first initializing your wandb project after creating your account on Weights and Biases. Don't worry you will see ahead how. For amazing step-by-step guide please refer [here](https://www.kaggle.com/imeintanis/cnn-track-your-experiments-weights-biases/notebook).\n\n* Inside your notebook workspace on top header you will see options Click on **Add-ons**.\n\n* Now click on **secrets**.\n\n* When you'll run this line *wandb.init(project=\"project-folder-on-W&B\", name= 'project_name')* in code cell in upcoming section ahead in this notebook. After running this line you will see a link in output. You have to click on it, copy the key and paste it in the **Value** section inside **secret**.\n\n* See the below image to get some idea."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"Image('../input/lyftl5googlecolab/wandb.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Now copy the code as given in the image or check the cell code cell below to use the login key. This way you can hide your credentials."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"user_secrets = UserSecretsClient()\nsecret_value = user_secrets.get_secret(\"WANDB_KEY\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Run the below cell to append your wandb API key."},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!wandb login $secret_value","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we are defining are hyperparameters which we will track using Weights and Biases"},{"metadata":{"trusted":true},"cell_type":"code","source":"Params = dict(\n DEVICE = 'tpu',\n RUN = 3, # Successful version number to be tracked by W&B\n SEED = 0, # seed for reproducibility\n BATCH_SIZE = 32 * REPLICAS, # batch size\n IMAGE_DIM = 256, # image dimension\n ENCODER = 'seresnext50', # segmentation encoder\n WEIGHT = 'imagenet', # imagenet weights\n VERBOSE = 0, # interactive/silent training\n DISPLAY_PLOT = True, # display plot at end of each fold training\n EPOCHS = 70, # epoch\n LR = 1e-5, # learning rate\n FOLDS = 4, # number of folds\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DIM = Params['IMAGE_DIM'] # image dimension\nRUN = Params['RUN'] # successful wandb run number\nMODEL = Params['ENCODER'] # segmentation encoder ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we are defining are hyperparameters which we will track using Weights and Biases.\n\nFinally I have initialized my run. Please keep in mind that each run is single execution of the training script. After running below cell you will get some links in output including link to your project page which you need to create first inside your W&B profile.\n\nHere you can see -\n\n* projects -- your project directory at your W&B profile.\n* name -- name of every run (single training script/notebook execution). You can keep to default depends on your choice.\n* config -- save all your hyperparameters in a config object."},{"metadata":{"trusted":true},"cell_type":"code","source":"wandb.init(project=\"hubmap-hacking-the-kidney\",\n name= f'TPU-No-Tfrec-Public-{DIM}-{MODEL}-V{RUN}',\n config=Params)\n\nconfig = wandb.config","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"config.keys()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 9.2 Data Preprocessing🔬\n[Table of contents](#0.1)\n\nWe will write our data pipeline using [tf.data](https://www.tensorflow.org/tutorials/load_data/images#using_tfdata_for_finer_control) API. Please check the [utility](#3) section for implemented utilities. TPU will be reading our data from **GCS buckets**."},{"metadata":{"trusted":true},"cell_type":"code","source":"# obtain the GCS path of a Kaggle dataset\nGCS_PATH = KaggleDatasets().get_gcs_path('hubmap-256x256')\n\n# appending GCS PATH for train images and masks\nTIFF = tf.io.gfile.glob(str(GCS_PATH + '/train/*'))\nMASK = tf.io.gfile.glob(str(GCS_PATH + '/masks/*'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"TRAIN_TIFF = tf.data.Dataset.from_tensor_slices(TIFF)\nTRAIN_MASK = tf.data.Dataset.from_tensor_slices(MASK)\n\nTIFF_COUNT = tf.data.experimental.cardinality(TRAIN_TIFF).numpy()\nMASK_COUNT = tf.data.experimental.cardinality(TRAIN_MASK).numpy()\n\nprint('Training Data')\nprint(f'Total Tiff Images: {TIFF_COUNT}')\nprint(f'Total Masks: {MASK_COUNT}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's print our files from GCS bucket and check them."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for files in TRAIN_TIFF.take(5):\n print(files.numpy())\nprint('\\n')\nfor files in TRAIN_MASK.take(5):\n print(files.numpy())","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_ds = generate_data(TRAIN_TIFF, TRAIN_MASK, batch_size=config.BATCH_SIZE,\n shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"for image, mask in train_ds.take(1):\n image_batch, mask_batch = image, mask\n print(\"Image shape: \", image_batch.numpy().shape)\n print(\"Mask shape: \", mask_batch.numpy().shape)\n \ndel train_ds","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,16))\nfor i,(img,mask) in enumerate(zip(image_batch[:64], mask_batch[:64])):\n plt.subplot(8,8,i+1)\n plt.imshow(img,vmin=0,vmax=255)\n plt.imshow(np.squeeze(mask), alpha=0.4, cmap='plasma')\n plt.axis('off')\n plt.subplots_adjust(wspace=None, hspace=None)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 9.3 Data Augmentations\n[Table of contents](#0.1)\n\nWe will perform augmentations using GPU/TPU using tf.data API. For more information check this amazing notebook by [Chris Deotte](https://www.kaggle.com/cdeotte) and [Dimitre Oliveira](https://www.kaggle.com/dimitreoliveira). The notebooks can be found [here](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96#Data-Augmentation-using-GPU/TPU-for-Maximum-Speed!) and [here](https://www.kaggle.com/dimitreoliveira/flower-with-tpus-advanced-augmentations#Advanced-augmentations)."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ds = generate_data(TRAIN_TIFF, TRAIN_MASK, batch_size=config.BATCH_SIZE,\n shuffle=False, augment=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for image, mask in train_ds.take(1):\n image_batch, mask_batch = image, mask\n print(\"Image shape: \", image_batch.numpy().shape)\n print(\"Mask shape: \", mask_batch.numpy().shape)\n \ndel train_ds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,16))\nfor i,(img,mask) in enumerate(zip(image_batch[:64], mask_batch[:64])):\n plt.subplot(8,8,i+1)\n plt.imshow(img,vmin=0,vmax=255)\n plt.imshow(np.squeeze(mask), alpha=0.4, cmap='plasma')\n plt.axis('off')\n plt.subplots_adjust(wspace=None, hspace=None)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 9.4 Metric: Dice Coefficient\t🎲\n[Table of contents](#0.1)\n\nDice similarity coefficient is ideal for segmentation tasks. It is measure of how well two contours overlap. The dice index ranges from **0 (imperfect match) to 1 (perfect match)**.\n\n
\n\nThe dice coefficient is givern as follows -\n\n$$\\text{DSC}(A, B) = \\frac{2 \\times |A \\cap B|}{|A| + |B|}.$$\n\nHere,\n* A = predicted mask.\n* B = ground truth mask.\n\n$$\\text{DSC}(f, x, y) = \\frac{2 \\times \\sum_{i, j} f(x)_{ij} \\times y_{ij} + \\epsilon}{\\sum_{i,j} f(x)_{ij} + \\sum_{i, j} y_{ij} + \\epsilon}$$\n\nHere,\n* x = input image.\n* f(x) = predicted output mask by model.\n* y = ground truth mask.\n* epsilon = small number to avoid divide by zero."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# dice coefficient\ndef diceCoefficient(y_true, y_pred, epsilon = 1e-10):\n\n '''\n Dice Coefficient in Tensorflow\n ------------------------------\n\n Arguments: \n y_true (Tensorflow tensor) -- tensor of ground truth values.\n y_pred (Tensorflow tensor) -- tensor of predicted values.\n epsilon -- constant to avoid divide by 0 errors.\n\n Returns:\n dice_coefficient\n '''\n y_true_f = K.flatten(y_true)\n y_pred_f = K.flatten(y_pred)\n intersection = K.sum(y_true_f * y_pred_f)\n return (2. * intersection + epsilon) / (K.sum(y_true_f) + K.sum(y_pred_f) + epsilon)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 9.5 Loss🎲\n### Dice Loss\n[Table of contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# soft dice loss\ndef dice_loss(y_true, y_pred):\n loss = 1 - diceCoefficient(y_true, y_pred)\n return loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Tversky Loss"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# tversky loss\ndef tversky(y_true, y_pred, alpha=0.7, beta=0.3, smooth=1):\n\n\n '''\n Tversky in Keras\n ------------------------------\n\n Arguments: \n y_true (Tensorflow tensor) -- tensor of ground truth values.\n y_pred (Tensorflow tensor) -- tensor of predicted values.\n smooth -- constant to avoid divide by 0 errors.\n alpha -- constant to control penalties for false positives. \n beta -- constant to control penalties for false negatives.\n\n Returns:\n tversky loss\n '''\n\n y_true_pos = K.flatten(y_true)\n y_pred_pos = K.flatten(y_pred)\n true_pos = K.sum(y_true_pos * y_pred_pos)\n false_neg = K.sum(y_true_pos * (1 - y_pred_pos))\n false_pos = K.sum((1 - y_true_pos) * y_pred_pos)\n return (true_pos + smooth) / (true_pos + alpha * false_neg + beta * false_pos + smooth)\n\n# tversky loss\ndef tversky_loss(y_true, y_pred):\n return 1 - tversky(y_true, y_pred)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Focal Tversky Loss"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# focal tversky loss\ndef focal_tversky_loss(y_true, y_pred, gamma=0.75):\n tv = tversky(y_true, y_pred)\n return K.pow((1 - tv), gamma)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Lovasz Loss"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# \"\"\"\n# Lovasz-Softmax and Jaccard hinge loss in Tensorflow\n# Maxim Berman 2018 ESAT-PSI KU Leuven (MIT License)\n# \"\"\"\ndef lovasz_loss(y_true, y_pred):\n y_true, y_pred = K.cast(K.squeeze(y_true, -1), 'int32'), K.cast(K.squeeze(y_pred, -1), 'float32')\n logits = K.log(y_pred / (1. - y_pred))\n loss = lovasz_hinge(logits, y_true, per_image=True, ignore=None)\n return loss\n\n\ndef lovasz_grad(gt_sorted):\n \"\"\"\n Computes gradient of the Lovasz extension w.r.t sorted errors\n See Alg. 1 in paper\n \"\"\"\n gts = tf.reduce_sum(gt_sorted)\n intersection = gts - tf.cumsum(gt_sorted)\n union = gts + tf.cumsum(1. - gt_sorted)\n jaccard = 1. - intersection / union\n jaccard = tf.concat((jaccard[0:1], jaccard[1:] - jaccard[:-1]), 0)\n return jaccard\n\n\ndef lovasz_hinge(logits, labels, per_image=True, ignore=None):\n \"\"\"\n Binary Lovasz hinge loss\n logits: [B, H, W] Variable, logits at each pixel (between -\\infty and +\\infty)\n labels: [B, H, W] Tensor, binary ground truth masks (0 or 1)\n per_image: compute the loss per image instead of per batch\n ignore: void class id\n \"\"\"\n if per_image:\n def treat_image(log_lab):\n log, lab = log_lab\n log, lab = tf.expand_dims(log, 0), tf.expand_dims(lab, 0)\n log, lab = flatten_binary_scores(log, lab, ignore)\n return lovasz_hinge_flat(log, lab)\n\n losses = tf.map_fn(treat_image, (logits, labels), dtype=tf.float32)\n\n # Fixed python3\n losses.set_shape((None,))\n\n loss = tf.reduce_mean(losses)\n else:\n loss = lovasz_hinge_flat(*flatten_binary_scores(logits, labels, ignore))\n return loss\n\n\ndef lovasz_hinge_flat(logits, labels):\n \"\"\"\n Binary Lovasz hinge loss\n logits: [P] Variable, logits at each prediction (between -\\infty and +\\infty)\n labels: [P] Tensor, binary ground truth labels (0 or 1)\n ignore: label to ignore\n \"\"\"\n\n def compute_loss():\n labelsf = tf.cast(labels, logits.dtype)\n signs = 2. * labelsf - 1.\n errors = 1. - logits * tf.stop_gradient(signs)\n errors_sorted, perm = tf.nn.top_k(errors, k=tf.shape(errors)[0], name=\"descending_sort\")\n gt_sorted = tf.gather(labelsf, perm)\n grad = lovasz_grad(gt_sorted)\n # loss = tf.tensordot(tf.nn.relu(errors_sorted), tf.stop_gradient(grad), 1, name=\"loss_non_void\")\n # ELU + 1\n loss = tf.tensordot(tf.nn.elu(errors_sorted) + 1., tf.stop_gradient(grad), 1, name=\"loss_non_void\")\n return loss\n\n # deal with the void prediction case (only void pixels)\n loss = tf.cond(tf.equal(tf.shape(logits)[0], 0),\n lambda: tf.reduce_sum(logits) * 0.,\n compute_loss,\n# strict=True,\n name=\"loss\"\n )\n return loss\n\ndef flatten_binary_scores(scores, labels, ignore=None):\n \"\"\"\n Flattens predictions in the batch (binary case)\n Remove labels equal to 'ignore'\n \"\"\"\n scores = tf.reshape(scores, (-1,))\n labels = tf.reshape(labels, (-1,))\n if ignore is None:\n return scores, labels\n valid = tf.not_equal(labels, ignore)\n vscores = tf.boolean_mask(scores, valid, name='valid_scores')\n vlabels = tf.boolean_mask(labels, valid, name='valid_labels')\n return vscores, vlabels\n\n# lovasz loss\ndef symmetric_lovasz(y_true, y_pred):\n return 0.5*(lovasz_hinge(y_pred, y_true) + lovasz_hinge(-y_pred, 1.0 - y_true))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### BCE Dice Loss"},{"metadata":{"trusted":true},"cell_type":"code","source":"def bce_dice_loss(y_true, y_pred):\n loss = 0.5*binary_crossentropy(y_true, y_pred) + 0.5*dice_loss(y_true, y_pred)\n return loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_custom_objects().update({\"dice\": dice_loss})\nget_custom_objects().update({\"bce_dice\": bce_dice_loss})\nget_custom_objects().update({\"lovasz\": symmetric_lovasz})\nget_custom_objects().update({\"tversky\": tversky_loss})\nget_custom_objects().update({\"focal_tversky\": focal_tversky_loss})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 9.6 Model🚀\n[Table of contents](#0.1)"},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"model = sm.FPN(config.ENCODER,\n classes=1,\n activation='sigmoid',\n encoder_weights=None)\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"del model\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 9.7 Callbacks\n## Learning Rate Schedulers\n[Table of contents](#0.1) "},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"###############################\n#OneCycleLearningRateSchedular#\n###############################\n\nLR_START = 0.00001\nLR_MAX = 0.00005\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = .8\n\ndef lrfn(epoch):\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n return lr\n\nlr_one_cycle = LearningRateScheduler(lrfn, verbose=config.VERBOSE)\n\n##########################\n#CosineAnnealingScheduler#\n##########################\n\nclass CosineAnnealingScheduler(Callback):\n \"\"\"Cosine annealing scheduler.\n \"\"\"\n\n def __init__(self, T_max, eta_max, eta_min=0, verbose=1):\n super(CosineAnnealingScheduler, self).__init__()\n self.T_max = T_max\n self.eta_max = eta_max\n self.eta_min = eta_min\n self.verbose = verbose\n\n def on_epoch_begin(self, epoch, logs=None):\n if not hasattr(self.model.optimizer, 'lr'):\n raise ValueError('Optimizer must have a \"lr\" attribute.')\n lr = self.eta_min + (self.eta_max - self.eta_min) * (1 + math.cos(math.pi * epoch / self.T_max)) / 2\n K.set_value(self.model.optimizer.lr, lr)\n print('\\nEpoch %05d: CosineAnnealingScheduler setting learning ''rate to %s.' % (epoch + 1, lr))\n\n def on_epoch_end(self, epoch, logs=None):\n logs = logs or {}\n logs['lr'] = K.get_value(self.model.optimizer.lr)\n\n\ncosine_annealer = CosineAnnealingScheduler(T_max=10, eta_max=1e-5, eta_min=1e-6, verbose=config.VERBOSE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Early Stopping"},{"metadata":{"trusted":true},"cell_type":"code","source":"early_stop = EarlyStopping(monitor='val_diceCoefficient', mode = 'max',\n patience=10, restore_best_weights=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reduce On Plateau"},{"metadata":{"trusted":true},"cell_type":"code","source":"reduce = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=8, min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## WandbCallback"},{"metadata":{"trusted":true},"cell_type":"code","source":"# wandb = WandbCallback(monitor='val_diceCoefficient')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 9.8 Training\n[Table of contents](#0.1) \n\nWe are going to train our model for the number of FOLDS defined above in the configurations. We will save the model with best `val_diceCoefficient`. Set `VERBOSE` and `DISPLAY_PLOT` if you wish to see the train and val loss and dice coefficient improvement and Ploting results respectively.\n\nAlso we are using 4 fold cross validation here. Keeping in mind to avoid possible leakage we will keep tiles from particular image in train or val set."},{"metadata":{"trusted":true},"cell_type":"code","source":"avg_val_folds = []\n\nids = train.id.values\nfold = KFold(n_splits=config.FOLDS, shuffle=True, random_state=config.SEED)\nfor fold,(idxT,idxV) in enumerate(fold.split(ids)):\n \n print('#'*16); print(f'#### FOLD {fold+1} ####'); print('#'*16)\n print(f'Image Size: {config.IMAGE_DIM}, Batch Size: {config.BATCH_SIZE}, Epochs: {config.EPOCHS}')\n \n tr = set(ids[idxT])\n val = set(ids[idxV])\n \n # CREATE TRAIN AND VALIDATION SUBSETS\n print('preparing data...')\n train_images = [fname for fname in TIFF if fname.split('/')[4].split('_')[0] in tr]\n train_mask = [fname for fname in MASK if fname.split('/')[4].split('_')[0] in tr]\n valid_images = [fname for fname in TIFF if fname.split('/')[4].split('_')[0] in val]\n valid_mask = [fname for fname in MASK if fname.split('/')[4].split('_')[0] in val]\n \n \n # BUILD MODEL\n print('initializing model...')\n K.clear_session()\n with strategy.scope(): \n \n model = sm.FPN(config.ENCODER,\n classes=1,\n activation='sigmoid',\n encoder_weights='imagenet')\n \n model.compile(optimizer=Adam(lr = config.LR),\n loss='bce_dice',\n metrics=[diceCoefficient])\n \n # CALLBACKS\n checkpoint = ModelCheckpoint(f'/kaggle/working/hubmap-tf-keras-{config.DEVICE}-fold-%i.h5'%fold,\n verbose=config.VERBOSE,\n monitor='val_diceCoefficient',\n mode='max',\n save_best_only=True)\n \n # TRAIN-VAL DATA\n print('generating folds...')\n TRAIN_TIFF = tf.data.Dataset.from_tensor_slices(train_images)\n TRAIN_MASK = tf.data.Dataset.from_tensor_slices(train_mask)\n VAL_TIFF = tf.data.Dataset.from_tensor_slices(valid_images)\n VAL_MASK = tf.data.Dataset.from_tensor_slices(valid_mask)\n \n print('Training Model...')\n history = model.fit(\n generate_data(TRAIN_TIFF, TRAIN_MASK, batch_size=config.BATCH_SIZE, augment=True),\n epochs = config.EPOCHS,\n steps_per_epoch = tf.data.experimental.cardinality(TRAIN_TIFF).numpy() // config.BATCH_SIZE,\n callbacks = [checkpoint, early_stop, cosine_annealer],\n validation_data = generate_data(VAL_TIFF, VAL_MASK, batch_size=config.BATCH_SIZE, shuffle=False),\n validation_steps = tf.data.experimental.cardinality(VAL_TIFF).numpy() // config.BATCH_SIZE,\n verbose=config.VERBOSE\n )\n \n del TRAIN_TIFF, TRAIN_MASK, VAL_TIFF, VAL_MASK, model\n gc.collect()\n \n # PLOT TRAINING\n # https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\n if config.DISPLAY_PLOT:\n \n plt.figure(figsize=(15,5))\n TOTAL = np.arange(len(history.history['diceCoefficient']))\n plt.plot(TOTAL,history.history['diceCoefficient'],'-o',label='Train Dice Coefficient',color='#ff7f0e')\n plt.plot(TOTAL,history.history['val_diceCoefficient'],'-o',label='Val Dice Coefficient',color='#1f77b4')\n \n x = np.argmax( history.history['val_diceCoefficient'] ); dy = np.max( history.history['val_diceCoefficient'] )\n xdist = plt.xlim()[1] - plt.xlim()[0]; ydist = plt.ylim()[1] - plt.ylim()[0]\n plt.scatter(x, dy,s=200,color='#1f77b4'); plt.text(x-0.03*xdist, dy-0.13*ydist,'max dice_coe\\n%.2f'%dy,size=14)\n \n plt.ylabel('Dice Coefficient',size=14); plt.xlabel('Epoch',size=14)\n plt.legend(loc=2)\n \n plt2 = plt.gca().twinx()\n \n plt2.plot(TOTAL,history.history['loss'],'-o',label='Train Loss',color='#2ca02c')\n plt2.plot(TOTAL,history.history['val_loss'],'-o',label='Val Loss',color='#d62728')\n \n x = np.argmin( history.history['val_loss'] ); y = np.min( history.history['val_loss'] )\n ydist = plt.ylim()[1] - plt.ylim()[0]\n plt.scatter(x,y,s=200,color='#d62728'); plt.text(x-0.03*xdist,y+0.05*ydist,'min loss\\n%.2f'%y,size=14)\n \n plt.ylabel('Loss',size=14)\n plt.legend(loc=3)\n plt.show()\n \n avg_val_folds.append(dy)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Let's Have a look at some example images of what your interactive training results will look like in weights and biases. Weights and Biases may significantly help you in keeping track of your experiments."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"Image('../input/lyftl5googlecolab/public_run.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 10. Resources\n[Table of contents](#0.1)\n\n* [What is HuBMAP?](https://hubmapconsortium.org/about/)\n* https://cdn.kastatic.org/ka-perseus-images/0e7bfc98302c3e45dc7ec73ab142566a57513ec3.svg\n* https://ohiostate.pressbooks.pub/vethisto/chapter/11-the-glomerulus/\n* https://www.kaggle.com/paulorzp/rle-functions-run-lenght-encode-decode\n* [what is RLE?](https://en.wikipedia.org/wiki/Run-length_encoding)\n* [Image Preprocessing](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/197887)\n* https://www.tensorflow.org/tutorials/load_data/images#using_tfdata_for_finer_control\n* https://www.tensorflow.org/api_docs/python/tf\n* [dice loss](https://www.kaggle.com/marcosnovaes/hubmap-3-unet-models-with-keras-cpu-gpu/notebook)\n* [Plot from chris deotte's notebook](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords)\n* https://www.kaggle.com/dimitreoliveira/flower-with-tpus-advanced-augmentations#Advanced-augmentations\n* [HuBMAP fast.ai starter](https://www.kaggle.com/iafoss/hubmap-pytorch-fast-ai-starter/notebook)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}
![](\"https://ohiostate.pressbooks.pub/app/uploads/sites/36/h5p/content/37/images/file-599206597bdbc.jpg\")
\n![](\"https://cdn.kastatic.org/ka-perseus-images/0e7bfc98302c3e45dc7ec73ab142566a57513ec3.svg\")
\n![\"HuBMAP\"](\"https://www.ml.cmu.edu/news/news-archive/2018/september/research-scientists-will-help-build-3d-cellular-map-of-human-body-machine-learning.jpg\")
"},{"metadata":{},"cell_type":"markdown","source":"[image source](https://www.ml.cmu.edu/news/news-archive/2018/september/research-scientists-will-help-build-3d-cellular-map-of-human-body-machine-learning.html)\n
![](\"https://miro.medium.com/max/536/1*yUd5ckecHjWZf6hGrdlwzA.png\")