{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"Hello fellow Kagglers,\n\nThis notebook demonstrates how MixUp and CutMix augmentations are applied to achieve a cross validation accuracy of >0.90.\n\n\n**[MixUp paper](https://arxiv.org/abs/1710.09412)**\n\nMixUp combines two images, *a* and *b*, into one image. Each pixel is *x%* image *a* and *y%* image b, as is the label. The idea behind this augmentation method is to train the model on images with a label which lies in between two classes.\n\n**[CutMix paper](https://arxiv.org/abs/1905.04899)**\n\nCutMix also combines two images, *a* and *b*, and uses complete parts of different images to create a new image without overlap. For example, the left part of image *a* and the right part of image *b*, this results the label to be 0.50 label *a* + 0.50 label *b*. The idea behind this augmentation method is to roughly the same as with MixUp, train the model on a combination of 2 images with a mixed label. In contrast to MixUp, this method only uses original pixels and applies a regional dropout of the image as only a certain part of the original image is used.\n\n**[GridMask paper](https://arxiv.org/pdf/2001.04086.pdf)**\n\nGridMask is one of many image cutout methods, but it distinguishes itself by using a grid shaped mask, hence the name GridMask. The size of the grid is in a certain range and also positioned over the image with a random top and left offset.\n\nIf these augmentation methods sound abstract, don't worry, examples will be shown in this notebook.\n\nValidation is performed on a stratified kfold for n=5, thus 20% of the training data is used for validation with equal proportions of samples per class for the training and validation dataset.\nAlthough a cross validation accuracy of >0.90 is achieved the leaderboard score is 0.893, test time augmentation and comining multiple models from different fold could improve this score.\n\n\n**V4** Added GridMask and changed the batch size from 64 to 32. Also reduced the number of epochs from 25 to 15. Using only CutMix and GridMask augmentations, as this gave the best results. All together, LB score improved from 0.893 to 0.896.\n\n**V5** Added 2019 Competition data (no duplicates). Improved train dataset speed, increased epochs to 30."},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q --upgrade pip\n!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport tensorflow as tf\nimport tensorflow_probability as tfp\nimport matplotlib.pyplot as plt\nimport efficientnet.tfkeras as efn\nimport seaborn as sns\n\nfrom kaggle_datasets import KaggleDatasets\nfrom tensorflow.keras.mixed_precision import experimental as mixed_precision\nfrom tqdm.notebook import tqdm\nfrom sklearn.metrics import classification_report, confusion_matrix\n\nimport sys\nimport glob\nimport math\nimport gc\nimport time\n\nprint(f'tensorflow version: {tf.__version__}')\nprint(f'tensorflow keras version: {tf.keras.__version__}')\nprint(f'python version: P{sys.version}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TPU and  bfloat16 Configuration\n\nA bfloat16 is a 16 bits float with the range of a 32 bits float, but with a lower precision. Using a bfloat16 instead of a float32 reduces memory consumption and speeds up training and augmentation. The loss in numerical precision is in practice not a problem for machine learning models, as performance, in most cases, won't be affected by a loss of precision after the 3rd decimal number.\n\n**[Some background knowledge on bfloat16](https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus)**"},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n# Detect hardware, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')\n\n# # set half precision policy\nmixed_precision.set_policy('mixed_bfloat16')\n\n# enable XLA optmizations\ntf.config.optimizer.set_jit(True)\n\nprint(f'Compute dtype: {mixed_precision.global_policy().compute_dtype}')\nprint(f'Variable dtype: {mixed_precision.global_policy().variable_dtype}')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"IMG_HEIGHT = 600\nIMG_WIDTH = 800\n\nIMG_SIZE = 600\nIMG_TARGET_SIZE = 512\nN_CHANNELS = 3\n\nN_TRAIN_IMGS = 21642\nN_VAL_IMGS = 5410\nBATCH_SIZE_VAL = 64 * REPLICAS # 5410 / 8 / 4\n\nN_LABELS = 5\nN_FOLDS = 5\nEPOCHS = 30\n\nBATCH_SIZE_BASE = 16\nBATCH_SIZE = BATCH_SIZE_BASE * REPLICAS\n\nTARGET_DTYPE = tf.bfloat16\n\n# ImageNet mean and standard deviation\nIMAGENET_MEAN = tf.constant([0.485, 0.456, 0.406], dtype=tf.float32)\nIMAGENET_STD = tf.constant([0.229, 0.224, 0.225], dtype=tf.float32)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_DS_PATH = KaggleDatasets().get_gcs_path('cassava-leaf-disease-tfrecords-600x600')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train Dataset\n\nA public dataset is used where jpegs are combioned into TFRecords. This allows for a faster data pipeline, as images do not have to be read one-by-one, but 1024 at a time. The original jpegs are used for data augmentation purposes. The original images are 800\\*600 pixels and each epoch a random square is used.\n\nV5: Image height and width is added as 2019 competition images have a wide variety of resolutions and Tensorflow needs an explicit shape."},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_tfrecord_train(record_bytes):\n    features = tf.io.parse_single_example(record_bytes, {\n        'image': tf.io.FixedLenFeature([], tf.string),\n        'label': tf.io.FixedLenFeature([], tf.int64),\n        'height': tf.io.FixedLenFeature([], tf.int64),\n        'width': tf.io.FixedLenFeature([], tf.int64),\n    })\n    \n    height = features['height']\n    width = features['width']\n\n    image = tf.io.decode_jpeg(features['image'])\n    image = tf.reshape(image, [height, width, N_CHANNELS])\n    \n    # get random square\n    if height > width:\n        offset = tf.random.uniform(shape=(), minval=0, maxval=height-width, dtype=tf.int64)\n        image = tf.slice(image, [offset, 0, 0], [width, width, N_CHANNELS])\n    elif width > height:\n        offset = tf.random.uniform(shape=(), minval=0, maxval=width-height, dtype=tf.int64)\n        image = tf.slice(image, [0, offset, 0], [height, height, N_CHANNELS])\n    else:\n        image = tf.slice(image, [0, 0, 0], [height, width, N_CHANNELS])\n        \n    size = tf.cast(height if height < width else width, tf.float32)\n    \n    # cast label to int8\n    label = tf.cast(features['label'], tf.uint8)\n\n    return image, label, size","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# chance of x in y to return true, used for conditional data augmentation\ndef chance(x, y):\n    return tf.random.uniform(shape=[], minval=0, maxval=y, dtype=tf.int32) < x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def augment_image(image, label, size):\n    # random flip image horizontally\n    image = tf.image.random_flip_left_right(image)\n    # random flip image vertically\n    image = tf.image.random_flip_up_down(image)\n    \n    # random transpose\n    if chance(1,2):\n        image = tf.image.transpose(image)\n    \n    # random crop between 75%-100%\n    crop_size = tf.random.uniform(shape=(), minval=size*0.75, maxval=size)\n    image = tf.image.random_crop(image, [crop_size, crop_size, N_CHANNELS])\n    \n    # cast to target dtype and resize\n    image = tf.image.resize(image, [IMG_TARGET_SIZE, IMG_TARGET_SIZE])\n    \n    # normalize according to imagenet mean and std\n    image /= 255.0\n    image = (image - IMAGENET_MEAN) / IMAGENET_STD\n    \n    # one hot encode label\n    label = tf.one_hot(label, N_LABELS, dtype=tf.float32)\n    \n    return image, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_augment_image(record_bytes):\n    image, label, size = decode_tfrecord_train(record_bytes)\n    image, label = augment_image(image, label, size)\n    \n    return image, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This next function returns a random index from another image in the batch. This method performs better than using a random image with another label. This could be due to the fact the class inbalance is changed by choosing an image from another class. Images from the most dominant class, class 3, will be always mixed with an image from another class, thus the other classes will be more present in the training data."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_mix_img_idx(labels_idxs, idx):\n    idx_candidates = tf.where(labels_idxs != idx)\n    r = tf.random.uniform(minval=0, maxval=len(idx_candidates), shape=[], dtype=tf.int32)\n    idx = tf.gather(idx_candidates, r)\n    idx = tf.cast(idx, tf.int32)\n    idx = tf.squeeze(idx)\n    \n    return idx","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MixUp implementation"},{"metadata":{"trusted":true},"cell_type":"code","source":"def mixup(images, labels, alpha=0.40):\n    l = len(images)\n    # get image factors\n    a = tfp.distributions.Beta(alpha, alpha).sample(l)\n    a_label = tf.reshape(a, shape=(l,1))\n    a_label = tf.tile(a_label, [1, N_LABELS])\n    b_label = 1 - a_label\n    \n    a_image = tf.reshape(a, shape=(l,1,1,1))\n    a_image = tf.tile(a_image, [1, IMG_TARGET_SIZE, IMG_TARGET_SIZE ,N_CHANNELS])\n    a_image = tf.cast(a_image, tf.float32)\n    b_image = 1 - a_image\n    \n    # get mixup image indices\n    if l == 2:\n        idxs = tf.constant([1, 0])\n    else:\n        labels_idxs = tf.range(len(labels))\n        idxs = tf.map_fn(lambda idx: get_mix_img_idx(labels_idxs, idx), tf.range(len(labels)))\n    \n    images_mixup = tf.gather(images, idxs)\n    labels_mixup = tf.gather(labels, idxs)\n    \n    # mixup images and labels\n    images =  images * a_image + images_mixup * b_image\n    labels = labels * a_label + labels_mixup * b_label\n    \n    images = tf.cast(images, TARGET_DTYPE)\n    \n    return images, labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# CutMix implementation\n\nThis cutmix implementation differs from the one in version 3. This implementation is closer to the method described in the paper. A random mask whose size is defined by the beta destribution with $Beta(\\alpha, \\alpha)|\\alpha=1$. The original image will be masked with another image, the label will be updated accordingly to label and size of the masking image."},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_cutmix_mask(a):\n    # create random mask size and coordinates\n    r_w = tf.cast(IMG_TARGET_SIZE * tf.math.sqrt(1 - a), tf.int32)\n    r_h = tf.cast(IMG_TARGET_SIZE * tf.math.sqrt(1 - a), tf.int32)\n    \n    if r_w == IMG_TARGET_SIZE:\n        r_x = 0\n    else:\n        r_x = tf.random.uniform(minval=0, maxval=IMG_TARGET_SIZE - r_w, shape=[], dtype=tf.int32)\n        \n    if r_h == IMG_TARGET_SIZE:\n        r_y = 0\n    else:\n        r_y = tf.random.uniform(minval=0, maxval=IMG_TARGET_SIZE - r_w, shape=[], dtype=tf.int32)\n\n    # compute padding sizes\n    pad_left = r_x\n    pad_right = IMG_TARGET_SIZE - (r_x + r_w)\n    pad_top = r_y\n    pad_bottom = IMG_TARGET_SIZE - (r_y + r_h)\n    \n    # create mask_a and mask_b\n    mask_a = tf.ones(shape=[r_w, r_h], dtype=tf.float32)\n    mask_a = tf.pad(mask_a, [[pad_left, pad_right], [pad_top, pad_bottom]], mode='CONSTANT', constant_values=0)\n    mask_a = tf.expand_dims(mask_a, axis=2)\n    \n    return mask_a\n\ndef cutmix(images, labels):\n    l = len(images)\n    a_float32 = tfp.distributions.Beta(1.0, 1.0).sample([l])\n\n    mask_b = tf.map_fn(create_cutmix_mask, a_float32)\n    mask_a = tf.math.abs(mask_b - 1)\n    \n    # images_idxs\n    if l == 2:\n        idxs = tf.constant([1, 0])\n    else:\n        labels_idxs = tf.range(len(labels))\n        idxs = tf.map_fn(lambda idx: get_mix_img_idx(labels_idxs, idx), tf.range(len(labels)))\n    \n    images_cutmix = tf.gather(images, idxs)\n    labels_cutmix = tf.gather(labels, idxs)\n    \n    a_float32_labels = tf.expand_dims(a_float32, axis=1)\n    a_float32_labels = tf.repeat(a_float32_labels, N_LABELS, axis=1)\n    labels_factor = a_float32_labels\n    labels_cutmix_factor = 1 - a_float32_labels\n    \n    # cutmix images and labels\n    images = images * mask_a + images_cutmix * mask_b\n    labels = labels * labels_factor + labels_cutmix * labels_cutmix_factor\n    \n    images = tf.cast(images, TARGET_DTYPE)\n    \n    return images, labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# GridMask implementation"},{"metadata":{"trusted":true},"cell_type":"code","source":"def gridmask(images, labels):\n    l = len(images)\n    \n    d = tf.random.uniform(minval=int(IMG_TARGET_SIZE * (96/224)), maxval=IMG_TARGET_SIZE, shape=[], dtype=tf.int32)\n    grid = tf.constant([[[0], [1]],[[1], [0]]], dtype=tf.float32)\n    grid = tf.image.resize(grid, [d, d], method='nearest')\n    \n    # 50% chance to rotate mask\n    if chance(1, 2):\n        grid = tf.image.rot90(grid, 1)\n\n    repeats = IMG_TARGET_SIZE // d + 1\n    grid = tf.tile(grid, multiples=[repeats, repeats, 1])\n    grid = tf.image.random_crop(grid, [IMG_TARGET_SIZE, IMG_TARGET_SIZE, 1])\n    grid = tf.expand_dims(grid, axis=0)\n    grid = tf.tile(grid, multiples=[l, 1, 1, 1])\n\n    images = images * grid\n    images = tf.cast(images, TARGET_DTYPE)\n    \n    return images, labels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def augment_batch(images, labels, augmentations=None):\n    if augmentations is None:\n        r = tf.random.uniform(minval=0, maxval=4, shape=[], dtype=tf.int32)\n    else:\n        r = tf.random.uniform(minval=0, maxval=len(augmentations), shape=[], dtype=tf.int32)\n        r = tf.gather(augmentations, r)\n        \n    if r == 0:\n        images = tf.cast(images, TARGET_DTYPE)\n        return images, labels\n    elif r == 1:\n        return mixup(images, labels)\n    elif r == 2:\n        return cutmix(images, labels)\n    elif r == 3:\n        return gridmask(images, labels)\n    else:\n        images = tf.cast(images, TARGET_DTYPE)\n        return images, labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Reshaping the batch from [x,y,...] to [x*y,...]. This greatly improves the dataset speed in comparison with unbatching and batching."},{"metadata":{"trusted":true},"cell_type":"code","source":"def reshape_batch(images, labels):\n    images = tf.reshape(images, shape=[BATCH_SIZE, IMG_TARGET_SIZE, IMG_TARGET_SIZE, N_CHANNELS])\n    labels = tf.reshape(labels, shape=[BATCH_SIZE, N_LABELS])\n    \n    random_idxs = tf.random.shuffle(tf.range(BATCH_SIZE))\n    images = tf.gather(images, random_idxs)\n    labels = tf.gather(labels, random_idxs)\n    \n    return images, labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"V3: Improved dataset pipeline speed by adding a prefetch for the TFRecords samples and a static number of parallel calls for the batch augmentations\n\nV4: Using a reshape map instead of unbatch and batch, greatly improves throughput by ~30%"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_train_dataset(bs=BATCH_SIZE, fold=0, augmentations=None):\n    ignore_order = tf.data.Options()\n    ignore_order.experimental_deterministic = False\n    \n    FNAMES_TRAIN_TFRECORDS = tf.io.gfile.glob(f'{GCS_DS_PATH}/fold_{fold}/train/*.tfrecords')\n    train_dataset = tf.data.TFRecordDataset(FNAMES_TRAIN_TFRECORDS, num_parallel_reads=AUTO)\n    train_dataset = train_dataset.with_options(ignore_order)\n    train_dataset = train_dataset.prefetch(AUTO)\n    train_dataset = train_dataset.repeat()\n    train_dataset = train_dataset.map(read_augment_image, num_parallel_calls=AUTO)\n\n    train_dataset = train_dataset.batch(BATCH_SIZE_BASE)\n    train_dataset = train_dataset.map(lambda images, labels: augment_batch(images, labels, augmentations=augmentations), num_parallel_calls=REPLICAS)\n    \n    train_dataset = train_dataset.batch(REPLICAS)\n    train_dataset = train_dataset.map(reshape_batch, num_parallel_calls=1)\n    \n    train_dataset = train_dataset.prefetch(1)\n    \n    return train_dataset\n\ntrain_dataset = get_train_dataset()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def benchmark(num_epochs=3, n_steps_per_epoch=10, augmentations=None, bs=BATCH_SIZE):\n    dataset = get_train_dataset(augmentations=augmentations)\n    start_time = time.perf_counter()\n    for epoch_num in range(num_epochs):\n        epoch_start = time.perf_counter()\n        for idx, (images, labels) in enumerate(dataset.take(n_steps_per_epoch)):\n            if idx is 1:\n                print(images.shape, labels.shape)\n            pass\n        print(f'epoch {epoch_num} took: {round(time.perf_counter() - epoch_start, 2)}')\n    print(\"Execution time:\", round(time.perf_counter() - start_time, 2))\n    \nbenchmark(num_epochs=3, augmentations=[2,3])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Batch Example\n\nThe next function plots examples of the final augmented images. The title of each image shows the RGB and label information. Note all possible augmentations are shown here, only CutMix and GridMask are used for training."},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_first_train_batch(augmentations=None, rows=4, cols=4, print_info=False):\n    # log info of batch and first few train images\n    imgs, lbls = next(iter(get_train_dataset(augmentations=augmentations)))\n    if print_info:\n        print(f'Number of train images: {N_TRAIN_IMGS}')\n        print(f'imgs.shape: {imgs.shape}, images.dtype: {imgs.dtype}, lbls.shape: {lbls.shape}, lbls.dtype: {lbls.dtype}')\n        img0 = imgs[0].numpy().astype(np.float32)\n        print('img0 mean: {:.3f}, img0 std {:.3f}, img0 min: {:.3f}, img0 max: {:.3f}'.format(img0.mean(), img0.std(), img0.min(), img0.max()))\n        print(f'first label: {lbls[0]}')\n\n    fig, axes = plt.subplots(nrows=rows, ncols=cols, figsize=(cols*6, rows*6))\n    for r in range(rows):\n        for c in range(cols):\n            img = imgs[r*rows+c].numpy().astype(np.float32)\n            lbl = lbls[r*rows+c].numpy().astype(np.float32).tolist()\n            \n            # add title with image information\n            lbl_str = '[' + ', '.join(['%.3f' % i for  i in lbl]) + ']'\n            axes[r, c].set_title('mean: {:.3f}, std {:.3f}, min: {:.3f}, max: {:.3f}\\n label: {}'.format(img.mean(), img.std(), img.min(), img.max(), lbl_str))\n            axes[r, c].axhline(y=IMG_TARGET_SIZE // 2, color='r')\n            axes[r, c].axvline(x=IMG_TARGET_SIZE // 2, color='r')\n            \n            img += abs(img.min())\n            img /= img.max()\n            axes[r, c].imshow(img)\n            \nshow_first_train_batch(augmentations=[0,1,2,3])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# MixUp examples\nshow_first_train_batch(augmentations=[1], rows=2, cols=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# CutMix examples\nshow_first_train_batch(augmentations=[2], rows=2, cols=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# GridMask examples\nshow_first_train_batch(augmentations=[3], rows=2, cols=3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This function shows the per image augmentation. This is the augmentation before CutMix or MixUp is applied ans shows how images can differ each epoch."},{"metadata":{"trusted":true},"cell_type":"code","source":"def resize_image(image, label, size):\n    image = tf.image.resize(image, [IMG_TARGET_SIZE, IMG_TARGET_SIZE])\n    \n    return image, label, tf.cast(IMG_TARGET_SIZE, tf.float32)\n\ndef show_data_augmentations():\n    FNAMES_TRAIN_TFRECORDS = tf.io.gfile.glob(f'{GCS_DS_PATH}/fold_0/train/*.tfrecords')\n    dataset = tf.data.TFRecordDataset(FNAMES_TRAIN_TFRECORDS)\n    dataset = dataset.map(decode_tfrecord_train)\n    dataset = dataset.map(resize_image)\n    dataset = dataset.batch(BATCH_SIZE)\n\n    imgs, lbls, szs = next(iter(dataset))\n    \n    # to test data augmentation\n    rows, cols = 4, 4\n    fig, axes = plt.subplots(nrows=rows, ncols=cols, figsize=(cols*6, rows*6))\n    for r in range(rows):\n        for c in range(cols):\n            img, _ = augment_image(imgs[15], -1, szs[15])\n            img = img.numpy().astype(np.float32)\n            \n            # add title with image information\n            axes[r, c].set_title('mean: {:.3f}, std {:.3f}, min: {:.3f}, max: {:.3f}'.format(img.mean(), img.std(), img.min(), img.max()))\n            \n            img += abs(img.min())\n            img /= img.max()\n            \n            axes[r, c].imshow(img)\n                \nshow_data_augmentations()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Validation Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_tfrecord_val(record_bytes):\n    features = tf.io.parse_single_example(record_bytes, {\n        'image': tf.io.FixedLenFeature([], tf.string),\n        'label': tf.io.FixedLenFeature([], tf.int64),\n        'height': tf.io.FixedLenFeature([], tf.int64),\n        'width': tf.io.FixedLenFeature([], tf.int64),\n    })\n    \n    height = features['height']\n    width = features['width']\n\n    image = tf.io.decode_jpeg(features['image'])\n    image = tf.reshape(image, [height, width, N_CHANNELS])\n    \n    # get random square\n    if height > width:\n        offset = (height - width) // 2\n        image = tf.slice(image, [offset, 0, 0], [width, width, N_CHANNELS])\n    elif width > height:\n        offset = (width - height) // 2\n        image = tf.slice(image, [0, offset, 0], [height, height, N_CHANNELS])\n    else:\n        image = tf.slice(image, [0, 0, 0], [height, width, N_CHANNELS])\n    \n    # resize to target size\n    image = tf.image.resize(image, [IMG_TARGET_SIZE, IMG_TARGET_SIZE])\n    \n    # normalize according to imagenet mean and std\n    image /= 255.0\n    image = (image - IMAGENET_MEAN) / IMAGENET_STD\n    \n    # cast to TARGET_DTYPE\n    image = tf.cast(image, TARGET_DTYPE)\n    \n    label = tf.cast(features['label'], tf.int32)\n    \n    # one hot encode label\n    label = tf.one_hot(label, N_LABELS, dtype=tf.int32)\n    \n    return image, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_val_dataset(bs=BATCH_SIZE, fold=0):\n    FNAMES_VAL_TFRECORDS = tf.io.gfile.glob(f'{GCS_DS_PATH}/fold_{fold}/val/*.tfrecords')\n    val_dataset = tf.data.TFRecordDataset(FNAMES_VAL_TFRECORDS, num_parallel_reads=AUTO)\n    val_dataset = val_dataset.prefetch(BATCH_SIZE_VAL)\n    val_dataset = val_dataset.repeat()\n    val_dataset = val_dataset.map(decode_tfrecord_val, num_parallel_calls=AUTO)\n    val_dataset = val_dataset.batch(bs, drop_remainder=True)\n    val_dataset = val_dataset.prefetch(1)\n    \n    return val_dataset\n\nval_dataset = get_val_dataset()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Show batch info and first few test images\ndef show_first_val_batch():\n    imgs, lbls = next(iter(val_dataset))\n    \n    print(f'Number of val images: {N_VAL_IMGS}')\n    print(f'imgs.shape: {imgs.shape}, images.dtype: {imgs.dtype}, lbls.shape: {lbls.shape}, lbls.dtype: {lbls.dtype}')\n    img0 = imgs[0].numpy().astype(np.float32)\n    print('img0 mean: {:.3f}, img0 std {:.3f}, img0 min: {:.3f}, img0 max: {:.3f}'.format(img0.mean(), img0.std(), img0.min(), img0.max()))\n    print(f'first label: {lbls[0]}')\n\n    rows, cols = 4, 5\n    fig, axes = plt.subplots(nrows=rows, ncols=cols, figsize=(cols*6, rows*6))\n    for r in range(rows):\n        for c in range(cols):\n            img = imgs[r * (rows + 1) + c].numpy().astype(np.float32)\n            \n            # add title with image information\n            axes[r, c].set_title('mean: {:.3f}, std {:.3f}, min: {:.3f}, max: {:.3f}'.format(img.mean(), img.std(), img.min(), img.max()))\n            \n            img += abs(img.min())\n            img /= img.max()\n\n            axes[r, c].imshow(img)\n            \nshow_first_val_batch()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Learning Rate Scheduler\n\nThe learning rate used is a exponential warmup with cosine decay. The warmup is used to prevent the model from early overfitting on the first images. When the model starts learning the loss will be high as the model is trained on ImageNet, not on the training dataset. When starting with a high learning rate the model will learn the first few batches very well due to the high loss and could overfit on those samples. When starting with a very low learning rate the model will see all training images and make small adjustment to the weights and therefore learn from all training images equally when the loss is high and weights are modified strongly."},{"metadata":{"trusted":true},"cell_type":"code","source":"def lrfn(epoch, bs=BATCH_SIZE, epochs=EPOCHS):\n    # Config\n    LR_START = 1e-6\n    LR_MAX = 2e-4\n    LR_FINAL = 1e-6\n    LR_RAMPUP_EPOCHS = 4\n    LR_SUSTAIN_EPOCHS = 0\n    DECAY_EPOCHS = epochs  - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS - 1\n    LR_EXP_DECAY = (LR_FINAL / LR_MAX) ** (1 / (EPOCHS - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS - 1))\n\n    if epoch < LR_RAMPUP_EPOCHS: # exponential warmup\n        lr = LR_START + (LR_MAX + LR_START) * (epoch / LR_RAMPUP_EPOCHS) ** 2.5\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS: # sustain lr\n        lr = LR_MAX\n    else: # cosine decay\n        epoch_diff = epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS\n        decay_factor = (epoch_diff / DECAY_EPOCHS) * math.pi\n        decay_factor= (tf.math.cos(decay_factor).numpy() + 1) / 2        \n        lr = LR_FINAL + (LR_MAX - LR_FINAL) * decay_factor\n\n    return lr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plots the learning rate schedule\ndef show_lr_schedule(bs=BATCH_SIZE, epochs=EPOCHS):\n    rng = [i for i in range(epochs)]\n    y = [lrfn(x, bs=bs, epochs=epochs) for x in rng]\n    x = np.arange(epochs)\n    x_axis_labels = list(map(str, np.arange(1, epochs+1)))\n    print('init lr {:.1e} to {:.1e} final {:.1e}'.format(y[0], max(y), y[-1]))\n    \n    plt.figure(figsize=(30, 10))\n    plt.xticks(x, x_axis_labels, fontsize=16) # set tick step to 1 and let x axis start at 1\n    plt.yticks(fontsize=16)\n    plt.plot(rng, y)\n    plt.grid()\n    plt.show()\n    \nshow_lr_schedule()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_model():\n    # reset to free memory and training variables\n    tf.keras.backend.clear_session()\n    \n    with strategy.scope():\n        \n        net = efn.EfficientNetB5(\n            include_top=False,\n            weights='noisy-student',\n            input_shape=(IMG_TARGET_SIZE, IMG_TARGET_SIZE, N_CHANNELS),\n        )\n        \n        for layer in reversed(net.layers):\n            if isinstance(layer, tf.keras.layers.BatchNormalization):\n                layer.trainable = False\n            else:\n                layer.trainable = True\n        \n        model = tf.keras.Sequential([\n            net,\n            tf.keras.layers.Dropout(0.45),\n            tf.keras.layers.GlobalAveragePooling2D(),\n            tf.keras.layers.Dropout(0.45),\n            tf.keras.layers.Dense(N_LABELS, activation='softmax', dtype=tf.float32),\n        ])\n\n        # add metrics\n        metrics = [\n            tf.keras.metrics.CategoricalAccuracy(name='accuracy'),\n            tf.keras.metrics.TopKCategoricalAccuracy(k=2, name='top_2_accuracy'),\n        ]\n\n        optimizer = tf.keras.optimizers.Adam()\n        loss = tf.keras.losses.CategoricalCrossentropy()\n\n        model.compile(optimizer=optimizer, loss=loss, metrics=metrics)\n\n        return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Validation Report"},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_validation_report_per_class(model, dataset, steps, name, bs):\n    print(f'--- {name} REPORT ---')\n    # classification report\n    y = np.ndarray(shape=steps * bs, dtype=np.uint16)\n    y_pred = np.ndarray(shape=steps * bs, dtype=np.uint16)\n    for idx, (images, labels) in tqdm(enumerate(dataset.take(steps)), total=steps):\n        with tf.device('cpu:0'):\n            y[idx*bs:(idx+1)*bs] = np.argmax(labels, axis=1)\n            y_pred[idx*bs:(idx+1)*bs] = np.argmax(model.predict(images).astype(np.float32), axis=1)\n            \n    print(classification_report(y, y_pred))\n    \n    # Confusion matrix\n    fig, ax = plt.subplots(1, 1, figsize=(20, 12))\n    cfn_matrix = confusion_matrix(y, y_pred, labels=range(N_LABELS))\n    cfn_matrix = (cfn_matrix.T / cfn_matrix.sum(axis=1)).T\n    df_cm = pd.DataFrame(cfn_matrix, index=np.arange(N_LABELS), columns=np.arange(N_LABELS))\n    ax = sns.heatmap(df_cm, cmap='Blues', annot=True, fmt='.3f', linewidths=.5, annot_kws={'size':14}).set_title(f'{name} CONFUSION MATRIX')\n    plt.xticks(fontsize=16)\n    plt.yticks(fontsize=16)\n    plt.xlabel('PREDICTED', fontsize=24, labelpad=10)\n    plt.ylabel('ACTUAL', fontsize=24, labelpad=10)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training History"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_history_metric(history, metric):\n    TRAIN_EPOCHS = len(history.history['loss'])\n    x = np.arange(TRAIN_EPOCHS)\n    x_axis_labels = list(map(str, np.arange(1, TRAIN_EPOCHS+1)))\n    val = 'val' in ''.join(history.history.keys())\n    # summarize history for accuracy\n    plt.figure(figsize=(20, 10))\n    plt.plot(history.history[metric])\n    if val:\n        plt.plot(history.history[f'val_{metric}'])\n    \n    plt.title(f'Model {metric}', fontsize=24)\n    plt.ylabel(metric, fontsize=20)\n    plt.yticks(fontsize=16)\n    plt.xlabel('epoch', fontsize=20)\n    plt.xticks(x, x_axis_labels, fontsize=16) # set tick step to 1 and let x axis start at 1\n    plt.legend(['train'] + ['test'] if val else [], loc='upper left')\n    plt.grid()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training\n\nThis is the training loop, the training metrics and confusion matrix are displayed after each fold.\n\nFrom the validation report it can be observed the model predicts label 3 with very high precision and accuracy. This is not surprising as label 3 is by far the most common label and the model will therefore most likely get biased towards this label. Label 0 is the least common label and also has the lowest precision and recall. Label 0 is more than 10 times less common than label 3, making the dataset highly unbalanced.\n\nThe confusion matrix shows how the model mixes up labels. Label 0 is mostly confused with label 4 and 1. Moreover, label 2 is often confused with label 3."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'TRAINING FOR {EPOCHS} EPOCHS WITH BATCH SIZE {BATCH_SIZE}\\n')\nprint(f'TRAIN IMAGES: {N_TRAIN_IMGS}, VAL IMAGES: {N_VAL_IMGS}\\n')\n\naugmentations_dic = dict({\n    0: 'None',\n    1: 'MixUp',\n    2: 'CutMix',\n    3: 'GridMask',\n})\n\nMEAN_VAL_ACC = []\naugmentations = [2,3] # only CutMix and GridMask is used\nfold = 0\nepochs = EPOCHS\n\nfor idx, fold in enumerate(range(N_FOLDS)):\n    # callbacks\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lambda epoch: lrfn(epoch, epochs=epochs), verbose=1)\n    show_lr_schedule(epochs=epochs)\n    \n    # get the model\n    model = get_model()\n    \n    if idx is 0:\n        # model summary\n        model.summary()\n        # compute and variable data types\n        print(f'Compute dtype: {mixed_precision.global_policy().compute_dtype}')\n        print(f'Variable dtype: {mixed_precision.global_policy().variable_dtype}')\n        \n    print('\\n')\n    print('*'*25, f'augmentations {augmentations}', '*'*25, '\\n')\n    print(f'fold: {fold}, epochs: {epochs}')\n    print(' AND '.join([augmentations_dic.get(i) for i in augmentations]), '\\n')\n    \n    train_dataset = get_train_dataset(bs=BATCH_SIZE, fold=fold, augmentations=augmentations)\n    val_dataset = get_val_dataset(bs=BATCH_SIZE_VAL, fold=fold)\n    \n    history = model.fit(\n        train_dataset,\n        steps_per_epoch = N_TRAIN_IMGS // BATCH_SIZE,\n\n        validation_data = val_dataset,\n        validation_steps = N_VAL_IMGS // BATCH_SIZE_VAL,\n\n        epochs = epochs,\n        callbacks = [\n            lr_callback,\n        ],\n        verbose=1,\n    )\n    \n    # add val accuracy to list\n    MEAN_VAL_ACC.append(history.history['val_accuracy'][-1])\n    \n    # plot training histories\n    plot_history_metric(history, 'loss')\n    plot_history_metric(history, 'accuracy')\n    plot_history_metric(history, 'top_2_accuracy')\n    \n    # show train and validation report\n    show_validation_report_per_class(model, val_dataset, N_VAL_IMGS // BATCH_SIZE_VAL, 'VALIDATION', BATCH_SIZE_VAL)\n\n    # save the model\n    model.save_weights(f'model_fold_{fold}_weights.h5')\n    \n    del model, train_dataset, val_dataset\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'OOF validation accuracy: {np.array(MEAN_VAL_ACC).mean()}')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}