{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Random Erasing/CutOut, MixUp and CutMix Comparison using TPU","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In this notebook I want to explore three extremely interesting method of data augmentation that I have recently used in [Plant Pathology 2020 - FGVC7](https://www.kaggle.com/c/plant-pathology-2020-fgvc7) competition: **Random Erasing**, **MixUp** and **CutMix**.\n\nThe goal of this notebook is to do some experimentation using these three methods on TPU, comparing the results obtained using a simple model based on EfficientNetB0 network architecture on [Flower Classification with TPUs](https://www.kaggle.com/c/flower-classification-with-tpus) dataset.\n\nThis notebook is not intended for providing general results but it gives only a glimpse about the usefulness of some recent data augmentation methods in the context of image recognition.\n\nI have tried to implement Random Erasing, MixUp and CutMix functions from scratch starting from the description contained in the original papers (section \"Related Papers and Links\") but I have also referred to two two excellent notebooks from *Martin Görner* and *Chris Deotte* that I reccomend to read and upvote:\n* [Getting started with 100+ flowers on TPU](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu) contains all the explanation about how to use TPUs on Kaggle with lots of helper function specific for the flower dataset.\n* [CutMix and MixUp on GPU/TPU](https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu) contains excellent Tensorflow implementations of MixUp and CutMix from which I took some ideas and tricks, the main one was how to cope with the difficulty in Tensorflow 2.2 of modifying directly some entries in a tensor.\n\nFor understanding how to organize the notebook (this is my first one!) I took ispiration from some notebooks by *Aleksandra Deis* that are always perfectly organized.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Libraries","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import math, re, os, gc\nfrom os import listdir\nfrom os.path import isfile, join\n\nimport tensorflow as tf\nimport numpy as np\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nfrom matplotlib import pyplot as plt\nfrom matplotlib.ticker import PercentFormatter\nplt.rcParams['figure.figsize'] = [20, 10]\n#plt.style.use('seaborn-deep')\n\n# numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)\n\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\nprint(\"Tensorflow version \" + tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# EfficientNet\n!pip install efficientnet\nimport efficientnet.tfkeras as efn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TPU Configuration","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Parameters","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_DS_PATH = KaggleDatasets().get_gcs_path() # you can list the bucket with \"!gsutil ls $GCS_DS_PATH\"\n\nAUTO = tf.data.experimental.AUTOTUNE\n\n# Seed for random number generation\nSEED = 42\n\n# Image size\nIMAGE_SIZE = [224, 224] \nN_CHANNELS = 3\n\n# Data path\nGCS_PATH_SELECT = {192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',  # available image sizes\n                   224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n                   331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n                   512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'}\n\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\n# Training parameters\nMODEL_NAME = 'effb0'\nEPOCHS = 50\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n\nN_ENSEMBLE = 2  # number of identical models to train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In order to perform the analisis on a labelled test set, I have split the original validation set in two subsets each containing rougly 15% of the labelled data. Finally, I used these two new sets for validation and test.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\n# Split original validation set in two new disjoint subset for validation and test.\nVALIDATION_TEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nVALIDATION_FILENAMES = VALIDATION_TEST_FILENAMES[:8]\nTEST_FILENAMES = VALIDATION_TEST_FILENAMES[8:]\n\n\nCLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 102","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualization Utilities","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def mixed_label(x):\n    ind = [i for i in range(x.shape[0]) if x[i] > 0]\n    if len(ind) == 1:\n        mixed_lab = CLASSES[ind[0]]\n    else:\n        mixed_lab = str(np.round(x[ind[0]], 3)) + ' ' + CLASSES[ind[0]] + '\\n' + str(np.round(x[ind[1]], 3)) + ' ' + CLASSES[ind[1]]\n    try:\n        foo = mixed_lab\n    except:\n        foo = 'ERROR!'\n    return foo\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 15.0 #13.0\n    SPACING = 0.2 #0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        try:\n            title = '' if label is None else CLASSES[label]\n        except:\n            title = mixed_label(label)\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        #dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        dynamic_titlesize = 13.0*0.1/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n\ndef display_confusion_matrix(cmat, score, precision, recall):\n    plt.figure(figsize=(15,15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def weights_calc(labels):\n    \"\"\"\n    Utility function for histogram weights calculation.\n    \"\"\"\n    return np.ones(len(labels)) / (len(labels))\n\n\ndef autolabel(rects):\n    \"\"\"\n    Utility function for attaching a text label above each bar in rects.\n    Credits: https://matplotlib.org/3.1.1/gallery/lines_bars_and_markers/barchart.html\n    \"\"\"\n    for rect in rects:\n        height = rect.get_height()\n        ax.annotate('{}'.format(np.round(height, 3)),\n                    xy=(rect.get_x() + rect.get_width() / 2, height),\n                    xytext=(0, 3),  # 3 points vertical offset\n                    textcoords=\"offset points\",\n                    ha='center', va='bottom', fontsize=16)\n\n        \ndef plot_metric(baseline, agumented_0, agumented_1, agumented_2, metric='loss', aug_names=['NoAugmentation', 'RandomErasing', 'MixUp', 'CutMix']):\n    \n    if metric == 'loss':\n        legend_location = 'upper right'\n        y_min = 0.0\n        y_max = 2.81\n        y_step = 0.2\n    else: \n        legend_location = 'lower right'\n        y_min = 0.4\n        y_max = 1.01\n        y_step = 0.05\n    \n    plt.subplot(2, 2, 1)\n    plt.plot(baseline[metric])\n    plt.plot(baseline['val_' + metric])\n    plt.title('Model ' + metric.capitalize() + ' - ' + aug_names[0], fontsize=18)\n    plt.ylabel(metric.capitalize(), fontsize=16)\n    plt.xlabel('Epoch', fontsize=16)\n    plt.xticks(np.arange(0, 51, step=5))\n    plt.yticks(np.arange(y_min, y_max, step=y_step))\n    plt.legend(['Train', 'Val'], loc=legend_location, fontsize=12)\n    plt.grid(axis='y')\n\n\n    plt.subplot(2, 2, 2)\n    plt.plot(agumented_0[metric])\n    plt.plot(agumented_0['val_' + metric])\n    plt.title('Model ' + metric.capitalize() + ' - ' + aug_names[1], fontsize=18)\n    plt.ylabel(metric.capitalize(), fontsize=16)\n    plt.xlabel('Epoch', fontsize=16)\n    plt.xticks(np.arange(0, 51, step=5))\n    plt.yticks(np.arange(y_min, y_max, step=y_step))\n    plt.legend(['Train', 'Val'], loc=legend_location, fontsize=12)\n    plt.grid(axis='y')\n    \n    plt.subplot(2, 2, 3)\n    plt.plot(agumented_1[metric])\n    plt.plot(agumented_1['val_' + metric])\n    plt.title('Model ' + metric.capitalize() + ' - ' + aug_names[2], fontsize=18)\n    plt.ylabel(metric.capitalize(), fontsize=16)\n    plt.xlabel('Epoch', fontsize=16)\n    plt.xticks(np.arange(0, 51, step=5))\n    plt.yticks(np.arange(y_min, y_max, step=y_step))\n    plt.legend(['Train', 'Val'], loc=legend_location, fontsize=12)\n    plt.grid(axis='y')\n\n\n    plt.subplot(2, 2, 4)\n    plt.plot(agumented_2[metric])\n    plt.plot(agumented_2['val_' + metric])\n    plt.title('Model ' + metric.capitalize() + ' - ' + aug_names[3], fontsize=18)\n    plt.ylabel(metric.capitalize(), fontsize=16)\n    plt.xlabel('Epoch', fontsize=16)\n    plt.xticks(np.arange(0, 51, step=5))\n    plt.yticks(np.arange(y_min, y_max, step=y_step))\n    plt.legend(['Train', 'Val'], loc=legend_location, fontsize=12)\n    plt.grid(axis='y')\n\n    plt.subplots_adjust(wspace=0.2, hspace=0.35)\n    plt.show()\n    \n\ndef plot_loss_error(baseline, agumented, augmentation_name='', set_type='validation'):\n    \"\"\"\n    Example: plot_loss_error(hist_no_augmentation[0].history, hist_random_erasing[0].history, augmentation_name='Random Erasing', set_type='validation')\n    \"\"\"\n    \n    if set_type == 'validation':\n        loss = 'val_loss'\n        acc = 'val_categorical_accuracy'\n    else:\n        loss = 'loss'\n        acc = 'categorical_accuracy'\n        \n    plt.subplot(1, 2, 1)\n    plt.plot(baseline[loss])\n    plt.plot(agumented[loss])\n    locs, labels = plt.yticks()\n    plt.yticks(np.arange(0, 1.41, step=0.2))\n    plt.title('Model Loss - '+ 'No Augmentation vs '+ augmentation_name + ' (' + set_type + ' set)')\n    plt.ylabel('Loss')\n    plt.xlabel('Epoch')\n    plt.legend(['No Augmentation', augmentation_name], loc='upper right')\n\n    error_baseline = [1-x for x in baseline[acc]]\n    error_agumented = [1-x for x in agumented[acc]]\n    \n    plt.subplot(1, 2, 2)\n    plt.plot(error_baseline)\n    plt.plot(error_agumented)\n    locs, labels = plt.yticks()\n    plt.yticks(np.arange(0, 0.41, step=0.05))\n    plt.title('Model Error - '+ 'No Augmentation vs '+ augmentation_name + ' (' + set_type + ' set)')\n    plt.ylabel('Top-1 Error')\n    plt.xlabel('Epoch')\n    plt.legend(['No Augmentation', augmentation_name], loc='upper right')\n\n    plt.show()\n\n    \ndef plot_loss_error_all(baseline, agumented_0, agumented_1, agumented_2, set_type='validation'):\n\n    if set_type == 'validation':\n        loss = 'val_loss'\n        acc = 'val_categorical_accuracy'\n    else:\n        loss = 'loss'\n        acc = 'categorical_accuracy'\n    \n    # Model Loss subplot\n    plt.subplot(1, 2, 1)\n    plt.plot(baseline[loss])\n    plt.plot(agumented_0[loss])\n    plt.plot(agumented_1[loss])\n    plt.plot(agumented_2[loss])\n    locs, labels = plt.yticks()\n    plt.xticks(np.arange(0, 51, step=5))\n    plt.yticks(np.arange(0, 1.41, step=0.2))\n    plt.title('Model Loss on ' + set_type.capitalize() + ' Set' +'\\n'+ 'NoAugmentation vs RandomErasing vs MixUp vs CutMix', fontsize=18)\n    plt.ylabel('Loss', fontsize=16)\n    plt.xlabel('Epoch', fontsize=16)\n    plt.legend(['No Augmentation', 'Random Erasing', 'MixUp', 'CutMix'], loc='upper right')\n    plt.grid(axis='y')\n\n    # Model Error subplot\n    error_baseline = [1-x for x in baseline[acc]]\n    error_agumented_0 = [1-x for x in agumented_0[acc]]\n    error_agumented_1 = [1-x for x in agumented_1[acc]]\n    error_agumented_2 = [1-x for x in agumented_2[acc]]\n    \n    plt.subplot(1, 2, 2)\n    plt.plot(error_baseline)\n    plt.plot(error_agumented_0)\n    plt.plot(error_agumented_1)\n    plt.plot(error_agumented_2)\n    plt.xticks(np.arange(0, 51, step=5))\n    locs, labels = plt.yticks()\n    plt.yticks(np.arange(0, 0.41, step=0.05))\n    plt.title('Model Error on ' + set_type.capitalize() + ' Set' +'\\n'+ 'NoAugmentation vs RandomErasing vs MixUp vs CutMix', fontsize=18)\n    plt.ylabel('Top-1 Error', fontsize=16)\n    plt.xlabel('Epoch', fontsize=16)\n    plt.legend(['No Augmentation', 'Random Erasing', 'MixUp', 'CutMix'], loc='upper right')\n    plt.grid(axis='y')\n\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Other Utility Functions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def accuracy(y_pred, y):\n    \"\"\"\n    Calculate the accuracy.\n    Args:\n        y_pred: list, predicted labels.\n        y: list, true labels.\n    Returns:\n        Accuracy measure (float).\n    \"\"\"\n    return sum([y_pred[i] == y[i] for i in range(len(y))]) / len(y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    #label = tf.cast(example['class'], tf.int32)\n    label = tf.one_hot(indices=tf.cast(example['class'], tf.int32), depth=len(CLASSES))\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset\n\ndef get_training_dataset(augmentation='base'):\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    \n    if augmentation == 'random_erasing':\n        dataset = dataset.map(random_erasing, num_parallel_calls=AUTO)\n    elif augmentation == 'mixup':\n        dataset = dataset.map(mixup, num_parallel_calls=AUTO)\n    elif augmentation == 'cutmix':\n        dataset = dataset.map(cutmix, num_parallel_calls=AUTO)\n        \n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_test_dataset(ordered=False, labeled=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=labeled, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nprint('Dataset: {} training images, {} validation images, {} test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Training and Validation labels\nlabels_train = []\nfor i, x in enumerate(iter(get_training_dataset().unbatch().batch(1))):\n    labels_train.append(np.argmax(x[1].numpy()))\n    if i >= NUM_TRAINING_IMAGES:\n        break\nlabels_val = [np.argmax(x[1].numpy()) for x in iter(get_validation_dataset().unbatch().batch(1))] \n#labels_test = [np.argmax(x[1].numpy()) for x in iter(get_test_dataset(labeled=True).unbatch().batch(1))]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Ordered Test labels\nlabels_test = np.concatenate([y for x, y in get_test_dataset(labeled=True, ordered=True)], axis=0)\nlabels_test = [x.argmax() for x in labels_test]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = [labels_train, labels_val, labels_test]\nweights = [weights_calc(labels_train), weights_calc(labels_val), weights_calc(labels_train)]\nweights2=np.ones(len(labels_val)) / (len(labels_val))\nweights3=np.ones(len(labels_train)) / (len(labels_train))\nplt.hist(data, weights=[weights3, weights2, weights2] , bins=104, label=['Train', 'Validation', 'Test'])\nplt.legend(loc='upper right')\nplt.title('Class distribution', fontsize=20)\nplt.xlabel('Classes', fontsize=16)\nplt.ylabel('Percentage in the set', fontsize=16)\n\nplt.gca().yaxis.set_major_formatter(PercentFormatter(1))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see in the above graph, the class distribution is quite similar among the three sets of data (Training, Validation and Test sets).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Dataset Structure","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# data dump\nprint(\"Training data shapes:\")\nfor image, label in get_training_dataset().take(2):\n    print(image.numpy().shape, label.numpy().shape)\nprint()\nprint(\"Training data label examples:\\n\", label.numpy()[:2])\nprint()\nprint(\"Validation data shapes:\")\nfor image, label in get_validation_dataset().take(2):\n    print(image.numpy().shape, label.numpy().shape)\nprint()\nprint(\"Validation data label examples:\\n\", label.numpy()[:2])\nprint()\nprint(\"Test data shapes:\")\nfor image, idnum in get_test_dataset().take(2):\n    print(image.numpy().shape, idnum.numpy().shape)\nprint()\nprint(\"Test data IDs:\\n\", idnum.numpy().astype('U')) # U=unicode string","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Augmentation","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Random Erasing / Cutout","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def random_erasing(images, labels, p=0.5, s_l=0.02, s_h=0.4, r_1=0.3, r_2=1/0.3, \n                   erasing_type='RE-R', batch=BATCH_SIZE, rand_seed=SEED, max_attempts=100):\n    \"\"\"\n    Perform Random Erasing over a batch of images, following the original \n    description in \"https://arxiv.org/abs/1708.04896\".\n    This function is meant to be used with TPUs.\n    \n    Args:\n        images: a batch of images, i.e. a tensor of size [BATCH_SIZE, WIDE, HEIGH, N_CHANNELS]. \n                Pixels should be normalized in [0, 1].\n        labels: a batch of tf tensor, image labels using one-hot encoding\n        p: float in [0, 1], probability of performing random erasing over an image.\n        s_l: float in [0, 1], minimum percentage of the random erased rectangle \n             surface with respect the image area.\n        s_h: float in [0, 1], maximum percentage of the random erased rectangle \n             surface with respect the image area.\n        r_1: float, minimum aspect ratio (height / width) of the random erased \n             rectangle area.\n        r_2: float, maximum aspect ratio (height / width) of the random erased \n             rectangle area.\n        erasing_type: how to fill the random erased rectangle areas:\n                      * \"RE-R\": Each pixel is assigned with a random value\n                      * \"RE-M\": All pixels are assigned with the mean ImageNet \n                                pixel value, i.e. [125, 122, 114] / 255\n                      * \"RE-1\": All pixels assigned with 1\n                      Otherwise \"RE-0\" is implemented (all pixels assigned with 0)\n        batch: positive integer, batch size.\n        rand_seed: integer, random seed.\n        max_attempts: maximum number of attempts in order to find a valid patch \n                      position.\n    \n    Returns:\n        A tuple (batch of images, batch of labels) modified using Random Erasing.\n    \"\"\"\n    # modified images\n    mod_img = []\n    \n    # Image dimensions\n    h = images.shape[1]  # image height\n    w = images.shape[2]  # image width\n    n_chan = images.shape[3]  # number of channels\n    n_classes = labels.shape[1]  # number of classes\n\n    for i in range(batch):\n        q = np.random.uniform()  # random probability using tensorflow\n        if q >= p:\n            mod_img.append(images[i])  # the image is kept unchanged\n        else:  # Random Erasing\n            attempts = 0\n            while True and attempts < max_attempts:\n            \n                # Area of the erased rectangle patch\n                s_e = np.random.uniform(low=s_l, high=s_h) * h * w\n                # Aspect ratio of the erased rectangle patch\n                r_e = np.random.uniform(low=r_1, high=r_2)\n            \n                h_e = int((s_e * r_e)**0.5)  # height of the Random Erased patch\n                w_e = int((s_e / r_e)**0.5)  # width of the Random Erased patch\n            \n                # (x_e, y_e) are the coordinates of the upper-left corner of the \n                # Random Erased patch.\n                x_e = int(np.random.uniform(low=0, high=h))\n                y_e = int(np.random.uniform(low=0, high=w))\n\n                if (x_e + w_e <= w) and (y_e + h_e <= h):  # assess if the patch top-left corner position is valid\n                    \n                    # Choose the type of patch\n                    if erasing_type == 'RE-R':\n                        # RE-R: Each pixel is assigned with a random value \n                        # rainging [0, 255] / 255.\n                        patch = tf.random.uniform(shape=[h_e, w_e, n_chan], \n                                                  minval=0, maxval=1, seed=rand_seed)\n                    elif erasing_type == 'RE-M':\n                        # RE-M: All pixels are assigned with the mean ImageNet \n                        # pixel value, i.e. [125, 122, 114] / 255.\n                        foo = tf.ones([h_e, w_e, 1], dtype=tf.float32)\n                        patch = tf.concat([foo * 125 / 255., foo * 122 / 255., foo * 114 / 255.], axis=2)\n                    elif erasing_type == 'RE-1':\n                        # RE-1: All pixels assigned with 255 / 255.\n                        patch = tf.ones([h_e, w_e, n_chan], dtype=tf.float32)\n                    else: \n                        # RE-0: All pixels assigned with 0\n                        patch = tf.zeros([h_e, w_e, n_chan], dtype=tf.float32)\n                    \n                    # Section of the image exactly on the top and bottom of the \n                    # Random Erased patch.\n                    top = images[i][(y_e+h_e):, x_e:(x_e+w_e), :]  \n                    bot = images[i][:y_e, x_e:(x_e+w_e), :]\n\n                    mid = tf.concat([bot, patch, top], axis=0)\n                \n                    # Part of the image on the left of the Random Erased patch\n                    left= images[i][:, :x_e, :]\n                    # Part of the image on the right of the Random Erased patch\n                    right= images[i][:, (x_e+w_e):, :]  \n                \n                    mod_img.append(tf.concat([left, mid, right], axis=1))\n                    break\n                \n                attempts += 1  # number of attempts in order to find a valid patch position\n\n            # At the time I am writing this notebook Tensorflow doesn't support \n            # else clause in while loop, so an extra \"if\" should be added to \n            # cover the case when the maximum number of attempts is excedded.\n            ## Not currently allowed:\n            ## else:  \n            ##     mod_img.append(images[i])\n            #\n            # If the maximum number of attempts is excedded the image is kept \n            # unchanged.\n            if attempts == max_attempts:\n                mod_img.append(images[i])\n\n    output_images = tf.reshape(tf.stack(mod_img), shape=(batch, h, w, n_chan))\n    output_labels = tf.reshape(labels, shape=(batch, n_classes))\n    \n    return (output_images, output_labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### MixUp","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def mixup(images, labels, alpha=0.4, batch=BATCH_SIZE, rand_seed=SEED):\n    \"\"\"\n    Perform MixUp over a batch of images, similar to the algorithm described in \n    \"https://arxiv.org/pdf/1710.09412.pdf\".\n    NOTE: in the paper some details such as how to choose the images for \n    mixing-up are not explained, so an approach similar to CutMix was choisen in \n    this implementation.\n    This function is meant to be used with TPUs.\n    \n    Args:\n        images: a batch of images, i.e. a tensor of size [BATCH_SIZE, WIDE, HEIGH, N_CHANNELS]. \n                Pixels should be normalized in [0, 1].\n        labels: a batch of tf tensor, image labels using one-hot encoding\n        alpha: non negative float, parameters of a Beta(alpha, alpha) distribution\n        batch: int, batch size.\n        rand_seed: int, random seed.\n            \n    Returns:\n        A tuple (batch of images, batch of labels) modified using MixUp.\n    \"\"\"\n\n    # Set numpy random seed\n    np.random.seed(rand_seed)\n    \n    # Modified images that would be reshaped in a tensor with the same dimensions \n    # as the initial image batch.\n    mod_img = []\n    # Modified labels\n    mod_lab = []\n    \n    # Input dimensions   \n    h = images.shape[1]  # image height\n    w = images.shape[2]  # image width\n    n_chan = images.shape[3]  # number of channels\n    n_classes = labels.shape[1]  # number of classes\n    \n    # Shuffle mini-batch\n    batch_shuffle = np.arange(batch) \n    np.random.shuffle(batch_shuffle)\n\n    for i, j in enumerate(batch_shuffle):\n        lamb = np.random.beta(alpha, alpha)  # beta distribution using numpy\n        \n        # New image\n        new_img = lamb * images[i] + (1 - lamb) * images[j]\n        mod_img.append(new_img)\n        \n        # New \"mixed\" label\n        new_lab = lamb * labels[i] + (1 - lamb) * labels[j]\n        mod_lab.append(new_lab) \n\n    output_images = tf.reshape(tf.stack(mod_img), shape=(batch, h, w, n_chan))\n    output_labels = tf.reshape(tf.stack(mod_lab), shape=(batch, n_classes))\n    \n    return (output_images, output_labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### CutMix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def cutmix(images, labels, batch=BATCH_SIZE, rand_seed=SEED):\n    \"\"\"\n    Perform CutMix over a batch of images, following the original description in \n    \"https://arxiv.org/abs/1905.04899\".\n    This function is meant to be used with TPUs.\n    \n    Args:\n        images: a batch of images, i.e. a tensor of size [BATCH_SIZE, WIDE, HEIGH, N_CHANNELS]. \n                Pixels should be normalized in [0, 1].\n        labels: a batch of tf tensor, image labels using one-hot encoding\n        batch: integer, batch size\n        rand_seed: random seed\n            \n    Returns:\n        A tuple (batch of images, batch of labels) modified using CutMix.\n    \"\"\"\n    # Set numpy random seed\n    np.random.seed(rand_seed)\n    \n    # Modified images that would be reshaped in a tensor with the same \n    # dimensions as the initial image batch.\n    mod_img = []\n    # Modified labels\n    mod_lab = []\n\n    # Input dimensions\n    h = images.shape[1]  # image height\n    w = images.shape[2]  # image width\n    n_chan = images.shape[3]  # number of channels\n    n_classes = labels.shape[1]  # number of classes\n    \n    # Shuffle mini-batch\n    batch_shuffle = np.arange(batch)\n    np.random.shuffle(batch_shuffle)\n    \n    for i, j in enumerate(batch_shuffle):\n        lamb = np.random.uniform()\n        \n        # Coordinates of a random point inside the image, \"center\" of the patch\n        r_x = np.random.randint(0, w)\n        r_y = np.random.randint(0, h)\n        \n        r_w = int((1 - lamb)**0.5 * w)  # patch width\n        r_h = int((1 - lamb)**0.5 * h)  # patch height\n        \n        # Bottom-left corner of the patch\n        x_1 = int(np.max([r_x - r_w / 2, 0]))\n        y_1 = int(np.max([r_y - r_h / 2, 0]))\n        # Top-right corner of the patch\n        x_2 = int(np.min([r_x + r_w / 2, w]))\n        y_2 = int(np.min([r_y + r_h / 2, h]))\n        \n        patch = images[j][y_1:y_2, x_1:x_2, :]\n        \n        # Sections of the image exactly on the top and bottom of the  patch\n        top = images[i][y_2:, x_1:x_2, :]\n        bot = images[i][:y_1, x_1:x_2, :]   \n\n        mid = tf.concat([bot, patch, top], axis=0)\n\n        # Sections of the image exactly on the left and right of the  patch\n        left= images[i][:, :x_1, :]\n        right= images[i][:, x_2:, :]\n        \n        mod_img.append(tf.concat([left, mid, right], axis=1))\n\n        # Real lambda coefficient applied, i.e. 1 - [(area patch) / (area image)]\n        lamb = 1 - (x_2 - x_1) * (y_2 - y_1) / (w * h)\n        \n        # New \"mixed\" label\n        new_lab = lamb * labels[i] + (1 - lamb) * labels[j]\n        \n        mod_lab.append(new_lab)\n        \n    output_images = tf.reshape(tf.stack(mod_img), shape=(batch, h, w, n_chan))\n    output_labels = tf.reshape(tf.stack(mod_lab), shape=(batch, n_classes))\n        \n    return (output_images, output_labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Augmentation Techniques Visual Comparison","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get a batch of images to display\ntraining_dataset = get_training_dataset()\ntraining_dataset = training_dataset.unbatch().batch(20)\ntrain_batch = iter(training_dataset)\nbatch = next(train_batch)\nimages = batch[0]\nlabels = batch[1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### No augmentation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display_batch_of_images((images, labels))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Random erasing\n\nIn the Random Erasing/CutOut Method the original labels are preserved. \n\nHere I have used patches filled with random noise but there are other options used in the original article: the mean ImageNet pixel values, all ones (or 255s) and all zeros. My function implementation can handle all these options using the parameter *erasing_type*.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display_batch_of_images((random_erasing(images, labels, erasing_type='RE-R', batch=20)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### MixUp\n\nThe MixUp method modifies both images and labels. In particular, non-integer labels are allowed: each sample can have a label in the original fomat (one-hot encoding) or a vector of 104 entries in which there are only two entries different from zero that sum to one.\n\nFor example, \"0.336 petunia 0.664 water lily\" means that the augmented images is generated throught the folmula $\\lambda \\cdot images_{i} + (1 - \\lambda) \\cdot images_{j}$ where $\\lambda = 0.336$ and $images_{i}$ belongs to class \"petunia\" while $images_{j}$ belongs to class \"water lily\".","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display_batch_of_images((mixup(images, labels, batch=20)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### CutMix\n\nThe CutMix method modifies both images and labels. In particular, non-integer labels are allowed: each sample can have a label in the original fomat (one-hot encoding) or a vector of 104 entries in which there are only two entries different from zero that sum to one.\n\nFor example, \"0.336 petunia 0.664 water lily\" means that the augmented images is generated from two images belonging to classes \"petunia\" and \"water lily\" respectively. The area from the \"petunia\" original image cover the 33.6% of the area of the aumented image and the area from the \"water lily\" original image cover the remaining 66.4% of the area of the aumented image.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display_batch_of_images((cutmix(images, labels, batch=20)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"To experiment the three method a simple EfficientNetB0 with Global Average Pooling applied to the output of the last convolutional layer was chosen. This choice was made because EfficientNet architecture is becoming very popular, it has a [Keras implementation](https://keras.io/api/applications/efficientnet/) and in the \"B0\" version is quite fast to train.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_model(model_name=''):\n    \"\"\"\n    Utility function to create a model based on EfficientNetB0 architecture.\n    Args:\n        model_name: string, if empty string imagenet weights are loaded and all\n                    layers are trainable. Otherwise, the weights in the \n                    model_name.h5 file are loaded (for predicting/evaluating).\n    \"\"\"\n    backbone = efn.EfficientNetB0(weights='imagenet', \n                                  include_top=False, \n                                  pooling='avg', \n                                  input_shape=[*IMAGE_SIZE, N_CHANNELS])\n    if model_name == '':\n        backbone.trainable = True  # All layers of the backbone net are trainable \n        \n    model = tf.keras.Sequential([backbone,\n                                 tf.keras.layers.Dense(len(CLASSES), \n                                                       activation='softmax', \n                                                       dtype='float32')])\n    if model_name != '':\n        model.load_weights(model_name)\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    model = get_model()\n        \nmodel.compile(optimizer='adam',\n              loss = 'categorical_crossentropy',\n              metrics=['categorical_accuracy'])\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def compile_and_train(augmentation='nothing', iterations=N_ENSEMBLE):\n    \"\"\"\n    Function for compiling and training a model based on EfficientNetB0 \n    architecture.\n    Args:\n        augmentation: string, type of agumentation to perform among \"cutmix\",\n                      \"mixup\" and \"random_erasing\". Otherwise no data \n                      augmentation is performed.\n        iterations: positive int, number of models with the same augmentations \n                    to be trained.\n    Returns:\n         A list of all the traininig histories, one for each iteration.\n         Note that at each iteration the best model in terms of \n         val_categorical_accuracy is saved in the output directory.\n    \"\"\"\n    \n    # Garbage collector\n    gc.collect()\n    \n    # Release TPU memory\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    \n    # List of all the training histories, one for each iteration \n    history_collection = []\n    \n    for i in range(iterations):\n        \n        print('Model ' + str(i) + '/' + str(iterations - 1))\n        model_name = 'effb0_' + augmentation + '_' + str(i)\n        \n        # Model definition\n        with strategy.scope():   \n            model = get_model()\n    \n        # Compile the model\n        model.compile(optimizer='adam',\n                      loss='categorical_crossentropy',\n                      metrics=['categorical_accuracy'])\n    \n        # Callbacks\n        checkpoint = tf.keras.callbacks.ModelCheckpoint(model_name + '.h5',\n                                                        save_best_only=True,\n                                                        save_weights_only=True,\n                                                        #monitor='val_categorical_accuracy',  # alternative: val_loss\n                                                        monitor='val_loss',\n                                                        mode='auto',  # i.e. \"max\" for accuracy\n                                                        save_freq='epoch',\n                                                        verbose=0)\n    \n        # Training\n        history = model.fit(get_training_dataset(augmentation), \n                            steps_per_epoch=STEPS_PER_EPOCH, \n                            epochs=EPOCHS,\n                            validation_data=get_validation_dataset(),\n                            callbacks=[checkpoint])\n        \n        history_collection.append(history)\n        \n        print()\n    \n    return history_collection","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"hist_no_augmentation = compile_and_train()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hist_random_erasing = compile_and_train(augmentation='random_erasing')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hist_mixup = compile_and_train(augmentation='mixup')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hist_cutmix = compile_and_train(augmentation='cutmix')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predictions","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The accuracy is calculated on an esemble of N_ENSEMBLE models (averaging the predictions) in order to obtain results that are more reliable and robust than a single shot.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Names list of all the available models\nmodels = [f for f in listdir('/kaggle/working/') if isfile(join('/kaggle/working/', f)) and f[-2:]=='h5']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Accuracy on the test set \naccuracies = {}\n\nfor aug in ['nothing', 'random_erasing', 'mixup', 'cutmix']:\n    \n    models_pred = []\n    for i in range(N_ENSEMBLE):\n        \n        print(aug, i)\n        \n        gc.collect()  # garbage collector\n        \n        # Release TPU memory\n        tf.tpu.experimental.initialize_tpu_system(tpu)\n        \n        \n        weights = '_'.join([MODEL_NAME, aug, str(i)]) +'.h5'\n        \n        with strategy.scope():\n            model = get_model(weights)\n\n        # Compile the model\n        model.compile(optimizer='adam',\n                      loss='categorical_crossentropy',\n                      metrics=['categorical_accuracy'])\n        # Predictions\n        preds = model.predict(get_test_dataset(labeled=True, ordered=True))\n        models_pred.append(preds)\n    \n    # Ensemble: averaging predictions\n    avg_pred = sum(models_pred) / len (models_pred)\n    avg_pred = [x.argmax() for x in avg_pred]\n    \n    # Ordered Test labels\n    labels_test = np.concatenate([y for x, y in get_test_dataset(labeled=True, ordered=True)], axis=0)\n    labels_test = [x.argmax() for x in labels_test]\n    \n    # Accuracy calculation\n    accuracies[aug] = accuracy(avg_pred, labels_test)\n    \n    print()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Analysis","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Graph 1: Accuracy","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots()\nbar_graph = ax.bar(accuracies.keys(), accuracies.values(), width=0.6, color=['grey', 'blue', 'blue', 'blue'])\nax.set_ylabel('Accuracy', fontsize=16)\nax.set_title(\"Ensemble Models' Accuracy\", fontsize=20)\nlocs, labels = plt.yticks()\nplt.yticks(np.arange(0, 1.01, step=0.1))\nplt.xticks(np.arange(4), ('NoAugmentation', 'RandomErasing', 'MixUp', 'CutMix'), fontsize=16)\nautolabel(bar_graph)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see in the above graph, all the three data augmentation techniques generates an improvement in accuracy in this example. \n\nAn ensemble of N_ENSEMBLE models (section \"Parameters\") is used to improve result robustness (only for this graph!).\n\nThe improvement using CutMix seems to be quite significant with respect the non-augmented model.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"****","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Graph 2: Model Loss - Training Set vs Validation Set","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_metric(hist_no_augmentation[0].history, hist_random_erasing[0].history, hist_mixup[0].history, hist_cutmix[0].history, metric='loss')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using MixUp and CutMix the Loss on the Validation Set results lower than the loss on the Training Set. This is apparently a counterintuitive result that can be explained since the examples in the Validation Set are not augmented, so they are easyer to classify correctly.\n\nNotice that both MixUp and CutMix validation curves are smoother than the non-augmented and the Random Erasing cases.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Graph 3: Model Accuracy - Training Set vs Validation Set","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_metric(hist_no_augmentation[0].history, hist_random_erasing[0].history, hist_mixup[0].history, hist_cutmix[0].history, metric='categorical_accuracy')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using CutMix the accuracy on the Validation Set results lower than the accuracy on the Training Set. As before, it can be explained since the examples in the Validation Set are not augmented, so they are easyer to classify correctly.\n\nIn the MixUp case, both Training and Validation accuracy curves are very close. I would expected a behaviour similar to CutMix and I do not know why there is this difference (Any idea?:))","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"****","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Graph 4: Model Loss and Error - No Augmentation vs Random Erasing vs MixUp vs CutMix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":" plot_loss_error_all(hist_no_augmentation[0].history, hist_random_erasing[0].history, hist_mixup[0].history, hist_cutmix[0].history, set_type='validation')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The introduction of augmentation techniques seem to be beneficial on the Validation Set starting from epoch 5 in this example.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"****","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Observations\n\n* As I stated in the first paragraph, all the above code is intended only for giving a glimpse about the usefulness of Random Erasing, MixUp and CutMix in the context of image recognition throught some examples.\n\n\n* From the graph \"Ensemble Models' Accuracy\" we can notice an increasing accuracy from the models trained without augmentation to the models trained using CutMix. All the experimented augmentation methods showed some improvements with respect the basic model without aumentation. In our example, Random Erasing increase accuracy of about 0.5 percentage point (p.p.), MixUp of about 1 p.p. and finally CutMix of about 2 p.p. Similar results are reported in [\\[1\\]](https://arxiv.org/abs/1708.04896) (\"Related Articles and Links\") on an ImageNet classification problem.\n\n\n* Another intresting evidence about the usefulness of these aumentation methods, at least in our specific example, is given by the \"Model Loss\" / \"Model Error\" graphs using teh Validation Set:\n    * Random Erasing's Loss and Top-1 Error is only slightly lower that the one of the non augmented model on average.\n    * MixUp's Loss is substantially lower than the non augmented one starting from 10th epoch (about 0.4 vs about 0.6).\n    * CutMix shows both the lowest Loss and Top-1 Error\n    \n    \n* All these methods can be used combined. In particular, a common combo is CutMix-MixUp.\n\n\n* There are several improvements that can be done to this notebook, for istance:\n    * Repeat the analysis using different parameters for each augmnetation methods\n    * Replicate the class activation mapping (CAM) analysis as in [\\[1\\]](https://arxiv.org/abs/1708.04896)\n    * Use different datasets to see if different results are obtained\n    * Try to implement [AugMix](https://arxiv.org/abs/1912.02781)\n    * ...","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"****","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Related Articles and Links\n\n1. [Random Erasing Data Augmentation](https://arxiv.org/abs/1708.04896) was the article used for the Random Erasing implementation in this notebook. Another interesting article that describe basically the same method (called CutOut) is: [Improved Regularization of Convolutional Neural Networks with Cutout](https://arxiv.org/abs/1708.04552)\n2. [Mixup: Beyond Empirical Risk Minimization](https://arxiv.org/abs/1710.09412).\n3. [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://arxiv.org/abs/1905.04899).\n4. [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://arxiv.org/abs/1912.02781) explains the brand new AugMix method that could be implemented and included in the notebook.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"***","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}