{"cells":[{"metadata":{"id":"2sK4N8oEpxrB"},"cell_type":"markdown","source":"# Flower Classification with TPUs\n\n**In the past 5 years or so, machine learning models have been able to outperform humans on image classification tasks. What started as distinguishing between horses and humans has now evolved to detecting melanoma in images of moles with over 96% accuracy. This is largly because of transfer learning - the practice of importing pretrained models that are already intelligent and using them for your own task**\n\n**But these models are deep and wide convolutional neural networks, with tens of millions of parameters that make training on regular hardware almost impossible. Luckily, there are tensor processing units (TPUs), hardware accelerators that are specifically designed for these deep learning tasks. The TPU available through Kaggle has 8 TPU cores which means we can funnel our data through 8 different channels, drastically decreasing computation time**\n\n**Before we begin, thank you  to the starter kernel which helped kickstart this notebook. It can be found [here](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu). If you enjoy this notebook, please leave an upvote and feel free to comment with any questions/suggestions. Let's begin:**","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"id":"ot-sd-nvpxrB","outputId":"87645e76-1c3b-4ecd-8a45-c8c61226d9db"},"cell_type":"code","source":"#the basics\nfrom matplotlib import pyplot as plt\nimport math, os, re\nimport numpy as np, pandas as pd\n\n#deep learning basics\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\n\n#get current TensorFlow version fo\nprint(\"Currently using Tensorflow version \" + tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{"id":"OoU1g5QwpxrF"},"cell_type":"markdown","source":"# I. Configuration\n\n**To take advantage of TPUs, we have to do some extra work. For the uninitiated, [this](http://www.tensorflow.org/guide/tpu) is an excellent place to start. We start by checking to see if TensorFlow is using a TPU or not - if it isn't, we set the 'strategy' to its default, which works on CPU and a single GPU, though we will definitely need to use the TPU for the current parameter setups of this notebook (if you use smaller image sizes, you might get away with running on CPU/GPU)**","execution_count":null},{"metadata":{"trusted":true,"id":"EkGzSMH4pxrF","outputId":"aab2f9b1-db18-46ed-ea72-6934dff8c53a"},"cell_type":"code","source":"DEVICE = 'TPU'   #or GPU\n\nif DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except _:\n            print(\"failed to initialize TPU\")\n    else:\n        DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n    print(\"Using default strategy for CPU and single GPU\")\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n    \n\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\n#REPLICAS = 8\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{"id":"pYU48G_gpxrI"},"cell_type":"markdown","source":"**TPUs read data directly from Google Cloud Storage (GCS), so we actually need to copy our dataset to a GCS 'bucket' that is near or 'co-located' with the TPU. The below chunk of code accomplishes this using the handy kaggle_datasets:** ","execution_count":null},{"metadata":{"trusted":true,"id":"6TBv62tNpxrI"},"cell_type":"code","source":"#get GCS path for flower classification data set\nfrom kaggle_datasets import KaggleDatasets\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('tpu-getting-started')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"PZ9xKjB6pxrK","outputId":"10ca9ede-016a-4e6c-b115-d1488e0a887f"},"cell_type":"code","source":"#can use this path in Google Colabs if you want to host notebook else where\nprint(GCS_DS_PATH)","execution_count":null,"outputs":[]},{"metadata":{"id":"9E2oUBaspxrM"},"cell_type":"markdown","source":"**To optimize the TPUs bandwith, we cut our dataset into files and then send these files to the different TPU cores. The common format for these files is TFRecords which essentially just takes the pixels of the image and some other information (e.g. a label) and stuffs it into a file. A good number of TFRecord files is 16: so we take our dataset and split it into 16 different TFRecord files and send them to the TPUs**\n\n**Note: we need to modify some parameters accordingly because of this, most notably, we need to multiply whatever batch size we intend to use for our model(s) by 16**","execution_count":null},{"metadata":{"trusted":true,"id":"xpYJbPsFpxrM"},"cell_type":"code","source":"#for reproducibility\nSEED = 34     #my favorite number\n\n#define image size we will use\n#IMAGE_SIZE = [192, 192]               #if you aren't using TPU\n#IMAGE_SIZE = [331, 331]               #middle ground\nIMAGE_SIZE = [512, 512]               #if you are using TPU\n\n#how many training samples we want going to TPUs \nBATCH_SIZE = 16 * strategy.num_replicas_in_sync \n\n#define aug batch size\nAUG_BATCH = BATCH_SIZE\n\n#how many folds we will use to train our model on\nFOLDS = 3\n\n#how many TTA steps to apply\nTTA = 5\n\n#list other options we have for image sizes\nGCS_PATH_SELECT = {\n    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'\n}\n\n#choose 512 image size for best performance\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]","execution_count":null,"outputs":[]},{"metadata":{"id":"xTlwnCYTpxrR"},"cell_type":"markdown","source":"**Now we need to create some functions that allow us to extract information from these `TFRecords`. We will create functions that read the image and label from the `TFRecords`. For more about this, see [here](http://www.tensorflow.org/tutorials/load_data/tfrecord)**\n\n**We can also perform some easy augmentations to be used during training and also for test time augmentation. For a quick reference on using `tf.image` to perform image augmentation, see [this](http://www.tensorflow.org/tutorials/images/data_augmentation)**\n\n**Note: to achieve peak performance, we can use a pipeline that 'prefetches' data for the next step before the current step has finished using `tf.data`. You can learn more [here](http://www.tensorflow.org/guide/data_performance)**","execution_count":null},{"metadata":{"trusted":true,"id":"7PEHqJZ0pxrR"},"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    \n    #returns a dataset of (image, label) pairs\n    return image, label\n\ndef read_unlabeled_tfrecord(example, return_image_name):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # [] means single entry\n    }\n    \n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    #returns a dataset of image(s)\n    return image, idnum if return_image_name else 0\n\n#some simply image augmentation we can perform with tf.image\ndef data_augment(image, label):\n    \n    #random augmentations\n    image = tf.image.random_flip_left_right(image)\n    #image = tf.image.random_flip_up_down(image)\n    #image = tf.image.random_hue(image, 0.01)\n    #image = tf.image.random_saturation(image, 0.7, 1.3)\n    #image = tf.image.random_contrast(image, 0.8, 1.2)\n    #image = tf.image.random_brightness(image, 0.1)\n    \n    #fixed augmentations\n    #image = tf.image.adjust_saturation(image, max_delta = .2)\n    #image = tf.image.central_crop(image, central_fraction = 0.5)\n    return image, label  \n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"In751JnwpxrU","outputId":"b29b637e-4f9a-4064-c862-75ab4619b2f8"},"cell_type":"code","source":"#define pre fetching strategy\nAUTO = tf.data.experimental.AUTOTUNE\n\n#use tf.io.gfile.glob to find our training and test files from GCS bucket\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec') + tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec')\n    \n#show item counts\nNUM_TRAINING_IMAGES = int( count_data_items(TRAINING_FILENAMES) * (FOLDS-1.)/FOLDS )\nNUM_VALIDATION_IMAGES = int( count_data_items(TRAINING_FILENAMES) * (1./FOLDS) )\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\n\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"n6RnBbJLpxrX"},"cell_type":"code","source":"def get_train_ds(files, tta_aug = False, cutmix_aug = False, shuffle = True, \n                 repeat = True, labeled=True, return_image_names = True):\n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()\n    \n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(1024*8)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n\n    if labeled: \n        ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls=AUTO) \n    \n    if tta_aug:\n        ds = ds.map(data_augment, num_parallel_calls = AUTO)\n        ds = ds.map(transform, num_parallel_calls=AUTO)\n    \n    if cutmix_aug: \n        #need to batch to use CutMix/mixup\n        ds = ds.batch(AUG_BATCH)\n        ds = ds.map(mixup_and_cutmix, num_parallel_calls=AUTO) # note we put AFTER batching\n        \n        #now unbatch and shuffle before re-batching\n        ds = ds.unbatch()\n        #ds = ds.shuffle(2048)\n    \n    #prefetch next batch while training\n    ds = ds.batch(BATCH_SIZE)\n    ds = ds.prefetch(AUTO)\n    \n    return ds\n\n\ndef get_val_ds(files, shuffle = True, labeled=True, return_image_names=False):\n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()  \n    \n    if shuffle: \n        ds = ds.shuffle(1024*8)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n\n    if labeled: \n        ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls=AUTO)\n        \n    \n    #prefetch next batch while training\n    ds = ds.batch(BATCH_SIZE)\n    \n    #we must one hot encode if we use CutMix or mixup\n    ds = ds.map(onehot, num_parallel_calls=AUTO)\n    ds = ds.prefetch(AUTO)\n    \n    return ds\n","execution_count":null,"outputs":[]},{"metadata":{"id":"PTV3XNBmpxrd"},"cell_type":"markdown","source":"# II. Visualization\n\n**Now that we have dealt with all the configuring required to use TPUs, we can extract our images from the TPU and finally get a look at our data:**","execution_count":null},{"metadata":{"trusted":true,"id":"AK_VvHWkpxrd"},"cell_type":"code","source":"#define flower classes for labeling purposes\nclasses = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 102","execution_count":null,"outputs":[]},{"metadata":{"id":"Zav_SHtapxrf"},"cell_type":"markdown","source":"**Define some helper functions to plot our flower images:**","execution_count":null},{"metadata":{"trusted":true,"id":"foPjMRkhpxrf"},"cell_type":"code","source":"#numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    #binary strings are image IDs\n    if numpy_labels.dtype == object:\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    #If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return classes[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(classes[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                classes[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch, predictions=None):\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    #auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    #size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    #display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else classes[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #get optimal spacing\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"LSQ4KUiDpxrh"},"cell_type":"code","source":"#first look at training dataset\ntraining_dataset = get_train_ds(TRAINING_FILENAMES, cutmix_aug = False, tta_aug = False, labeled = True,\n                                        shuffle = True, repeat = True)\ntraining_dataset = training_dataset.unbatch().batch(20)\ntrain_batch = iter(training_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"L57gjrovpxrj"},"cell_type":"code","source":"#first look at test dataset\ntest_dataset = get_train_ds(TEST_FILENAMES, labeled = False, shuffle = True, repeat = False)\ntest_dataset = test_dataset.unbatch().batch(20)\ntest_batch = iter(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"jriUtSxvpxrl","outputId":"7f577c30-55d3-431a-db9b-687952a0a4af"},"cell_type":"code","source":"#view batch of flowers from train\ndisplay_batch_of_images(next(train_batch))\n#you can run this cell again and it will load a new batch","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"0sL0aKTipxrm","outputId":"2a6c6b67-73ce-443d-8cbb-6014c4604299"},"cell_type":"code","source":"#view batch of flowers from test\ndisplay_batch_of_images(next(test_batch))\n#you can run this cell again and it will load a new batch","execution_count":null,"outputs":[]},{"metadata":{"id":"vE65QNftpxro"},"cell_type":"markdown","source":"# III. Augmentation\n\n**Note: the following augmentation implementations are taken from (4X Kaggle Grandmaster) [Chris Deotte](https://www.kaggle.com/cdeotte)'s notebook, which can be found [here](https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu) and [here](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96). We will define CutMix and mixup functions to apply during training and a rotation/shift/shearing function to apply during testing for TTA**\n\n### Rotation, Shift, Zoom, Shear","execution_count":null},{"metadata":{"trusted":true,"id":"dId3DSXjpxrp"},"cell_type":"code","source":"def get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n    # returns 3x3 transformmatrix which transforms indicies\n    \n    rotation = math.pi * rotation / 180.\n    shear = math.pi * shear / 180.\n    \n    c1 = tf.math.cos(rotation)\n    s1 = tf.math.sin(rotation)\n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    rotation_matrix = tf.reshape( tf.concat([c1,s1,zero, -s1,c1,zero, zero,zero,one],axis=0),[3,3] )\n        \n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)\n    shear_matrix = tf.reshape( tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3] )    \n    \n    zoom_matrix = tf.reshape( tf.concat([one/height_zoom,zero,zero, zero,one/width_zoom,zero, zero,zero,one],axis=0),[3,3] )\n    \n    shift_matrix = tf.reshape( tf.concat([one,zero,height_shift, zero,one,width_shift, zero,zero,one],axis=0),[3,3] )\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), K.dot(zoom_matrix, shift_matrix))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"G58W7nFopxrq"},"cell_type":"code","source":"def transform(image,label):\n    DIM = IMAGE_SIZE[0]\n    XDIM = DIM%2 #fix for size 331\n    \n    rot = 15. * tf.random.normal([1],dtype='float32')\n    shr = 5. * tf.random.normal([1],dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1],dtype='float32')/10.\n    w_zoom = 1.0 + tf.random.normal([1],dtype='float32')/10.\n    h_shift = 16. * tf.random.normal([1],dtype='float32') \n    w_shift = 16. * tf.random.normal([1],dtype='float32') \n\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    idx2 = K.dot(m,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n         \n    idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image,tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM,DIM,3]),label","execution_count":null,"outputs":[]},{"metadata":{"id":"CUzorS6rpxrs"},"cell_type":"markdown","source":"### mixup\n\n**Now, the augmentation we did above is great, but we are still adding noise to the images which is also leading to information loss. Luckily, we can do better with mixup. Essentially, all mixup does is random converts images to convex combinations of pairs of images and their labels, as seen in the illustration below:**\n\n![mixup](http://miro.medium.com/max/362/0*yLCQYAtNAh28LQks.png)\nImage from [here](http://medium.com/swlh/how-to-do-mixup-training-from-image-files-in-keras-fe1e1c1e6da6)\n\n**We can see that we retain information about both images and their labels while introducing regularization into our model. For more on MixUp, read [this](https://arxiv.org/abs/1710.09412)**","execution_count":null},{"metadata":{"trusted":true,"id":"5r6qpLzspxrt"},"cell_type":"code","source":"#need to one hot encode images so we can blend their labels like above\ndef onehot(image,label):\n    CLASSES = len(classes)\n    return image,tf.one_hot(label,CLASSES)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"8-YKE75-pxru"},"cell_type":"code","source":"def mixup(image, label, PROBABILITY = 1.0):\n    DIM = IMAGE_SIZE[0]\n    CLASSES = 104\n    \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.float32)\n\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n        a = tf.random.uniform([],0,1)*P # this is beta dist with alpha=1.0\n\n        img1 = image[j,]\n        img2 = image[k,]\n        imgs.append((1-a)*img1 + a*img2)\n\n        if len(label.shape)==1:\n            lab1 = tf.one_hot(label[j],CLASSES)\n            lab2 = tf.one_hot(label[k],CLASSES)\n        else:\n            lab1 = label[j,]\n            lab2 = label[k,]\n        labs.append((1-a)*lab1 + a*lab2)\n\n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image2,label2","execution_count":null,"outputs":[]},{"metadata":{"id":"1vCXROMMpxrw"},"cell_type":"markdown","source":"### CutMix\n\n**CutMix is essentially the same as mixup except the images are not blended together, rather a random sized block of one image is superimposed on another image. You can read more about it [here](http://arxiv.org/pdf/1905.04899.pdf)**","execution_count":null},{"metadata":{"trusted":true,"id":"t5saSRoepxrw"},"cell_type":"code","source":"#first look at test dataset\ntest_dataset = get_train_ds(TEST_FILENAMES, cutmix_aug = False, tta_aug = False, labeled = False,\n                                        shuffle = True, repeat = False)\ntest_dataset = test_dataset.unbatch().batch(20)\ntest_batch = iter(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"35QPZNHypxry"},"cell_type":"code","source":"def cutmix(image, label, PROBABILITY = 1.0):\n    # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n    # output - a batch of images with cutmix applied\n    DIM = IMAGE_SIZE[0]\n    CLASSES = 104\n    \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.int32)\n\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        b = tf.random.uniform([],0,1) # this is beta dist with alpha=1.0\n        WIDTH = tf.cast( DIM * tf.math.sqrt(1-b),tf.int32) * P\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n\n        one = image[j,ya:yb,0:xa,:]\n        two = image[k,ya:yb,xa:xb,:]\n        three = image[j,ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        img = tf.concat([image[j,0:ya,:,:],middle,image[j,yb:DIM,:,:]],axis=0)\n        imgs.append(img)\n\n        a = tf.cast(WIDTH*WIDTH/DIM/DIM,tf.float32)\n        if len(label.shape)==1:\n            lab1 = tf.one_hot(label[j],CLASSES)\n            lab2 = tf.one_hot(label[k],CLASSES)\n        else:\n            lab1 = label[j,]\n            lab2 = label[k,]\n        labs.append((1-a)*lab1 + a*lab2)\n        \n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image2,label2","execution_count":null,"outputs":[]},{"metadata":{"id":"yxZVp7lVpxrz"},"cell_type":"markdown","source":"### mixup and CutMix\n\n**It is hard to choose which is better, mixup or CutMix. Luckily, we don't actually have to choose because we can just apply both:**\n\n**Note: CutMix will occur `SWITCH * CUTMIX_PROB` of the time and mixup will occur `(1 - SWITCH) * MIXUP_PROB` of the time. We will need to experiment a bit to see which convex combination delivers the best performance**\n\n**The current setup gives us mixup 33% of the time, CutMix 33% of the time, and no augmentation 33% of the time**","execution_count":null},{"metadata":{"trusted":true,"id":"4ohstL7Dpxrz"},"cell_type":"code","source":"#create function to apply both cutmix and mixup\ndef mixup_and_cutmix(image,label):\n    CLASSES = len(classes)\n    DIM = IMAGE_SIZE[0]\n    \n    #define how often we want to do activate cutmix or mixup\n    SWITCH = 1/2\n    \n    #define how often we want cutmix or mixup to activate when switch is active\n    CUTMIX_PROB = 2/3\n    MIXUP_PROB = 2/3\n    \n    #apply cutmix and mixup\n    image2, label2 = cutmix(image, label, CUTMIX_PROB)\n    image3, label3 = mixup(image, label, MIXUP_PROB)\n    imgs = []; labs = []\n    \n    for j in range(BATCH_SIZE):\n        P = tf.cast( tf.random.uniform([],0,1)<=SWITCH, tf.float32)\n        imgs.append(P*image2[j,]+(1-P)*image3[j,])\n        labs.append(P*label2[j,]+(1-P)*label3[j,])\n        \n    #must explicitly reshape so TPU complier knows output shape\n    image4 = tf.reshape(tf.stack(imgs),(BATCH_SIZE,DIM,DIM,3))\n    label4 = tf.reshape(tf.stack(labs),(BATCH_SIZE,CLASSES))\n    return image4,label4","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"9udUZGZNpxr1","outputId":"14388650-a1f7-4006-ff76-1b1be8ae1950"},"cell_type":"code","source":"row = 6; col = 4;\nrow = min(row,AUG_BATCH//col)\nall_elements = get_train_ds(TRAINING_FILENAMES, cutmix_aug = True, tta_aug = False, labeled = True,\n                                        shuffle = True, repeat = True)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,int(15*row/col)))\n    for j in range(row*col):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"id":"AiKoUtptpxr3"},"cell_type":"markdown","source":"# IV. Model Training\n\n**Now, it will take far too much time for us to train a model ourselves to learn the optimal weights for classifying our flower photos, so we will instead import a model that has already been pre-trained on ImageNet: a large labeled dataset of real-world images**\n\n**We will be importing several popular pre-trained models. For a list of all the pre-trained models that can be imported with Keras, see [here](https://keras.io/api/applications/)**","execution_count":null},{"metadata":{"trusted":true,"id":"lC9KziLUpxr3","outputId":"15629c08-7081-4eec-b3d2-e27f25378db8"},"cell_type":"code","source":"#define epoch parameters\nEPOCHS = 20                \nSTEPS_PER_EPOCH = count_data_items(TRAINING_FILENAMES) // BATCH_SIZE\n\n#define learning rate parameters\nLR_START = 0.00001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_DECAY = .8\n\n#define ramp up and decay\ndef lr_schedule(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lr_schedule, verbose = True)\n\n#visualize learning rate schedule\nrng = [i for i in range(EPOCHS)]\ny = [lr_schedule(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{"id":"P6HzJ-4Apxr5"},"cell_type":"markdown","source":"**If you decide to use an EfficientNet model for your final model, you need to install something as it is not yet supported by `keras.applications`. There is another weight option for EffNets to consider that outperforms Imagenet weights called 'Noisy Student' that you can read about [here](https://arxiv.org/abs/1911.04252). For more on EffNets in general, read [this](https://arxiv.org/pdf/1905.11946.pdf)**","execution_count":null},{"metadata":{"trusted":true,"id":"KBCwXOhTpxr5"},"cell_type":"code","source":"#import DenseNet201, Xception, InceptionV3, and InceptionResNetV2\nfrom tensorflow.keras.applications import DenseNet201, Xception, InceptionV3, InceptionResNetV2\n\n#requirements to use EfficientNet(s)\n!pip install -q efficientnet\nimport efficientnet.tfkeras as efn\n\n#helper function to create our model\ndef get_DenseNet201():\n    CLASSES = len(classes)\n    with strategy.scope():\n        dnet = DenseNet201(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'imagenet',\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        dnet.trainable = True\n        model = tf.keras.Sequential([\n            dnet,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax',dtype = 'float32')\n        ])\n    model.compile(\n        optimizer='adam',\n        loss = 'categorical_crossentropy', \n        metrics=['categorical_accuracy']\n    )\n    return model\n\n#create Xception model\ndef get_Xception():\n    CLASSES = len(classes)\n    with strategy.scope():\n        xception = Xception(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'imagenet',\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        xception.trainable = True\n        model = tf.keras.Sequential([\n            xception,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax',dtype = 'float32')\n        ])\n    model.compile(\n        optimizer='adam',\n        loss = 'categorical_crossentropy',\n        metrics=['categorical_accuracy']\n    )\n    return model\n\n#create Inception model\ndef get_InceptionV3():\n    CLASSES = len(classes)\n    with strategy.scope():\n        inception = InceptionV3(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'imagenet',\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        inception.trainable = True\n        model = tf.keras.Sequential([\n            inception,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax',dtype = 'float32')\n        ])\n    model.compile(\n        optimizer='adam',\n        loss = 'categorical_crossentropy',\n        metrics=['categorical_accuracy']\n    )\n    return model\n\n#create EfficientNetB4 model\ndef get_EfficientNetB4():\n    CLASSES = len(classes)\n    with strategy.scope():\n        efficient = efn.EfficientNetB4(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'noisy-student', #or imagenet\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        efficient.trainable = True\n        model = tf.keras.Sequential([\n            efficient,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax',dtype = 'float32')\n        ])\n    model.compile(\n        optimizer='adam',\n        loss = 'categorical_crossentropy',\n        metrics=['categorical_accuracy']\n    )\n    return model\n\n#create EfficientNetB5 model\ndef get_EfficientNetB5():\n    CLASSES = len(classes)\n    with strategy.scope():\n        efficient = efn.EfficientNetB5(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'noisy-student', #or imagenet\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        efficient.trainable = True\n        model = tf.keras.Sequential([\n            efficient,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax',dtype = 'float32')\n        ])\n    model.compile(\n        optimizer='adam',\n        loss = 'categorical_crossentropy',\n        metrics=['categorical_accuracy']\n    )\n    return model\n\n\n#create EfficientNetB6 model\ndef get_EfficientNetB6():\n    CLASSES = len(classes)\n    with strategy.scope():\n        efficient = efn.EfficientNetB6(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'imagenet', #or imagenet\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        efficient.trainable = True\n        model = tf.keras.Sequential([\n            efficient,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax',dtype = 'float32')\n        ])\n    model.compile(\n        optimizer='adam',\n        loss = 'categorical_crossentropy',\n        metrics=['categorical_accuracy']\n    )\n    return model\n\n#create EfficientNetB7 model\ndef get_EfficientNetB7():\n    CLASSES = len(classes)\n    with strategy.scope():\n        efficient = efn.EfficientNetB7(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'noisy-student', #or imagenet\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        efficient.trainable = True\n        model = tf.keras.Sequential([\n            efficient,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax',dtype = 'float32')\n        ])\n    model.compile(\n        optimizer='adam',\n        loss = 'categorical_crossentropy',\n        metrics=['categorical_accuracy']\n    )\n    return model\n\n#create InceptionResNet model\ndef get_InceptionResNetV2():\n    CLASSES = len(classes)\n    with strategy.scope():\n        inception_res = InceptionResNetV2(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'imagenet',\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        inception_res.trainable = True\n        model = tf.keras.Sequential([\n            inception_res,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax',dtype = 'float32')\n        ])\n    model.compile(\n        optimizer='adam',\n        loss = 'categorical_crossentropy',\n        metrics=['categorical_accuracy']\n    )\n    return model","execution_count":null,"outputs":[]},{"metadata":{"id":"dfUQSlRqpxr7"},"cell_type":"markdown","source":"**To make our model even more robust, we can also split our data into different folds during training. For example, if `FOLDS = 5`, we would split our data into 5 different folds, train on the first 4 folds, and then validate on the remaining fold. We then repeat this process and cycle through the folds until we have 5 different models that have been trained and validated on 5 different datasets. We then generate predictions with each one of these trained models and take the average of all 5 predictions as our final predictions:**\n\n**(Recall that the `FOLDS` parameter was defined much earlier in the notebook as it was needed to determine the size of the validation set)**","execution_count":null},{"metadata":{"trusted":true,"id":"NFw2A8cnpxr8","outputId":"11c21215-a5fa-4384-bdeb-dc800dbf7828"},"cell_type":"code","source":"from sklearn.model_selection import KFold\n#train and cross validate in folds\n\n\nhistories = []\nmodels = []\n#early_stopping = tf.keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 3)\nkfold = KFold(FOLDS, shuffle = True, random_state = SEED)\n\n#break into different fodls\nfor f, (train_index, val_index) in enumerate(kfold.split(TRAINING_FILENAMES)):\n#to clear the TPU memory each fold\n    #tf.tpu.experimental.initialize_tpu_system(tpu)\n    print(); print('-'*25)\n    print(f\"Training fold {f + 1} with EfficientNetB6\")\n    print('-'*25)\n    print('Getting training data...')\n    print('')\n    #get files for training and validation   \n    train_ds = get_train_ds(list(pd.DataFrame({'TRAINING_FILENAMES': TRAINING_FILENAMES}).loc[train_index]['TRAINING_FILENAMES']),\n                                    cutmix_aug = True, tta_aug = False, labeled = True, shuffle = True, repeat = True)\n    \n    print('Geting validation data...')\n    print('')\n    val_ds = get_val_ds(list(pd.DataFrame({'TRAINING_FILENAMES': TRAINING_FILENAMES}).loc[val_index]['TRAINING_FILENAMES']),\n                            labeled = True, shuffle = True, return_image_names = False)\n              \n    #train and cross validate\n    model = get_EfficientNetB6()\n    history = model.fit(train_ds, \n                        steps_per_epoch = STEPS_PER_EPOCH,\n                        epochs = EPOCHS,\n                        callbacks = [lr_callback], #,early_stopping]\n                        validation_data = val_ds,\n                        verbose = 2)\n    models.append(model)\n    histories.append(history)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"id":"XnOB1oilpxr9"},"cell_type":"code","source":"#define function to visualize learning curves\ndef plot_learning_curves(histories): \n    fig, ax = plt.subplots(1, 2, figsize = (20, 10))\n    \n    #plot accuracies\n    for i in range(0, 3):\n        ax[0].plot(histories[i].history['categorical_accuracy'], color = 'C0')\n        ax[0].plot(histories[i].history['val_categorical_accuracy'], color = 'C1')\n\n    #plot losses\n    for i in range(0, 3):\n        ax[1].plot(histories[i].history['loss'], color = 'C0')\n        ax[1].plot(histories[i].history['val_loss'], color = 'C1')\n\n    #fix legend\n    ax[0].legend(['train', 'validation'], loc = 'upper left')\n    ax[1].legend(['train', 'validation'], loc = 'upper right')\n    \n    #set master titles\n    fig.suptitle(\"Model Performance\", fontsize=14)\n    \n    #label axis\n    for i in range(0,2):\n        ax[0].set_ylabel('Accuracy')\n        ax[0].set_xlabel('Epoch')\n        ax[1].set_ylabel('Loss')\n        ax[1].set_xlabel('Epoch')\n\n    return plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"gX8GRENrpxr-","outputId":"8df5bff5-1b50-41aa-c857-5e3c5283f0cb"},"cell_type":"code","source":"#look at our learning curves to check bias/variance trade off\nplot_learning_curves(histories)","execution_count":null,"outputs":[]},{"metadata":{"id":"rMQnbXR4pxsA"},"cell_type":"markdown","source":"# V. Test Time Augmentation\n\n**Now we can perform augmentation on the test set as well. We will augment each image `TTA` many times and predict on each. Then we average the results of those predictions and use that as our final prediction. We cannot use CutMix/MixUp because they use label blending techniques (of which we have none in the test set) so we use the transformations with in the `data_augment` function and the rotation/shear/shift augmentation we defined earlier in the `transform` function:**","execution_count":null},{"metadata":{"trusted":true,"id":"dt5RYk37pxsC"},"cell_type":"code","source":"def get_test_dataset(filenames, shuffle = False, repeat = True, labeled = False, tta_aug = True, return_image_names = True):\n\n    ds = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO)\n    ds = ds.cache()\n    \n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(1024*8)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n\n    if labeled: \n        ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls=AUTO)\n\n    if tta_aug:\n        ds = ds.map(data_augment, num_parallel_calls = AUTO)\n        ds = ds.map(transform, num_parallel_calls=AUTO)\n    \n    ds = ds.batch(BATCH_SIZE)\n    ds = ds.prefetch(AUTO)\n\n    return ds","execution_count":null,"outputs":[]},{"metadata":{"id":"6j7c5gG5SZ-u","trusted":true},"cell_type":"code","source":"def average_tta_preds(tta_preds):\n    average_preds = np.zeros((ct_test, 104))\n    for fold in range(FOLDS):\n        average_preds += tta_preds[(fold)*ct_test:(fold + 1)*ct_test] / FOLDS\n    return average_preds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"_naKkr8fpxsD","outputId":"5dc8458f-797e-4479-a38e-78fdf6ccc71c"},"cell_type":"code","source":"#since we are splitting the dataset and iterating separately on images and ids, order matters.\ntest_ds = get_test_dataset(TEST_FILENAMES, tta_aug = True, labeled = False,\n                           repeat = True, return_image_names = True)\ntest_images_ds = test_ds.map(lambda image, idnum: image)\n    \n#set up TTA\nct_test = count_data_items(TEST_FILENAMES)\nSTEPS = TTA * ct_test/BATCH_SIZE\n\n#predict  \nprint('Getting TTA predictions...')\npred0 = models[0].predict(test_images_ds,steps = STEPS,verbose = 2)[:TTA*ct_test,] \npred1 = models[1].predict(test_images_ds,steps = STEPS,verbose = 2)[:TTA*ct_test,] \npred2 = models[2].predict(test_images_ds,steps = STEPS,verbose = 2)[:TTA*ct_test,] \nprint('')\n\n#get averages of each augmentation\nprint('Averaging TTA predictions...')\naverage_pred0 = average_tta_preds(pred0)\naverage_pred1 = average_tta_preds(pred1)\naverage_pred2 = average_tta_preds(pred1)\nprint('')\n\n#merge together and average\nprint('Merging predictions from models...')\nfinal_preds = np.zeros((ct_test, 104))\nfinal_preds += average_pred0\nfinal_preds += average_pred1\nfinal_preds += average_pred2\n\n#and take argmax to get in label form\nfinal_preds = np.argmax(final_preds, axis = 1)\n\n#lastly, get test ids for submission\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\nprint('Done')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"0UveZzmApxsG"},"cell_type":"code","source":"#create submission dataframe\nsubmission = pd.DataFrame()\nsubmission['id'] = test_ids\nsubmission['label'] = final_preds\nprint(submission.shape)\nsubmission.head(10)","execution_count":null,"outputs":[]},{"metadata":{"id":"3fgZZYSxaJyI","trusted":true},"cell_type":"code","source":"#submit to disk\nsubmission.to_csv('submission.csv', index = False)\nprint('Submission saved')","execution_count":null,"outputs":[]},{"metadata":{"id":"AJGAVMVbpxsH"},"cell_type":"markdown","source":"# V.Past Results\n\n**New version**  \nTTA = 5  \n1)EfficientNetB6 \n2)EfficientNetB6 \n3)EfficientNetB6 \nEpochs = 20  \nLB = 0.95455\n\n**Previous versions:**  \n1)EfficientNetB5  \n2)DenseNet201  \n3)EfficientNetB5  \nEpochs = 25  \nLB = 0.96380\n\n1)InceptionResNetV2  \n2)DenseNet201  \n3)EfficientNetB5  \nEpochs = 25  \nLB = 0.96290\n\n1)EfficientNetB5  \n2)EfficientNetB5  \n3)EfficientNetB5  \nEpochs = 25  \nLB = 0.96623\n\nAdded test time augmentation; TTA = 5  \n1)EfficientNetB5  \n2)EfficientNetB5  \n3)EfficientNetB5  \nEpochs = 25  \nLB = 0.95455","execution_count":null},{"metadata":{"trusted":true,"id":"cWj5G2vXpxsI"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}