{"cells":[{"metadata":{"id":"2sK4N8oEpxrB"},"cell_type":"markdown","source":"# Flower Classification with TPUs\n\n**In the past 5 years or so, machine learning models have been able to outperform humans on image classification tasks. What started as distinguishing between horses and humans has now evolved to detecting melanoma in images of moles with over 96% accuracy. This is largly because of transfer learning - the practice of importing pretrained models that are already intelligent and using them for your own task**\n\n**But these models are deep and wide convolutional neural networks, with tens (sometimes hundreds) of millions of parameters that make training on regular hardware almost impossible. Luckily, there are tensor processing units (TPUs), hardware accelerators that are specifically designed for these deep learning tasks. The TPU available through Kaggle has 8 TPU cores which means we can funnel our data through 8 different channels, drastically decreasing computation time**\n\n**Before we begin, thank you  to the starter kernel which helped kickstart this notebook. It can be found [here](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu). If you enjoy this notebook, please leave an upvote and feel free to comment with any questions/suggestions. Let's begin:**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"id":"ot-sd-nvpxrB","outputId":"4cb746df-6d08-4e75-bb41-039877c2c2bb"},"cell_type":"code","source":"#the basics\nfrom matplotlib import pyplot as plt\nimport math, os, re, gc\nimport numpy as np, pandas as pd\n\n#deep learning basics\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\n\n#get current TensorFlow version fo\nprint(\"Currently using Tensorflow version \" + tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{"id":"OoU1g5QwpxrF"},"cell_type":"markdown","source":"# I. Configuration\n\n**To take advantage of TPUs, we have to do some extra work. For the uninitiated, [this](http://www.tensorflow.org/guide/tpu) is an excellent place to start. We start by checking to see if TensorFlow is using a TPU or not - if it isn't, we set the 'strategy' to its default, which works on CPU and a single GPU, though we will definitely need to use the TPU for the current parameter setups of this notebook (if you use smaller image sizes, you might get away with running on CPU/GPU)**"},{"metadata":{"trusted":true,"id":"EkGzSMH4pxrF","outputId":"dd30b419-a0b7-41d5-8e6c-2e30824b1511"},"cell_type":"code","source":"DEVICE = 'TPU'   #or GPU\n\nif DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except _:\n            print(\"failed to initialize TPU\")\n    else:\n        DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n    print(\"Using default strategy for CPU and single GPU\")\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n    \n\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{"id":"pYU48G_gpxrI"},"cell_type":"markdown","source":"**TPUs read data directly from Google Cloud Storage (GCS), so we actually need to copy our dataset to a GCS 'bucket' that is near or 'co-located' with the TPU. The below chunk of code accomplishes this using the handy kaggle_datasets:** "},{"metadata":{"trusted":true,"id":"PZ9xKjB6pxrK"},"cell_type":"code","source":"#get GCS path for flower classification data set\nfrom kaggle_datasets import KaggleDatasets\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('tpu-getting-started')","execution_count":null,"outputs":[]},{"metadata":{"id":"9E2oUBaspxrM"},"cell_type":"markdown","source":"**To optimize the TPUs bandwith, we cut our dataset into files and then send these files to the different TPU cores. The common format for these files is TFRecords which essentially just takes the pixels of the image and some other information (e.g. a label) and stuffs it into a file. A good number of TFRecord files is 16: so we take our dataset and split it into 16 different TFRecord files and send them to the TPUs**\n\n**Note: we need to modify some parameters accordingly because of this, most notably, we need to multiply whatever batch size we intend to use for our model(s) by 16**"},{"metadata":{"trusted":true,"id":"xpYJbPsFpxrM"},"cell_type":"code","source":"#for reproducibility\nSEED = 34 \n\n#define image size we will use\n#IMAGE_SIZE = [224, 224]               \nIMAGE_SIZE = [512, 512]               \n\n#how many training samples we want going to TPUs \nBATCH_SIZE = 32 * REPLICAS \n\n#define aug batch size\nAUG_BATCH = BATCH_SIZE\n\n#how many folds we will use to train our model on\nFOLDS = 4\n\n#choose training display settings\nVERBOSE = 2\n\n#if debugging\nFIRST_FOLD_ONLY = False\n\n#list other options we have for image sizes\nGCS_PATH_SELECT = {\n    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'\n}\n\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]","execution_count":null,"outputs":[]},{"metadata":{"id":"xTlwnCYTpxrR"},"cell_type":"markdown","source":"**Now we need to create some functions that allow us to extract information from these `TFRecords`. We will create functions that read the image and label from the `TFRecords`. For more about this, see [here](http://www.tensorflow.org/tutorials/load_data/tfrecord)**\n\n**We can also perform some easy augmentations to be used during training and also for test time augmentation. For a quick reference on using `tf.image` to perform image augmentation, see [this](http://www.tensorflow.org/tutorials/images/data_augmentation)**\n\n**Note: to achieve peak performance, we can use a pipeline that 'prefetches' data for the next step before the current step has finished using `tf.data`. You can learn more [here](http://www.tensorflow.org/guide/data_performance)**"},{"metadata":{"trusted":true,"id":"gdAVUIB9BvDF"},"cell_type":"code","source":"#decodes and converts image to float in [0,1]\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    #TPU needs to be explicitly told image size\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"7PEHqJZ0pxrR"},"cell_type":"code","source":"def read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    \n    #returns a dataset of (image, label) pairs\n    return image, label\n\ndef read_unlabeled_tfrecord(example, return_image_name):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # [] means single entry\n    }\n    \n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    #returns a dataset of image(s)\n    return image, idnum if return_image_name else 0\n\n\n#apply some stock augmentations using tf.image\ndef data_augment(img, label, flip_only = True):\n\n    if flip_only:\n        img = tf.image.random_flip_left_right(img)\n    \n    else:\n        img = tf.image.random_flip_left_right(img)\n        #some other easy transformations we can apply\n        img = tf.image.random_hue(img, 0.01)\n        img = tf.image.random_saturation(img, 0.7, 1.3)\n        img = tf.image.random_contrast(img, 0.8, 1.2)\n        img = tf.image.random_brightness(img, 0.1)\n        \n    return img, label\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#define pre fetching strategy\nAUTO = tf.data.experimental.AUTOTUNE\n\n#use tf.io.gfile.glob to find our training and test files from GCS bucket\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec') + tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec')\n    \n#set up some parameters now\nNUM_TRAINING_IMAGES = int( count_data_items(TRAINING_FILENAMES) * (FOLDS-1.)/FOLDS )\nNUM_VALIDATION_IMAGES = int( count_data_items(TRAINING_FILENAMES) * (1./FOLDS) )\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES / BATCH_SIZE \n\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{"id":"E25X8VGgBvDK"},"cell_type":"markdown","source":"**Note: to use CutMix/MixUp we have to have a sample batch from which to blend our images/labels, so we need to apply it on the `get_dataset` function and not the above `prepare_image` function:**"},{"metadata":{"trusted":true,"id":"n6RnBbJLpxrX"},"cell_type":"code","source":"def get_dataset(files, course_drop = False, grid_mask = False, mat_aug = False, cutmixup = False, one_hot = False,\n                all_aug = False, shuffle = False, repeat = False, labeled = True, return_image_names = True,\n                batch_size = BATCH_SIZE, dim = IMAGE_SIZE[0]):\n   \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n\n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(2048)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls=AUTO)  \n        \n    #apply course drop\n    if course_drop:\n        ds = ds.map(data_augment, num_parallel_calls = AUTO)\n        ds = ds.map(lambda img, label: (dropout(img), label), num_parallel_calls = AUTO)\n    \n    #apply grid mask\n    if grid_mask:\n        ds = ds.map(data_augment, num_parallel_calls = AUTO)\n        ds = ds.map(apply_grid_mask, num_parallel_calls = AUTO)\n    \n    #apply rot/shear/zoom augmentation\n    if mat_aug:\n        ds = ds.map(data_augment, num_parallel_calls = AUTO)\n        ds = ds.map(transform, num_parallel_calls = AUTO)\n     \n    #apply all the above aug\n    if all_aug:\n        ds = ds.map(data_augment, num_parallel_calls = AUTO)\n        ds = ds.map(apply_all_aug, num_parallel_calls = AUTO)\n        \n    \n    #apply CutMix/MixUp combination\n    if cutmixup:\n        #ds = ds.map(data_augment, num_parallel_calls = AUTO)\n        \n        #need to batch to use CutMix/mixup\n        ds = ds.batch(AUG_BATCH)\n        ds = ds.map(cut_and_mix, num_parallel_calls = AUTO)\n        \n        #now unbatch and shuffle before re-batching\n        ds = ds.unbatch()\n        ds = ds.shuffle(2048)\n\n    if one_hot:\n        ds = ds.map(onehot, num_parallel_calls = AUTO)\n\n    ds = ds.batch(batch_size)\n    ds = ds.prefetch(AUTO)\n    \n    return ds","execution_count":null,"outputs":[]},{"metadata":{"id":"PTV3XNBmpxrd"},"cell_type":"markdown","source":"# II. Visualization\n\n**Now that we have dealt with all the configuring required to use TPUs, we can extract our images from the TPU and finally get a look at our data:**"},{"metadata":{"trusted":true,"id":"AK_VvHWkpxrd"},"cell_type":"code","source":"#define flower classes for labeling purposes\nclasses = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 102","execution_count":null,"outputs":[]},{"metadata":{"id":"Zav_SHtapxrf"},"cell_type":"markdown","source":"**Define some helper functions to plot our flower images:**"},{"metadata":{"trusted":true,"id":"foPjMRkhpxrf"},"cell_type":"code","source":"#numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    #binary strings are image IDs\n    if numpy_labels.dtype == object:\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    #If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return classes[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(classes[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                classes[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch, predictions=None):\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    #auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    #size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    #display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else classes[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #get optimal spacing\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"LSQ4KUiDpxrh"},"cell_type":"code","source":"#first look at training dataset\ntraining_dataset = get_dataset(TRAINING_FILENAMES, labeled = True, course_drop = False, all_aug = False,\n                               grid_mask = False, mat_aug = False, cutmixup = False, shuffle = True, repeat = True)\ntraining_dataset = training_dataset.unbatch().batch(20)\ntrain_batch = iter(training_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"L57gjrovpxrj"},"cell_type":"code","source":"#first look at test dataset\ntest_dataset = get_dataset(TEST_FILENAMES, labeled = False, course_drop = False, grid_mask = False,\n                           mat_aug = False, all_aug = False, shuffle = True, repeat = False)\ntest_dataset = test_dataset.unbatch().batch(20)\ntest_batch = iter(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"jriUtSxvpxrl","outputId":"038ef476-4764-497b-e7ec-7d779839c6c9"},"cell_type":"code","source":"#view batch of flowers from train\ndisplay_batch_of_images(next(train_batch))\n#you can run this cell again and it will load a new batch","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"0sL0aKTipxrm","outputId":"8f7ecba7-dfcd-4382-9dbb-9f3bdab77556"},"cell_type":"code","source":"#view batch of flowers from test\ndisplay_batch_of_images(next(test_batch))\n#you can run this cell again and it will load a new batch","execution_count":null,"outputs":[]},{"metadata":{"id":"vE65QNftpxro"},"cell_type":"markdown","source":"# III. Augmentation\n\n**Note: the following augmentation implementation is taken from (4X Kaggle Grandmaster) [Chris Deotte](https://www.kaggle.com/cdeotte)'s notebook, which can be found [here](https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu)**\n\n## Rotation, Shift, Zoom, Shear"},{"metadata":{"trusted":true,"id":"dId3DSXjpxrp"},"cell_type":"code","source":"def get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n    # returns 3x3 transformmatrix which transforms indicies\n        \n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    shear = math.pi * shear / 180.\n    \n    # ROTATION MATRIX\n    c1 = tf.math.cos(rotation)\n    s1 = tf.math.sin(rotation)\n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    rotation_matrix = tf.reshape( tf.concat([c1,s1,zero, -s1,c1,zero, zero,zero,one],axis=0),[3,3] )\n        \n    # SHEAR MATRIX\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)\n    shear_matrix = tf.reshape( tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3] )    \n    \n    # ZOOM MATRIX\n    zoom_matrix = tf.reshape( tf.concat([one/height_zoom,zero,zero, zero,one/width_zoom,zero, zero,zero,one],axis=0),[3,3] )\n    \n    # SHIFT MATRIX\n    shift_matrix = tf.reshape( tf.concat([one,zero,height_shift, zero,one,width_shift, zero,zero,one],axis=0),[3,3] )\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), K.dot(zoom_matrix, shift_matrix))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"G58W7nFopxrq"},"cell_type":"code","source":"def transform(image, label, DIM = IMAGE_SIZE[0]):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated, sheared, zoomed, and shifted\n\n    XDIM = DIM % 2\n    \n    rot = 15. * tf.random.normal([1],dtype='float32')\n    shr = 5. * tf.random.normal([1],dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1],dtype='float32')/10.\n    w_zoom = 1.0 + tf.random.normal([1],dtype='float32')/10.\n    h_shift = 16. * tf.random.normal([1],dtype='float32') \n    w_shift = 16. * tf.random.normal([1],dtype='float32') \n  \n    # GET TRANSFORMATION MATRIX\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    # LIST DESTINATION PIXEL INDICES\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(m,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES           \n    idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image,tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM,DIM,3]),label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"RJyIDNWcBvDc","outputId":"a4cdaf0a-4197-4f18-c6f1-ec92605bb4c5"},"cell_type":"code","source":"#view augmentation\nrow = 4; col = 4;\nrow = min(row,BATCH_SIZE//col)\nall_elements = get_dataset(TRAINING_FILENAMES, labeled = True, mat_aug = True,\n                           course_drop = False, cutmixup = False, shuffle = True, repeat = True)\n\nall_elements = all_elements.unbatch().batch(20)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,15))\n    for j in range(16):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"id":"eJAzFNJEBvDe"},"cell_type":"markdown","source":"## Coarse Dropout\n\n**We can use coarse dropout augmentation for online augmentation. Note that we have an option here: do we apply `transform` for augmentation or `dropout` and for what type of augmentation? You can easily experiment with either (or a combination of both) by changing the parameters of the `get_dataset` function earlier defined:**"},{"metadata":{"trusted":true,"id":"Psgq2EVwBvDe"},"cell_type":"code","source":"def dropout(image, DIM = IMAGE_SIZE[0], PROBABILITY = 1, CT = 8, SZ = 0.2):\n    \n    prob = tf.cast( tf.random.uniform([],0,1)<PROBABILITY, tf.int32)\n    if (prob==0)|(CT==0)|(SZ==0): return image\n    \n    for k in range(CT):\n\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        WIDTH = tf.cast( SZ*DIM,tf.int32) * prob\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n\n        one = image[ya:yb,0:xa,:]\n        two = tf.zeros([yb-ya,xb-xa,3]) \n        three = image[ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        image = tf.concat([image[0:ya,:,:],middle,image[yb:DIM,:,:]],axis=0)\n            \n    image = tf.reshape(image,[DIM,DIM,3])\n    \n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"FNQk1QpcBvDh","outputId":"077f46af-6492-4698-f8ba-57c71e4ca4a8"},"cell_type":"code","source":"#view what course dropout looks like\nrow = 4; col = 4;\nrow = min(row,BATCH_SIZE//col)\nall_elements = get_dataset(TRAINING_FILENAMES, labeled = True, mat_aug = False, cutmixup = False,\n                           grid_mask = False, course_drop = True, shuffle = True, repeat = True)\n\nall_elements = all_elements.unbatch().batch(20)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,15))\n    for j in range(16):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"id":"oKLsDyR1BvDi"},"cell_type":"markdown","source":"## GridMask\n\n**Now we can explore GridMask, which is essentially a less randomized Course Dropout The below code is taken from [this notebook](https://www.kaggle.com/xiejialun/gridmask-data-augmentation-with-tensorflow)**"},{"metadata":{"trusted":true,"id":"bg9-fid4BvDj"},"cell_type":"code","source":"AugParams = {\n    'd1' : 100,\n    'd2': 160,\n    'rotate' : 45,\n    'ratio' : 0.4\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"XGaqILypBvDk"},"cell_type":"code","source":"def transform2(image, inv_mat, image_shape):\n\n    h, w, c = image_shape\n    cx, cy = w//2, h//2\n\n    new_xs = tf.repeat( tf.range(-cx, cx, 1), h)\n    new_ys = tf.tile( tf.range(-cy, cy, 1), [w])\n    new_zs = tf.ones([h*w], dtype=tf.int32)\n\n    old_coords = tf.matmul(inv_mat, tf.cast(tf.stack([new_xs, new_ys, new_zs]), tf.float32))\n    old_coords_x, old_coords_y = tf.round(old_coords[0, :] + w//2), tf.round(old_coords[1, :] + h//2)\n\n    clip_mask_x = tf.logical_or(old_coords_x<0, old_coords_x>w-1)\n    clip_mask_y = tf.logical_or(old_coords_y<0, old_coords_y>h-1)\n    clip_mask = tf.logical_or(clip_mask_x, clip_mask_y)\n\n    old_coords_x = tf.boolean_mask(old_coords_x, tf.logical_not(clip_mask))\n    old_coords_y = tf.boolean_mask(old_coords_y, tf.logical_not(clip_mask))\n    new_coords_x = tf.boolean_mask(new_xs+cx, tf.logical_not(clip_mask))\n    new_coords_y = tf.boolean_mask(new_ys+cy, tf.logical_not(clip_mask))\n\n    old_coords = tf.cast(tf.stack([old_coords_y, old_coords_x]), tf.int32)\n    new_coords = tf.cast(tf.stack([new_coords_y, new_coords_x]), tf.int64)\n    rotated_image_values = tf.gather_nd(image, tf.transpose(old_coords))\n    rotated_image_channel = list()\n    for i in range(c):\n        vals = rotated_image_values[:,i]\n        sparse_channel = tf.SparseTensor(tf.transpose(new_coords), vals, [h, w])\n        rotated_image_channel.append(tf.sparse.to_dense(sparse_channel, default_value=0, validate_indices=False))\n\n    return tf.transpose(tf.stack(rotated_image_channel), [1,2,0])\n\ndef random_rotate(image, angle, image_shape):\n\n    def get_rotation_mat_inv(angle):\n        #transform to radian\n        angle = math.pi * angle / 180\n\n        cos_val = tf.math.cos(angle)\n        sin_val = tf.math.sin(angle)\n        one = tf.constant([1], tf.float32)\n        zero = tf.constant([0], tf.float32)\n\n        rot_mat_inv = tf.concat([cos_val, sin_val, zero,\n                                     -sin_val, cos_val, zero,\n                                     zero, zero, one], axis=0)\n        rot_mat_inv = tf.reshape(rot_mat_inv, [3,3])\n\n        return rot_mat_inv\n    angle = float(angle) * tf.random.normal([1],dtype='float32')\n    rot_mat_inv = get_rotation_mat_inv(angle)\n    return transform2(image, rot_mat_inv, image_shape)\n\n\ndef GridMask(image_height, image_width, d1, d2, rotate_angle=1, ratio=0.5):\n\n    h, w = image_height, image_width\n    hh = int(np.ceil(np.sqrt(h*h+w*w)))\n    hh = hh+1 if hh%2==1 else hh\n    d = tf.random.uniform(shape=[], minval=d1, maxval=d2, dtype=tf.int32)\n    l = tf.cast(tf.cast(d,tf.float32)*ratio+0.5, tf.int32)\n\n    st_h = tf.random.uniform(shape=[], minval=0, maxval=d, dtype=tf.int32)\n    st_w = tf.random.uniform(shape=[], minval=0, maxval=d, dtype=tf.int32)\n\n    y_ranges = tf.range(-1 * d + st_h, -1 * d + st_h + l)\n    x_ranges = tf.range(-1 * d + st_w, -1 * d + st_w + l)\n\n    for i in range(0, hh//d+1):\n        s1 = i * d + st_h\n        s2 = i * d + st_w\n        y_ranges = tf.concat([y_ranges, tf.range(s1,s1+l)], axis=0)\n        x_ranges = tf.concat([x_ranges, tf.range(s2,s2+l)], axis=0)\n\n    x_clip_mask = tf.logical_or(x_ranges <0 , x_ranges > hh-1)\n    y_clip_mask = tf.logical_or(y_ranges <0 , y_ranges > hh-1)\n    clip_mask = tf.logical_or(x_clip_mask, y_clip_mask)\n\n    x_ranges = tf.boolean_mask(x_ranges, tf.logical_not(clip_mask))\n    y_ranges = tf.boolean_mask(y_ranges, tf.logical_not(clip_mask))\n\n    hh_ranges = tf.tile(tf.range(0,hh), [tf.cast(tf.reduce_sum(tf.ones_like(x_ranges)), tf.int32)])\n    x_ranges = tf.repeat(x_ranges, hh)\n    y_ranges = tf.repeat(y_ranges, hh)\n\n    y_hh_indices = tf.transpose(tf.stack([y_ranges, hh_ranges]))\n    x_hh_indices = tf.transpose(tf.stack([hh_ranges, x_ranges]))\n\n    y_mask_sparse = tf.SparseTensor(tf.cast(y_hh_indices, tf.int64),  tf.zeros_like(y_ranges), [hh, hh])\n    y_mask = tf.sparse.to_dense(y_mask_sparse, 1, False)\n\n    x_mask_sparse = tf.SparseTensor(tf.cast(x_hh_indices, tf.int64), tf.zeros_like(x_ranges), [hh, hh])\n    x_mask = tf.sparse.to_dense(x_mask_sparse, 1, False)\n\n    mask = tf.expand_dims( tf.clip_by_value(x_mask + y_mask, 0, 1), axis=-1)\n\n    mask = random_rotate(mask, rotate_angle, [hh, hh, 1])\n    mask = tf.image.crop_to_bounding_box(mask, (hh-h)//2, (hh-w)//2, image_height, image_width)\n\n    return mask\n\ndef apply_grid_mask(image, label):\n    mask = GridMask(IMAGE_SIZE[0],\n                    IMAGE_SIZE[1],\n                    AugParams['d1'],\n                    AugParams['d2'],\n                    AugParams['rotate'],\n                    AugParams['ratio'])\n    \n    if IMAGE_SIZE[-1] == 3:\n        mask = tf.concat([mask, mask, mask], axis=-1)\n\n    return tf.cast(image * tf.cast(mask, tf.float32), tf.float32), label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"ZcjUaKzMBvDm","outputId":"74088035-fd38-4915-fc8b-fc9c70ebd86e"},"cell_type":"code","source":"#view what grid mask looks like\nrow = 4; col = 4;\nrow = min(row,BATCH_SIZE//col)\nall_elements = get_dataset(TRAINING_FILENAMES, labeled = True, grid_mask = True, mat_aug = False,\n                           cutmixup = False, course_drop = False, shuffle = True, repeat = True)\n\nall_elements = all_elements.unbatch().batch(20)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,15))\n    for j in range(16):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"id":"F7XqVflZBvDp"},"cell_type":"markdown","source":"## All The Above\n\n**And now we will create a function to apply all the above augmentations with certain probabilities. The current set up delivers a 25/25/25/25 split between rotation/shear/shift/zoom, course dropout, grid mask, and no augmentation. The `no_grid` parameter shifts this to a 50/25/25 split between rotation/shear/shift/zoom, course dropout, and no augmentation**"},{"metadata":{"trusted":true,"id":"DI69Cka3BvDp"},"cell_type":"code","source":"def apply_all_aug(img, label, no_grid = False):\n    \n    if tf.random.uniform([],0,1) > .5:\n        if tf.random.uniform([],0,1) > .5:\n            img, label = transform(img, label)\n            \n        #apply droupout  \n        else:\n            img = dropout(img, PROBABILITY = 1)\n            \n    else:\n        if not no_grid:\n        #apply grid mask\n            if tf.random.uniform([],0,1) > .5:\n                img, label = apply_grid_mask(img, label)\n                \n            else:\n                #do nothing\n                img, label = img, label\n                \n        else:\n        #apply transform again \n            if tf.random.uniform([],0,1) > .5:\n                img, label = transform(img, label)\n                \n            else:\n                #do nothing\n                img, label = img, label\n            \n    return img, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"v6anVdKqBvDq","outputId":"b03b1934-baa7-49bd-b803-34e12bd2f958"},"cell_type":"code","source":"#view what augmentation combination looks like\nrow = 4; col = 4;\nrow = min(row,BATCH_SIZE//col)\nall_elements = get_dataset(TRAINING_FILENAMES, labeled = True, grid_mask = False, all_aug = True, \n                           course_drop = False, cutmixup = False, shuffle = True, repeat = True)\n\nall_elements = all_elements.unbatch().batch(20)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,15))\n    for j in range(16):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"id":"CUzorS6rpxrs"},"cell_type":"markdown","source":"## MixUp\n\n**Now, the augmentations we did above are great, but we are still adding noise to the images which is also leading to information loss. Luckily, we can do better with mixup. Essentially, all mixup does is randomly converts images to convex combinations of pairs of images and their labels, as seen in the illustration below:**\n\n![mixup](http://miro.medium.com/max/362/0*yLCQYAtNAh28LQks.png)\nImage from [here](http://medium.com/swlh/how-to-do-mixup-training-from-image-files-in-keras-fe1e1c1e6da6)\n\n**We can see that we retain information about both images and their labels while introducing regularization into our model. For more on MixUp, read [this](https://arxiv.org/abs/1710.09412)**\n\n**The following MixUp and CutMix codes are taken from Chris Deotte (again) in his notebook [here](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96)**"},{"metadata":{"trusted":true,"id":"5r6qpLzspxrt"},"cell_type":"code","source":"#need to one hot encode images so we can blend their labels like above\ndef onehot(image,label):\n    CLASSES = len(classes)\n    return image,tf.one_hot(label,CLASSES)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"8-YKE75-pxru"},"cell_type":"code","source":"def mixup(image, label, PROBABILITY = 1.0, DIM = IMAGE_SIZE[0]):\n    CLASSES = 104\n    \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.float32)\n\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n        a = tf.random.uniform([],0,1)*P\n\n        img1 = image[j,]\n        img2 = image[k,]\n        imgs.append((1-a)*img1 + a*img2)\n\n        if len(label.shape)==1:\n            lab1 = tf.one_hot(label[j],CLASSES)\n            lab2 = tf.one_hot(label[k],CLASSES)\n        else:\n            lab1 = label[j,]\n            lab2 = label[k,]\n        labs.append((1-a)*lab1 + a*lab2)\n\n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image2,label2","execution_count":null,"outputs":[]},{"metadata":{"id":"1vCXROMMpxrw"},"cell_type":"markdown","source":"## CutMix\n\n**CutMix is essentially the same as mixup except the images are not blended together, rather a random sized block of one image is superimposed on another image. You can read more about it [here](http://arxiv.org/pdf/1905.04899.pdf)**"},{"metadata":{"trusted":true,"id":"35QPZNHypxry"},"cell_type":"code","source":"def cutmix(image, label, PROBABILITY = 1.0, DIM = IMAGE_SIZE[0]):\n    # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n    # output - a batch of images with cutmix applied\n    CLASSES = 104\n    \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.int32)\n\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        b = tf.random.uniform([],0,1)\n        WIDTH = tf.cast( DIM * tf.math.sqrt(1-b),tf.int32) * P\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n\n        one = image[j,ya:yb,0:xa,:]\n        two = image[k,ya:yb,xa:xb,:]\n        three = image[j,ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        img = tf.concat([image[j,0:ya,:,:],middle,image[j,yb:DIM,:,:]],axis=0)\n        imgs.append(img)\n\n        a = tf.cast(WIDTH*WIDTH/DIM/DIM,tf.float32)\n        if len(label.shape)==1:\n            lab1 = tf.one_hot(label[j],CLASSES)\n            lab2 = tf.one_hot(label[k],CLASSES)\n        else:\n            lab1 = label[j,]\n            lab2 = label[k,]\n        labs.append((1-a)*lab1 + a*lab2)\n        \n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image2,label2","execution_count":null,"outputs":[]},{"metadata":{"id":"yxZVp7lVpxrz"},"cell_type":"markdown","source":"### MixUp + CutMix\n\n**It is hard to choose which is better, mixup or CutMix. Luckily, we don't actually have to choose because we can just apply both. We can apply CutMix `SWITCH * CUTMIX_PROB` of the time and mixup `(1 - SWITCH) * MIXUP_PROB` of the time. We will need to experiment a bit to see which convex combination delivers the best performance, but this is a good starting point: mixup 33% of the time, CutMix 33% of the time, and no augmentation 33% of the time**"},{"metadata":{"trusted":true,"id":"4ohstL7Dpxrz"},"cell_type":"code","source":"#create function to apply both cutmix and mixup\ndef cut_and_mix(image, label, DIM = IMAGE_SIZE[0]):\n    CLASSES = len(classes)\n    \n    #define how often we want to do activate cutmix or mixup\n    SWITCH = 1/2\n    \n    #define how often we want cutmix or mixup to activate when switch is active\n    CUTMIX_PROB = 2/3\n    MIXUP_PROB = 2/3\n    \n    #apply cutmix and mixup\n    image2, label2 = cutmix(image, label, CUTMIX_PROB)\n    image3, label3 = mixup(image, label, MIXUP_PROB)\n    imgs = []; labs = []\n    \n    for j in range(BATCH_SIZE):\n        P = tf.cast( tf.random.uniform([],0,1)<=SWITCH, tf.float32)\n        imgs.append(P*image2[j,]+(1-P)*image3[j,])\n        labs.append(P*label2[j,]+(1-P)*label3[j,])\n        \n    #must explicitly reshape so TPU complier knows output shape\n    image4 = tf.reshape(tf.stack(imgs),(BATCH_SIZE,DIM,DIM,3))\n    label4 = tf.reshape(tf.stack(labs),(BATCH_SIZE,CLASSES))\n    return image4,label4","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"9udUZGZNpxr1","outputId":"ecd3cd4f-24b3-4696-a8c1-d2f4c2dbc089"},"cell_type":"code","source":"#view what CutMix/MixUp combination looks like\nrow = 4; col = 4;\nrow = min(row,BATCH_SIZE//col)\nall_elements = get_dataset(TRAINING_FILENAMES, labeled = True, grid_mask = False, all_aug = False,  \n                           cutmixup = True, course_drop = False, shuffle = True, repeat = True)\n\nall_elements = all_elements.unbatch().batch(20)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,15))\n    for j in range(16):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"id":"AiKoUtptpxr3"},"cell_type":"markdown","source":"# IV. Model Training\n\n**Now, it will take far too much time for us to train a model ourselves to learn the optimal weights for classifying our flower photos, so we will instead import a model that has already been pre-trained on ImageNet: a large labeled dataset of real-world images**\n\n**We will be importing several popular pre-trained models. For a list of all the pre-trained models that can be imported with Keras, see [here](https://keras.io/api/applications/)**"},{"metadata":{"trusted":true,"id":"lC9KziLUpxr3","outputId":"f1f1c35c-3437-43e7-bca9-3a5f4627ae28"},"cell_type":"code","source":"#define epoch parameters\nEPOCHS = 30                \n\n#define learning rate parameters\nLR_START = 0.00001\nLR_MAX = 0.00005 * 8\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_DECAY = .8\n\n#define ramp up and decay\ndef lr_schedule(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lr_schedule, verbose = True)\n\n#visualize learning rate schedule\nrng = [i for i in range(EPOCHS)]\ny = [lr_schedule(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{"id":"P6HzJ-4Apxr5"},"cell_type":"markdown","source":"**If you decide to use an EfficientNet model for your final model, you need to install something as it is not yet supported by `keras.applications`. There is another weight option for EffNets to consider that outperforms Imagenet weights called 'Noisy Student' that you can read about [here](https://arxiv.org/abs/1911.04252). For more on EffNets in general, read [this](https://arxiv.org/pdf/1905.11946.pdf)**"},{"metadata":{"trusted":true,"id":"Jv0NmYU2BvD3"},"cell_type":"code","source":"#import DenseNet201, Xception, InceptionV3, and InceptionResNetV2\nfrom tensorflow.keras.applications import DenseNet201, Xception, InceptionV3, InceptionResNetV2\n\n#requirements to use EfficientNet(s)\n!pip install -q efficientnet\nimport efficientnet.tfkeras as efn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"KBCwXOhTpxr5"},"cell_type":"code","source":"#helper function to create our model\ndef get_DenseNet201():\n    CLASSES = len(classes)\n    with strategy.scope():\n        dnet = DenseNet201(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'imagenet',\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        dnet.trainable = True\n        model = tf.keras.Sequential([\n            dnet,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax',dtype = 'float32')\n        ])\n    model.compile(\n        optimizer='adam',\n        loss = 'categorical_crossentropy', \n        metrics=['categorical_accuracy']\n    )\n    return model\n\n\n#create Xception model\ndef get_Xception():\n    CLASSES = len(classes)\n    with strategy.scope():\n        xception = Xception(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'imagenet',\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        xception.trainable = True\n        model = tf.keras.Sequential([\n            xception,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax',dtype = 'float32')\n        ])\n    model.compile(\n        optimizer='adam',\n        loss = 'categorical_crossentropy',\n        metrics=['categorical_accuracy']\n    )\n    return model\n\n\n#create Inception model\ndef get_InceptionV3():\n    CLASSES = len(classes)\n    with strategy.scope():\n        inception = InceptionV3(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'imagenet',\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        inception.trainable = True\n        model = tf.keras.Sequential([\n            inception,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax',dtype = 'float32')\n        ])\n    model.compile(\n        optimizer='adam',\n        loss = 'categorical_crossentropy',\n        metrics=['categorical_accuracy']\n    )\n    return model\n\n\n#create InceptionResNet model\ndef get_InceptionResNetV2():\n    CLASSES = len(classes)\n    with strategy.scope():\n        inception_res = InceptionResNetV2(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'imagenet',\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        inception_res.trainable = True\n        model = tf.keras.Sequential([\n            inception_res,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax',dtype = 'float32')\n        ])\n    model.compile(\n        optimizer='adam',\n        loss = 'categorical_crossentropy',\n        metrics=['categorical_accuracy']\n    )\n    return model\n\n\n#create EfficientNetB4 model\ndef get_EfficientNetB4(one_hot = False):\n    CLASSES = len(classes)\n    with strategy.scope():\n        efficient = efn.EfficientNetB4(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'noisy-student', #or imagenet\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        efficient.trainable = True\n        model = tf.keras.Sequential([\n            efficient,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax',dtype = 'float32')\n        ])\n\n    if one_hot: \n        model.compile(optimizer='adam', loss = 'categorical_crossentropy', metrics=['categorical_accuracy'])\n\n    else: \n        model.compile(optimizer='adam', loss = 'sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\n        \n    return model\n\n\n#create EfficientNetB5 model\ndef get_EfficientNetB5(one_hot = False):\n    CLASSES = len(classes)\n    with strategy.scope():\n        efficient = efn.EfficientNetB5(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'noisy-student', #or imagenet\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        efficient.trainable = True\n        model = tf.keras.Sequential([\n            efficient,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax', dtype = 'float32')\n        ])\n\n    if one_hot: \n        model.compile(optimizer='adam', loss = 'categorical_crossentropy', metrics=['categorical_accuracy'])\n\n    else: \n        model.compile(optimizer='adam', loss = 'sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\n\n    return model\n\n\n#create EfficientNetB6 model\ndef get_EfficientNetB6():\n    CLASSES = len(classes)\n    with strategy.scope():\n        efficient = efn.EfficientNetB6(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'imagenet', #or imagenet\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        efficient.trainable = True\n        model = tf.keras.Sequential([\n            efficient,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax',dtype = 'float32')\n        ])\n    model.compile(\n        optimizer='adam',\n        loss = 'categorical_crossentropy',\n        metrics=['categorical_accuracy']\n    )\n    return model\n\n\n#create EfficientNetB7 model\ndef get_EfficientNetB7():\n    CLASSES = len(classes)\n    with strategy.scope():\n        efficient = efn.EfficientNetB7(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'noisy-student', #or imagenet\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        efficient.trainable = True\n        model = tf.keras.Sequential([\n            efficient,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(CLASSES, activation = 'softmax', dtype = 'float32')\n        ])\n    model.compile(\n        optimizer='adam',\n        loss = 'categorical_crossentropy',\n        metrics=['categorical_accuracy']\n    )\n    return model","execution_count":null,"outputs":[]},{"metadata":{"id":"dfUQSlRqpxr7"},"cell_type":"markdown","source":"**To make our model even more robust, we can also split our data into different folds during training. For example, if `FOLDS = 5`, we would split our data into 5 different folds, train on the first 4 folds, and then validate on the remaining fold. We then repeat this process and cycle through the folds until we have 5 different models that have been trained and validated on 5 different datasets. We then generate predictions with each one of these trained models and take the average of all 5 predictions as our final predictions:**\n\n**(Recall that the `FOLDS` parameter was defined much earlier in the notebook as it was needed to determine the size of the validation set)**"},{"metadata":{"trusted":true,"id":"_6rTW-zlBvD6","outputId":"5a8d7e32-68e5-4deb-d2b4-8388cca2cc62"},"cell_type":"code","source":"from sklearn.model_selection import KFold\n#train and cross validate in folds\n\nhistories = []\npreds = np.zeros((count_data_items(TEST_FILENAMES),104))\n\n#early_stopping = tf.keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 3)\nkfold = KFold(FOLDS, shuffle = True, random_state = SEED)\n\nfor f, (train_index, val_index) in enumerate(kfold.split(TRAINING_FILENAMES)):\n    \n    #show fold info\n    if DEVICE=='TPU':\n        #hack to clear TPU memory\n        if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n    print('#'*25); print('FOLD',f+1); print('#'*25)\n    \n    #convert files to datasets\n    train_ds = get_dataset(list(pd.DataFrame({'TRAINING_FILENAMES': TRAINING_FILENAMES}).loc[train_index]['TRAINING_FILENAMES']),\n                           course_drop = False, mat_aug = False, grid_mask = False, all_aug = False, cutmixup = True,\n                           labeled = True, return_image_names = True, repeat = True, shuffle = True)  \n\n    val_ds = get_dataset(list(pd.DataFrame({'TRAINING_FILENAMES': TRAINING_FILENAMES}).loc[val_index]['TRAINING_FILENAMES']),\n                        course_drop = False, mat_aug = False, grid_mask = False, all_aug = False, cutmixup = False,\n                        one_hot = True, labeled = True, return_image_names = False, repeat = False, shuffle = True)\n\n    #and go!\n    print('Getting model...'); print(''); print('Training model...')\n    model = get_EfficientNetB5(one_hot = True)\n    history = model.fit(train_ds, validation_data = val_ds, callbacks = [lr_callback],\n                        verbose = VERBOSE, steps_per_epoch = STEPS_PER_EPOCH, epochs = EPOCHS\n    )\n    \n    histories.append(history)\n    \n    print('Predicting test without TTA')\n    ds_test = get_dataset(TEST_FILENAMES, course_drop = False, mat_aug = False, grid_mask = False, all_aug = False, cutmixup = False,\n                             labeled = False, return_image_names = False, repeat = False, shuffle = False)   \n    pred = model.predict(ds_test,verbose = VERBOSE)\n    preds += pred * 1 / FOLDS\n     \n    #so we don't hit memory limits\n    del model; z = gc.collect()\n    \n    if FIRST_FOLD_ONLY:\n        break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"id":"XnOB1oilpxr9"},"cell_type":"code","source":"#define function to visualize learning curves\ndef plot_learning_curves(histories, one_hot = False): \n    fig, ax = plt.subplots(1, 2, figsize = (20, 10))\n    \n    #plot accuracies\n    if one_hot:\n        for i in range(0, 3):\n            ax[0].plot(histories[i].history['categorical_accuracy'], color = 'C0')\n            ax[0].plot(histories[i].history['val_categorical_accuracy'], color = 'C1')\n\n    else:\n        for i in range(0, 3):\n            ax[0].plot(histories[i].history['sparse_categorical_accuracy'], color = 'C0')\n            ax[0].plot(histories[i].history['val_sparse_categorical_accuracy'], color = 'C1')\n\n    #plot losses\n    for i in range(0, 3):\n        ax[1].plot(histories[i].history['loss'], color = 'C0')\n        ax[1].plot(histories[i].history['val_loss'], color = 'C1')\n\n    #fix legend\n    ax[0].legend(['train', 'validation'], loc = 'upper left')\n    ax[1].legend(['train', 'validation'], loc = 'upper right')\n    \n    #set master titles\n    fig.suptitle(\"Model Performance\", fontsize=14)\n    \n    #label axis\n    for i in range(0,2):\n        ax[0].set_ylabel('Accuracy')\n        ax[0].set_xlabel('Epoch')\n        ax[1].set_ylabel('Loss')\n        ax[1].set_xlabel('Epoch')\n\n    return plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"gX8GRENrpxr-","outputId":"56786033-f8bc-4f8e-dc85-4d779fc2de90"},"cell_type":"code","source":"#look at our learning curves to check bias/variance trade off\nif not FIRST_FOLD_ONLY:\n    plot_learning_curves(histories, one_hot = True)","execution_count":null,"outputs":[]},{"metadata":{"id":"xc0lkkaxBvD-"},"cell_type":"markdown","source":"# V. Submission"},{"metadata":{"trusted":true,"id":"FWJkXI97BvD-","outputId":"e19418d1-97d7-4937-cc18-2408cfee9ae0"},"cell_type":"code","source":"#dummy test dataset to grab image names\ntest_ds = get_dataset(TEST_FILENAMES, labeled = False, repeat = False, return_image_names = True)\ntest_images_ds = test_ds.map(lambda image, idnum: image)\n\n#get test ids for submission\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n\n#create non TTA submission\nsubmission = pd.DataFrame()\nsubmission['id'] = test_ids\nsubmission['label'] = preds.argmax(axis = 1)\nsubmission.to_csv('submission.csv', index = False)\nsubmission.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"cWj5G2vXpxsI"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}