{"cells":[{"metadata":{"id":"2sK4N8oEpxrB","papermill":{"duration":0.034105,"end_time":"2020-11-25T21:20:10.673002","exception":false,"start_time":"2020-11-25T21:20:10.638897","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Cassava TensorFlow Starter\n\n**This notebook is a simple training pipeline in TensorFlow for the [Cassava Leaf Competition](https://www.kaggle.com/c/cassava-leaf-disease-classification) where we are given `21,367` labeled images of cassava leaves classified as 5 different groups (4 diseases and a healthy group) and asked to predict on unseen images of cassava leaves. As with most image classification problems, we can use and experiment with many different forms of augmentation and we can explore transfer learning.**\n\n**Note that I am using [Dimitre](https://www.kaggle.com/dimitreoliveira)'s TFRecords that can be found [here](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-512x512). He also has 128x128, 256x256, and 384x384 sized images that I added for experimental purposes. Please give his datasets an upvote (and his work in general, it is excellent).**"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:20:10.74954Z","iopub.status.busy":"2020-11-25T21:20:10.748761Z","iopub.status.idle":"2020-11-25T21:20:28.804906Z","shell.execute_reply":"2020-11-25T21:20:28.80417Z"},"papermill":{"duration":18.097858,"end_time":"2020-11-25T21:20:28.805042","exception":false,"start_time":"2020-11-25T21:20:10.707184","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"!pip install -q efficientnet\nfrom efficientnet.tfkeras import *","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2020-11-25T21:20:28.880459Z","iopub.status.busy":"2020-11-25T21:20:28.879368Z","iopub.status.idle":"2020-11-25T21:20:28.882779Z","shell.execute_reply":"2020-11-25T21:20:28.882046Z"},"id":"ot-sd-nvpxrB","outputId":"4cb746df-6d08-4e75-bb41-039877c2c2bb","papermill":{"duration":0.043157,"end_time":"2020-11-25T21:20:28.882906","exception":false,"start_time":"2020-11-25T21:20:28.839749","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#the basics\nfrom matplotlib import pyplot as plt\nimport math, os, re, gc\nimport numpy as np, pandas as pd\nfrom sklearn.metrics import classification_report, accuracy_score\n\n#deep learning basics\nimport tensorflow as tf\nimport tensorflow.keras.backend as K","execution_count":null,"outputs":[]},{"metadata":{"id":"OoU1g5QwpxrF","papermill":{"duration":0.033102,"end_time":"2020-11-25T21:20:28.949501","exception":false,"start_time":"2020-11-25T21:20:28.916399","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# I. Configuration\n\n**To take advantage of TPUs, we have to do some extra work. For the uninitiated, [this](http://www.tensorflow.org/guide/tpu) is an excellent place to start.**"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:20:29.038283Z","iopub.status.busy":"2020-11-25T21:20:29.029897Z","iopub.status.idle":"2020-11-25T21:20:33.048963Z","shell.execute_reply":"2020-11-25T21:20:33.048101Z"},"id":"EkGzSMH4pxrF","outputId":"dd30b419-a0b7-41d5-8e6c-2e30824b1511","papermill":{"duration":4.066283,"end_time":"2020-11-25T21:20:33.049102","exception":false,"start_time":"2020-11-25T21:20:28.982819","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"DEVICE = 'TPU'   #or GPU\n\nif DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except _:\n            print(\"failed to initialize TPU\")\n    else:\n        DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n    print(\"Using default strategy for CPU and single GPU\")\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n    \n\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{"id":"pYU48G_gpxrI","papermill":{"duration":0.034866,"end_time":"2020-11-25T21:20:33.119226","exception":false,"start_time":"2020-11-25T21:20:33.08436","status":"completed"},"tags":[]},"cell_type":"markdown","source":"**TPUs read data directly from Google Cloud Storage (GCS), so we actually need to copy our dataset to a GCS 'bucket' that is near or 'co-located' with the TPU. The below chunk of code accomplishes this using the handy `KaggleDatasets`:** "},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:20:34.008192Z","iopub.status.busy":"2020-11-25T21:20:34.006987Z","iopub.status.idle":"2020-11-25T21:20:34.010383Z","shell.execute_reply":"2020-11-25T21:20:34.011093Z"},"id":"xpYJbPsFpxrM","papermill":{"duration":0.045592,"end_time":"2020-11-25T21:20:34.011275","exception":false,"start_time":"2020-11-25T21:20:33.965683","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#for reproducibility\nSEED = 34 \n             \nIMAGE_SIZE = [512, 512]               \n\nBATCH_SIZE = 16 * REPLICAS \n\nAUG_BATCH = 2 * BATCH_SIZE\n\nFOLDS = 5\n\nVERBOSE = 1\n\nFIRST_FOLD_ONLY = False\n\n#AUG_TYPE = 'ATA'\n#AUG_TYPE = 'COURSE_DROP'\n#AUG_TYPE = 'MAT_AUG'\n#AUG_TYPE = 'GRID_MASK'\nAUG_TYPE = 'CUTMIXUP'","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:20:33.198919Z","iopub.status.busy":"2020-11-25T21:20:33.196908Z","iopub.status.idle":"2020-11-25T21:20:33.860222Z","shell.execute_reply":"2020-11-25T21:20:33.859412Z"},"id":"PZ9xKjB6pxrK","papermill":{"duration":0.706701,"end_time":"2020-11-25T21:20:33.860353","exception":false,"start_time":"2020-11-25T21:20:33.153652","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\nGCS_PATH = KaggleDatasets().get_gcs_path(f'cassava-leaf-disease-tfrecords-{IMAGE_SIZE[0]}x{IMAGE_SIZE[0]}')\nGCS_PATH_ORG = KaggleDatasets().get_gcs_path('cassava-leaf-disease-classification')\n\nprint(GCS_PATH, GCS_PATH_ORG)       #can use these in Colab notebook","execution_count":null,"outputs":[]},{"metadata":{"id":"xTlwnCYTpxrR","papermill":{"duration":0.036251,"end_time":"2020-11-25T21:20:34.083072","exception":false,"start_time":"2020-11-25T21:20:34.046821","status":"completed"},"tags":[]},"cell_type":"markdown","source":"**Now we need to create some functions that allow us to extract information from these `TFRecords`. We will create functions that read the image and label from the `TFRecords`. For more about this, see [here](http://www.tensorflow.org/tutorials/load_data/tfrecord)**\n\n**We can also perform some easy augmentations to be used during training and also for test time augmentation. For a quick reference on using `tf.image` to perform image augmentation, see [this](http://www.tensorflow.org/tutorials/images/data_augmentation)**"},{"metadata":{"_kg_hide-input":false,"execution":{"iopub.execute_input":"2020-11-25T21:20:34.184323Z","iopub.status.busy":"2020-11-25T21:20:34.179259Z","iopub.status.idle":"2020-11-25T21:20:34.195985Z","shell.execute_reply":"2020-11-25T21:20:34.195177Z"},"id":"7PEHqJZ0pxrR","papermill":{"duration":0.077843,"end_time":"2020-11-25T21:20:34.196133","exception":false,"start_time":"2020-11-25T21:20:34.11829","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#decodes and converts image to float in [0,1]\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    #TPU needs to be explicitly told image size\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"target\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['target'], tf.int32)\n    \n    #returns a dataset of (image, label) pairs\n    return image, label\n\ndef read_unlabeled_tfrecord(example, return_image_name):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # [] means single entry\n    }\n    \n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    #returns a dataset of image(s)\n    return image, idnum if return_image_name else 0\n\n\n#apply some stock augmentations using tf.image\ndef data_augment(img, label, flip_only = True):\n\n    if flip_only:\n        img = tf.image.random_flip_left_right(img)\n    \n    else:\n        img = tf.image.random_flip_left_right(img)\n        #some other easy transformations we can apply\n        img = tf.image.random_hue(img, 0.01)\n        img = tf.image.random_saturation(img, 0.7, 1.3)\n        img = tf.image.random_contrast(img, 0.8, 1.2)\n        img = tf.image.random_brightness(img, 0.1)\n        \n    return img, label\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\ndef get_dataset(files, course_drop = False, grid_mask = False, mat_aug = False, cutmixup = False, one_hot = False,\n                all_aug = False, shuffle = False, repeat = False, labeled = True, return_image_names = True,\n                batch_size = BATCH_SIZE, dim = IMAGE_SIZE[0]):\n   \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n\n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(2048)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls=AUTO)  \n        \n    #apply course drop\n    if course_drop:\n        ds = ds.map(data_augment, num_parallel_calls = AUTO)\n        ds = ds.map(lambda img, label: (dropout(img), label), num_parallel_calls = AUTO)\n    \n    #apply grid mask\n    if grid_mask:\n        ds = ds.map(data_augment, num_parallel_calls = AUTO)\n        ds = ds.map(apply_grid_mask, num_parallel_calls = AUTO)\n    \n    #apply rot/shear/zoom augmentation\n    if mat_aug:\n        ds = ds.map(data_augment, num_parallel_calls = AUTO)\n        ds = ds.map(transform, num_parallel_calls = AUTO)\n     \n    #apply all the above aug\n    if all_aug:\n        ds = ds.map(data_augment, num_parallel_calls = AUTO)\n        ds = ds.map(apply_all_aug, num_parallel_calls = AUTO)\n        \n    \n    #apply CutMix/MixUp combination\n    if cutmixup:\n        #ds = ds.map(data_augment, num_parallel_calls = AUTO)\n        \n        #need to batch to use CutMix/mixup\n        ds = ds.batch(AUG_BATCH)\n        ds = ds.map(cut_and_mix, num_parallel_calls = AUTO)\n        \n        #now unbatch and shuffle before re-batching\n        ds = ds.unbatch()\n        ds = ds.shuffle(2048)\n\n    if one_hot:\n        ds = ds.map(onehot, num_parallel_calls = AUTO)\n\n    ds = ds.batch(batch_size)\n    ds = ds.prefetch(AUTO)\n    \n    return ds\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r'-([0-9]*)\\.').search(filename).group(1)) for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:20:34.430074Z","iopub.status.busy":"2020-11-25T21:20:34.429047Z","iopub.status.idle":"2020-11-25T21:20:34.434388Z","shell.execute_reply":"2020-11-25T21:20:34.433788Z"},"papermill":{"duration":0.20319,"end_time":"2020-11-25T21:20:34.434525","exception":false,"start_time":"2020-11-25T21:20:34.231335","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"TRAINING_FILENAMES =  tf.io.gfile.glob(GCS_PATH + '/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH_ORG + '/test_tfrecords/ld_test*.tfrec')\n\nNUM_TRAINING_IMAGES = int( count_data_items(TRAINING_FILENAMES) * (FOLDS-1.)/FOLDS )\nNUM_VALIDATION_IMAGES = int( count_data_items(TRAINING_FILENAMES) * 1/FOLDS )\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\n\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\n\nprint(f'Dataset: {NUM_TRAINING_IMAGES} training images, {NUM_VALIDATION_IMAGES} validation images, {NUM_TEST_IMAGES} (unlabeled) test images')","execution_count":null,"outputs":[]},{"metadata":{"id":"PTV3XNBmpxrd","papermill":{"duration":0.035506,"end_time":"2020-11-25T21:20:34.507478","exception":false,"start_time":"2020-11-25T21:20:34.471972","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# II. Visualization\n\n**Now that we have dealt with all the configuring required to use TPUs, we can extract our images from the TPU and finally get a look at our data:**"},{"metadata":{"_kg_hide-input":false,"execution":{"iopub.execute_input":"2020-11-25T21:20:34.629901Z","iopub.status.busy":"2020-11-25T21:20:34.628974Z","iopub.status.idle":"2020-11-25T21:20:34.632416Z","shell.execute_reply":"2020-11-25T21:20:34.633176Z"},"id":"foPjMRkhpxrf","papermill":{"duration":0.08462,"end_time":"2020-11-25T21:20:34.633385","exception":false,"start_time":"2020-11-25T21:20:34.548765","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"np.set_printoptions(threshold=15, linewidth=80)\n\nclasses = ['0', '1', '2', '3', '4']          \n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: \n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_plant(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n\ndef display_batch_of_images(databatch, predictions=None):\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n    \n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else classes[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3\n        subplot = display_one_plant(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:20:34.729156Z","iopub.status.busy":"2020-11-25T21:20:34.728284Z","iopub.status.idle":"2020-11-25T21:20:35.044615Z","shell.execute_reply":"2020-11-25T21:20:35.04385Z"},"id":"LSQ4KUiDpxrh","papermill":{"duration":0.366429,"end_time":"2020-11-25T21:20:35.044774","exception":false,"start_time":"2020-11-25T21:20:34.678345","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"training_dataset = get_dataset(TRAINING_FILENAMES, labeled=True, course_drop=False, all_aug=False,\n                               grid_mask=False, mat_aug=False, cutmixup=False, shuffle=True, repeat=True)\ntraining_dataset = training_dataset.unbatch().batch(20)\ntrain_batch = iter(training_dataset)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:20:38.620011Z","iopub.status.busy":"2020-11-25T21:20:35.125495Z","iopub.status.idle":"2020-11-25T21:20:40.810363Z","shell.execute_reply":"2020-11-25T21:20:40.81096Z"},"id":"L57gjrovpxrj","papermill":{"duration":5.728805,"end_time":"2020-11-25T21:20:40.81111","exception":false,"start_time":"2020-11-25T21:20:35.082305","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# run this cell again for another randomized set of training images\ndisplay_batch_of_images(next(train_batch))","execution_count":null,"outputs":[]},{"metadata":{"id":"vE65QNftpxro","papermill":{"duration":0.074351,"end_time":"2020-11-25T21:20:40.960905","exception":false,"start_time":"2020-11-25T21:20:40.886554","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# III. Augmentation\n\n**Note: the following augmentation implementation is taken from [Chris Deotte](https://www.kaggle.com/cdeotte)'s notebook, which can be found [here](https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu)**\n\n## Rotation, Shift, Zoom, Shear"},{"metadata":{"_kg_hide-input":false,"execution":{"iopub.execute_input":"2020-11-25T21:20:41.142004Z","iopub.status.busy":"2020-11-25T21:20:41.13034Z","iopub.status.idle":"2020-11-25T21:20:41.145801Z","shell.execute_reply":"2020-11-25T21:20:41.144997Z"},"id":"dId3DSXjpxrp","papermill":{"duration":0.108653,"end_time":"2020-11-25T21:20:41.14593","exception":false,"start_time":"2020-11-25T21:20:41.037277","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n\n    rotation = math.pi * rotation / 180.\n    shear = math.pi * shear / 180.\n    \n    c1 = tf.math.cos(rotation)\n    s1 = tf.math.sin(rotation)\n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    rotation_matrix = tf.reshape( tf.concat([c1,s1,zero, -s1,c1,zero, zero,zero,one],axis=0),[3,3] )\n\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)\n    shear_matrix = tf.reshape( tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3] )    \n\n    zoom_matrix = tf.reshape( tf.concat([one/height_zoom,zero,zero, zero,one/width_zoom,zero, zero,zero,one],axis=0),[3,3] )\n    \n    shift_matrix = tf.reshape( tf.concat([one,zero,height_shift, zero,one,width_shift, zero,zero,one],axis=0),[3,3] )\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), K.dot(zoom_matrix, shift_matrix))\n\ndef transform(image, label, DIM = IMAGE_SIZE[0]):\n\n    XDIM = DIM % 2\n    \n    rot = 15. * tf.random.normal([1],dtype='float32')\n    shr = 5. * tf.random.normal([1],dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1],dtype='float32')/10.\n    w_zoom = 1.0 + tf.random.normal([1],dtype='float32')/10.\n    h_shift = 16. * tf.random.normal([1],dtype='float32') \n    w_shift = 16. * tf.random.normal([1],dtype='float32') \n\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    idx2 = K.dot(m,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n           \n    idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image,tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM,DIM,3]),label","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:20:41.306132Z","iopub.status.busy":"2020-11-25T21:20:41.303795Z","iopub.status.idle":"2020-11-25T21:20:45.070677Z","shell.execute_reply":"2020-11-25T21:20:45.071296Z"},"id":"RJyIDNWcBvDc","outputId":"a4cdaf0a-4197-4f18-c6f1-ec92605bb4c5","papermill":{"duration":3.851067,"end_time":"2020-11-25T21:20:45.071453","exception":false,"start_time":"2020-11-25T21:20:41.220386","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#view augmentation\nrow = 4; col = 4;\nrow = min(row,BATCH_SIZE//col)\nall_elements = get_dataset(TRAINING_FILENAMES, labeled=True, mat_aug=True,\n                           course_drop=False, cutmixup=False, shuffle=True, repeat=True)\n\nall_elements = all_elements.unbatch().batch(20)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,15))\n    for j in range(16):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"id":"eJAzFNJEBvDe","papermill":{"duration":0.107021,"end_time":"2020-11-25T21:20:45.28893","exception":false,"start_time":"2020-11-25T21:20:45.181909","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Coarse Dropout\n\n**We can use coarse dropout augmentation for online augmentation. Note that we have an option here: do we apply `transform` for augmentation or `dropout` and for what type of augmentation? You can easily experiment with either (or a combination of both) by changing the parameters of the `get_dataset` function earlier defined:**"},{"metadata":{"_kg_hide-input":false,"execution":{"iopub.execute_input":"2020-11-25T21:20:45.52003Z","iopub.status.busy":"2020-11-25T21:20:45.519037Z","iopub.status.idle":"2020-11-25T21:20:45.522268Z","shell.execute_reply":"2020-11-25T21:20:45.521622Z"},"id":"Psgq2EVwBvDe","papermill":{"duration":0.12721,"end_time":"2020-11-25T21:20:45.522394","exception":false,"start_time":"2020-11-25T21:20:45.395184","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def dropout(image, DIM = IMAGE_SIZE[0], PROBABILITY = 1, CT = 8, SZ = 0.2):\n    \n    prob = tf.cast( tf.random.uniform([],0,1)<PROBABILITY, tf.int32)\n    if (prob==0)|(CT==0)|(SZ==0): return image\n    \n    for k in range(CT):\n\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        WIDTH = tf.cast( SZ*DIM,tf.int32) * prob\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n\n        one = image[ya:yb,0:xa,:]\n        two = tf.zeros([yb-ya,xb-xa,3]) \n        three = image[ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        image = tf.concat([image[0:ya,:,:],middle,image[yb:DIM,:,:]],axis=0)\n            \n    image = tf.reshape(image,[DIM,DIM,3])\n    \n    return image","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:20:45.77374Z","iopub.status.busy":"2020-11-25T21:20:45.772629Z","iopub.status.idle":"2020-11-25T21:20:49.425845Z","shell.execute_reply":"2020-11-25T21:20:49.426441Z"},"id":"FNQk1QpcBvDh","outputId":"077f46af-6492-4698-f8ba-57c71e4ca4a8","papermill":{"duration":3.774611,"end_time":"2020-11-25T21:20:49.426597","exception":false,"start_time":"2020-11-25T21:20:45.651986","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"row = 4; col = 4;\nrow = min(row,BATCH_SIZE//col)\nall_elements = get_dataset(TRAINING_FILENAMES, labeled=True, mat_aug=False, cutmixup=False,\n                           grid_mask=False, course_drop=True, shuffle=True, repeat=True)\n\nall_elements = all_elements.unbatch().batch(20)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,15))\n    for j in range(16):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"id":"oKLsDyR1BvDi","papermill":{"duration":0.136847,"end_time":"2020-11-25T21:20:49.703414","exception":false,"start_time":"2020-11-25T21:20:49.566567","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## GridMask\n\n**Now we can explore GridMask, which is essentially a less randomized Course Dropout. The below code is taken from [this notebook](https://www.kaggle.com/xiejialun/gridmask-data-augmentation-with-tensorflow)**"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:20:49.985912Z","iopub.status.busy":"2020-11-25T21:20:49.984815Z","iopub.status.idle":"2020-11-25T21:20:49.988239Z","shell.execute_reply":"2020-11-25T21:20:49.98749Z"},"id":"bg9-fid4BvDj","papermill":{"duration":0.146581,"end_time":"2020-11-25T21:20:49.988365","exception":false,"start_time":"2020-11-25T21:20:49.841784","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"AugParams = {\n    'd1' : 100,\n    'd2': 160,\n    'rotate' : 45,\n    'ratio' : 0.4\n}","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"execution":{"iopub.execute_input":"2020-11-25T21:20:50.435029Z","iopub.status.busy":"2020-11-25T21:20:50.434215Z","iopub.status.idle":"2020-11-25T21:20:50.536688Z","shell.execute_reply":"2020-11-25T21:20:50.536008Z"},"id":"XGaqILypBvDk","papermill":{"duration":0.410475,"end_time":"2020-11-25T21:20:50.536816","exception":false,"start_time":"2020-11-25T21:20:50.126341","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def transform2(image, inv_mat, image_shape):\n\n    h, w, c = image_shape\n    cx, cy = w//2, h//2\n\n    new_xs = tf.repeat( tf.range(-cx, cx, 1), h)\n    new_ys = tf.tile( tf.range(-cy, cy, 1), [w])\n    new_zs = tf.ones([h*w], dtype=tf.int32)\n\n    old_coords = tf.matmul(inv_mat, tf.cast(tf.stack([new_xs, new_ys, new_zs]), tf.float32))\n    old_coords_x, old_coords_y = tf.round(old_coords[0, :] + w//2), tf.round(old_coords[1, :] + h//2)\n\n    clip_mask_x = tf.logical_or(old_coords_x<0, old_coords_x>w-1)\n    clip_mask_y = tf.logical_or(old_coords_y<0, old_coords_y>h-1)\n    clip_mask = tf.logical_or(clip_mask_x, clip_mask_y)\n\n    old_coords_x = tf.boolean_mask(old_coords_x, tf.logical_not(clip_mask))\n    old_coords_y = tf.boolean_mask(old_coords_y, tf.logical_not(clip_mask))\n    new_coords_x = tf.boolean_mask(new_xs+cx, tf.logical_not(clip_mask))\n    new_coords_y = tf.boolean_mask(new_ys+cy, tf.logical_not(clip_mask))\n\n    old_coords = tf.cast(tf.stack([old_coords_y, old_coords_x]), tf.int32)\n    new_coords = tf.cast(tf.stack([new_coords_y, new_coords_x]), tf.int64)\n    rotated_image_values = tf.gather_nd(image, tf.transpose(old_coords))\n    rotated_image_channel = list()\n    for i in range(c):\n        vals = rotated_image_values[:,i]\n        sparse_channel = tf.SparseTensor(tf.transpose(new_coords), vals, [h, w])\n        rotated_image_channel.append(tf.sparse.to_dense(sparse_channel, default_value=0, validate_indices=False))\n\n    return tf.transpose(tf.stack(rotated_image_channel), [1,2,0])\n\ndef random_rotate(image, angle, image_shape):\n\n    def get_rotation_mat_inv(angle):\n        angle = math.pi * angle / 180\n\n        cos_val = tf.math.cos(angle)\n        sin_val = tf.math.sin(angle)\n        one = tf.constant([1], tf.float32)\n        zero = tf.constant([0], tf.float32)\n\n        rot_mat_inv = tf.concat([cos_val, sin_val, zero,\n                                     -sin_val, cos_val, zero,\n                                     zero, zero, one], axis=0)\n        rot_mat_inv = tf.reshape(rot_mat_inv, [3,3])\n\n        return rot_mat_inv\n    angle = float(angle) * tf.random.normal([1],dtype='float32')\n    rot_mat_inv = get_rotation_mat_inv(angle)\n    return transform2(image, rot_mat_inv, image_shape)\n\n\ndef GridMask(image_height, image_width, d1, d2, rotate_angle=1, ratio=0.5):\n\n    h, w = image_height, image_width\n    hh = int(np.ceil(np.sqrt(h*h+w*w)))\n    hh = hh+1 if hh%2==1 else hh\n    d = tf.random.uniform(shape=[], minval=d1, maxval=d2, dtype=tf.int32)\n    l = tf.cast(tf.cast(d,tf.float32)*ratio+0.5, tf.int32)\n\n    st_h = tf.random.uniform(shape=[], minval=0, maxval=d, dtype=tf.int32)\n    st_w = tf.random.uniform(shape=[], minval=0, maxval=d, dtype=tf.int32)\n\n    y_ranges = tf.range(-1 * d + st_h, -1 * d + st_h + l)\n    x_ranges = tf.range(-1 * d + st_w, -1 * d + st_w + l)\n\n    for i in range(0, hh//d+1):\n        s1 = i * d + st_h\n        s2 = i * d + st_w\n        y_ranges = tf.concat([y_ranges, tf.range(s1,s1+l)], axis=0)\n        x_ranges = tf.concat([x_ranges, tf.range(s2,s2+l)], axis=0)\n\n    x_clip_mask = tf.logical_or(x_ranges <0 , x_ranges > hh-1)\n    y_clip_mask = tf.logical_or(y_ranges <0 , y_ranges > hh-1)\n    clip_mask = tf.logical_or(x_clip_mask, y_clip_mask)\n\n    x_ranges = tf.boolean_mask(x_ranges, tf.logical_not(clip_mask))\n    y_ranges = tf.boolean_mask(y_ranges, tf.logical_not(clip_mask))\n\n    hh_ranges = tf.tile(tf.range(0,hh), [tf.cast(tf.reduce_sum(tf.ones_like(x_ranges)), tf.int32)])\n    x_ranges = tf.repeat(x_ranges, hh)\n    y_ranges = tf.repeat(y_ranges, hh)\n\n    y_hh_indices = tf.transpose(tf.stack([y_ranges, hh_ranges]))\n    x_hh_indices = tf.transpose(tf.stack([hh_ranges, x_ranges]))\n\n    y_mask_sparse = tf.SparseTensor(tf.cast(y_hh_indices, tf.int64),  tf.zeros_like(y_ranges), [hh, hh])\n    y_mask = tf.sparse.to_dense(y_mask_sparse, 1, False)\n\n    x_mask_sparse = tf.SparseTensor(tf.cast(x_hh_indices, tf.int64), tf.zeros_like(x_ranges), [hh, hh])\n    x_mask = tf.sparse.to_dense(x_mask_sparse, 1, False)\n\n    mask = tf.expand_dims( tf.clip_by_value(x_mask + y_mask, 0, 1), axis=-1)\n\n    mask = random_rotate(mask, rotate_angle, [hh, hh, 1])\n    mask = tf.image.crop_to_bounding_box(mask, (hh-h)//2, (hh-w)//2, image_height, image_width)\n\n    return mask\n\ndef apply_grid_mask(image, label):\n    mask = GridMask(IMAGE_SIZE[0],\n                    IMAGE_SIZE[1],\n                    AugParams['d1'],\n                    AugParams['d2'],\n                    AugParams['rotate'],\n                    AugParams['ratio'])\n    \n    if IMAGE_SIZE[-1] == 3:\n        mask = tf.concat([mask, mask, mask], axis=-1)\n\n    return tf.cast(image * tf.cast(mask, tf.float32), tf.float32), label","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:20:50.825251Z","iopub.status.busy":"2020-11-25T21:20:50.824424Z","iopub.status.idle":"2020-11-25T21:20:56.681057Z","shell.execute_reply":"2020-11-25T21:20:56.682208Z"},"id":"ZcjUaKzMBvDm","outputId":"74088035-fd38-4915-fc8b-fc9c70ebd86e","papermill":{"duration":6.007043,"end_time":"2020-11-25T21:20:56.682453","exception":false,"start_time":"2020-11-25T21:20:50.67541","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#view what grid mask looks like\nrow = 4; col = 4;\nrow = min(row,BATCH_SIZE//col)\nall_elements = get_dataset(TRAINING_FILENAMES, labeled=True, grid_mask=True, mat_aug=False,\n                           cutmixup=False, course_drop=False, shuffle=True, repeat=True)\n\nall_elements = all_elements.unbatch().batch(20)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,15))\n    for j in range(16):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"id":"F7XqVflZBvDp","papermill":{"duration":0.177295,"end_time":"2020-11-25T21:20:57.036156","exception":false,"start_time":"2020-11-25T21:20:56.858861","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## All The Above\n\n**And now we will create a function to apply all the above augmentations with certain probabilities. The current set up delivers a 25/25/25/25 split between rotation/shear/shift/zoom, course dropout, grid mask, and no augmentation. The `no_grid` parameter shifts this to a 50/25/25 split between rotation/shear/shift/zoom, course dropout, and no augmentation**"},{"metadata":{"_kg_hide-input":false,"execution":{"iopub.execute_input":"2020-11-25T21:20:57.403735Z","iopub.status.busy":"2020-11-25T21:20:57.402714Z","iopub.status.idle":"2020-11-25T21:20:57.405894Z","shell.execute_reply":"2020-11-25T21:20:57.405304Z"},"id":"DI69Cka3BvDp","papermill":{"duration":0.194275,"end_time":"2020-11-25T21:20:57.406031","exception":false,"start_time":"2020-11-25T21:20:57.211756","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def apply_all_aug(img, label, no_grid = False):\n    \n    if tf.random.uniform([],0,1) > .5:\n        if tf.random.uniform([],0,1) > .5:\n            img, label = transform(img, label)\n            \n        #apply droupout  \n        else:\n            img = dropout(img, PROBABILITY = 1)\n            \n    else:\n        if not no_grid:\n        #apply grid mask\n            if tf.random.uniform([],0,1) > .5:\n                img, label = apply_grid_mask(img, label)\n                \n            else:\n                #do nothing\n                img, label = img, label\n                \n        else:\n        #apply transform \n            if tf.random.uniform([],0,1) > .5:\n                img, label = transform(img, label)\n                \n            else:\n                #do nothing\n                img, label = img, label\n            \n    return img, label","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:20:57.774456Z","iopub.status.busy":"2020-11-25T21:20:57.771274Z","iopub.status.idle":"2020-11-25T21:21:02.057928Z","shell.execute_reply":"2020-11-25T21:21:02.058638Z"},"id":"v6anVdKqBvDq","outputId":"b03b1934-baa7-49bd-b803-34e12bd2f958","papermill":{"duration":4.474291,"end_time":"2020-11-25T21:21:02.058859","exception":false,"start_time":"2020-11-25T21:20:57.584568","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#view what augmentation combination looks like\nrow = 4; col = 4;\nrow = min(row,BATCH_SIZE//col)\nall_elements = get_dataset(TRAINING_FILENAMES, labeled=True, grid_mask=False, all_aug=True, \n                           course_drop=False, cutmixup=False, shuffle=True, repeat=True)\n\nall_elements = all_elements.unbatch().batch(20)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,15))\n    for j in range(16):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"id":"CUzorS6rpxrs","papermill":{"duration":0.204933,"end_time":"2020-11-25T21:21:02.541994","exception":false,"start_time":"2020-11-25T21:21:02.337061","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## MixUp\n\n**Now, the augmentations we did above are great, but we are still adding noise to the images which is also leading to information loss. Luckily, we can do better with mixup. Essentially, all mixup does is randomly converts images to convex combinations of pairs of images and their labels, as seen in the illustration below:**\n\n![mixup](http://miro.medium.com/max/362/0*yLCQYAtNAh28LQks.png)\nImage from [here](http://medium.com/swlh/how-to-do-mixup-training-from-image-files-in-keras-fe1e1c1e6da6)\n\n**We can see that we retain information about both images and their labels while introducing regularization into our model. For more on MixUp, read [this](https://arxiv.org/abs/1710.09412). The following MixUp and CutMix codes are taken from Chris Deotte (again) in his notebook [here](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96)**"},{"metadata":{"_kg_hide-input":false,"execution":{"iopub.execute_input":"2020-11-25T21:21:02.971425Z","iopub.status.busy":"2020-11-25T21:21:02.96853Z","iopub.status.idle":"2020-11-25T21:21:02.974884Z","shell.execute_reply":"2020-11-25T21:21:02.97411Z"},"id":"8-YKE75-pxru","papermill":{"duration":0.22801,"end_time":"2020-11-25T21:21:02.97502","exception":false,"start_time":"2020-11-25T21:21:02.74701","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#need to one hot encode images so we can blend their labels like above\ndef onehot(image,label):\n    CLASSES = len(classes)\n    return image,tf.one_hot(label,CLASSES)\n\ndef mixup(image, label, PROBABILITY = 1.0, DIM = IMAGE_SIZE[0]):\n    CLASSES = len(classes)\n    \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.float32)\n\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n        a = tf.random.uniform([],0,1)*P\n\n        img1 = image[j,]\n        img2 = image[k,]\n        imgs.append((1-a)*img1 + a*img2)\n\n        if len(label.shape)==1:\n            lab1 = tf.one_hot(label[j],CLASSES)\n            lab2 = tf.one_hot(label[k],CLASSES)\n        else:\n            lab1 = label[j,]\n            lab2 = label[k,]\n        labs.append((1-a)*lab1 + a*lab2)\n\n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image2,label2","execution_count":null,"outputs":[]},{"metadata":{"id":"1vCXROMMpxrw","papermill":{"duration":0.215127,"end_time":"2020-11-25T21:21:03.395073","exception":false,"start_time":"2020-11-25T21:21:03.179946","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## CutMix\n\n**CutMix is essentially the same as mixup except the images are not blended together, rather a random sized block of one image is superimposed on another image. You can read more about it [here](http://arxiv.org/pdf/1905.04899.pdf)**"},{"metadata":{"_kg_hide-input":false,"execution":{"iopub.execute_input":"2020-11-25T21:21:03.841284Z","iopub.status.busy":"2020-11-25T21:21:03.840178Z","iopub.status.idle":"2020-11-25T21:21:03.843935Z","shell.execute_reply":"2020-11-25T21:21:03.843206Z"},"id":"35QPZNHypxry","papermill":{"duration":0.23608,"end_time":"2020-11-25T21:21:03.84407","exception":false,"start_time":"2020-11-25T21:21:03.60799","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def cutmix(image, label, PROBABILITY = 1.0, DIM = IMAGE_SIZE[0]):\n\n    CLASSES = len(classes)\n    \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.int32)\n\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        b = tf.random.uniform([],0,1)\n        WIDTH = tf.cast( DIM * tf.math.sqrt(1-b),tf.int32) * P\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n\n        one = image[j,ya:yb,0:xa,:]\n        two = image[k,ya:yb,xa:xb,:]\n        three = image[j,ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        img = tf.concat([image[j,0:ya,:,:],middle,image[j,yb:DIM,:,:]],axis=0)\n        imgs.append(img)\n\n        a = tf.cast(WIDTH*WIDTH/DIM/DIM,tf.float32)\n        if len(label.shape)==1:\n            lab1 = tf.one_hot(label[j],CLASSES)\n            lab2 = tf.one_hot(label[k],CLASSES)\n        else:\n            lab1 = label[j,]\n            lab2 = label[k,]\n        labs.append((1-a)*lab1 + a*lab2)\n        \n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image2,label2","execution_count":null,"outputs":[]},{"metadata":{"id":"yxZVp7lVpxrz","papermill":{"duration":0.208471,"end_time":"2020-11-25T21:21:04.259855","exception":false,"start_time":"2020-11-25T21:21:04.051384","status":"completed"},"tags":[]},"cell_type":"markdown","source":"### MixUp + CutMix\n\n**It is hard to choose which is better, mixup or CutMix. Luckily, we don't actually have to choose because we can just apply both. We can apply CutMix `SWITCH * CUTMIX_PROB` of the time and mixup `(1 - SWITCH) * MIXUP_PROB` of the time. We will need to experiment a bit to see which convex combination delivers the best performance, but this is a good starting point: mixup 33% of the time, CutMix 33% of the time, and no augmentation 33% of the time**"},{"metadata":{"_kg_hide-input":false,"execution":{"iopub.execute_input":"2020-11-25T21:21:04.698843Z","iopub.status.busy":"2020-11-25T21:21:04.697865Z","iopub.status.idle":"2020-11-25T21:21:04.701046Z","shell.execute_reply":"2020-11-25T21:21:04.700303Z"},"id":"4ohstL7Dpxrz","papermill":{"duration":0.226321,"end_time":"2020-11-25T21:21:04.701188","exception":false,"start_time":"2020-11-25T21:21:04.474867","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#create function to apply both cutmix and mixup\ndef cut_and_mix(image, label, DIM = IMAGE_SIZE[0]):\n    CLASSES = len(classes)\n    \n    #define how often we want to do activate cutmix or mixup\n    SWITCH = 1/2\n    \n    #define how often we want cutmix or mixup to activate when switch is active\n    CUTMIX_PROB = 2/3\n    MIXUP_PROB = 2/3\n    \n    #apply cutmix and mixup\n    image2, label2 = cutmix(image, label, CUTMIX_PROB)\n    image3, label3 = mixup(image, label, MIXUP_PROB)\n    imgs = []; labs = []\n    \n    for j in range(BATCH_SIZE):\n        P = tf.cast( tf.random.uniform([],0,1)<=SWITCH, tf.float32)\n        imgs.append(P*image2[j,]+(1-P)*image3[j,])\n        labs.append(P*label2[j,]+(1-P)*label3[j,])\n        \n    #must explicitly reshape so TPU complier knows output shape\n    image4 = tf.reshape(tf.stack(imgs),(BATCH_SIZE,DIM,DIM,3))\n    label4 = tf.reshape(tf.stack(labs),(BATCH_SIZE,CLASSES))\n    return image4,label4","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:21:05.131488Z","iopub.status.busy":"2020-11-25T21:21:05.130631Z","iopub.status.idle":"2020-11-25T21:22:19.754218Z","shell.execute_reply":"2020-11-25T21:22:19.754809Z"},"id":"9udUZGZNpxr1","outputId":"ecd3cd4f-24b3-4696-a8c1-d2f4c2dbc089","papermill":{"duration":74.845653,"end_time":"2020-11-25T21:22:19.754976","exception":false,"start_time":"2020-11-25T21:21:04.909323","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#view what CutMix/MixUp combination looks like\nrow = 4; col = 4;\nrow = min(row,BATCH_SIZE//col)\nall_elements = get_dataset(TRAINING_FILENAMES, labeled=True, grid_mask=False, all_aug=False,  \n                           cutmixup=True, course_drop=False, shuffle=True, repeat=True)\n\nall_elements = all_elements.unbatch().batch(20)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,15))\n    for j in range(16):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"id":"AiKoUtptpxr3","papermill":{"duration":0.241469,"end_time":"2020-11-25T21:22:20.234616","exception":false,"start_time":"2020-11-25T21:22:19.993147","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# IV. Model Training\n\n**We will use a specific learning rate schedule for this task. Since we are transfer learning, we do not want to start off with too large a learning rate, or we will erase the intelligence of the model already contained in its weights. Instead, we begin with a very small learning rate and increase it gradually before lowering it again to fine-tune the weights.**"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:22:20.747883Z","iopub.status.busy":"2020-11-25T21:22:20.741983Z","iopub.status.idle":"2020-11-25T21:22:20.912Z","shell.execute_reply":"2020-11-25T21:22:20.91257Z"},"id":"lC9KziLUpxr3","outputId":"f1f1c35c-3437-43e7-bca9-3a5f4627ae28","papermill":{"duration":0.43589,"end_time":"2020-11-25T21:22:20.912736","exception":false,"start_time":"2020-11-25T21:22:20.476846","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"EPOCHS = 20          \n\nLR_START = 0.000005\nLR_MAX = 0.00000125 * BATCH_SIZE\nLR_MIN = 0.000001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_DECAY = .8\n\ndef lr_schedule(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lr_schedule, verbose = True)\n\nrng = [i for i in range(EPOCHS)]\ny = [lr_schedule(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"execution":{"iopub.execute_input":"2020-11-25T21:22:21.916468Z","iopub.status.busy":"2020-11-25T21:22:21.915458Z","iopub.status.idle":"2020-11-25T21:22:21.919079Z","shell.execute_reply":"2020-11-25T21:22:21.919743Z"},"id":"KBCwXOhTpxr5","papermill":{"duration":0.269239,"end_time":"2020-11-25T21:22:21.919935","exception":false,"start_time":"2020-11-25T21:22:21.650696","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def efficientnet(image_size, classes=len(classes), b=0, LR=1e-3,\n                 one_hot=False, dropout=.2):\n    efns = [EfficientNetB0, EfficientNetB1, EfficientNetB2,\n            EfficientNetB3, EfficientNetB4, EfficientNetB5,\n            EfficientNetB6]\n    with strategy.scope():\n        efficient = efns[b](\n            input_shape=(image_size, image_size, 3),\n            weights='noisy-student', #or imagenet\n            include_top=False\n        )\n        efficient.trainable=True\n        model = tf.keras.Sequential([\n            efficient,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            tf.keras.layers.Dropout(dropout),\n            tf.keras.layers.Dense(classes, activation='softmax')\n        ])\n\n    if one_hot: \n        model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n\n    else: \n        model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\n        \n    return model\n\ndef densenet(image_size, classes=len(classes), b=0, LR=1e-3,\n                 one_hot=False, dropout=.2):\n    densenets = [tf.keras.applications.DenseNet121, tf.keras.applications.DenseNet169,\n            tf.keras.applications.DenseNet201]\n    with strategy.scope():\n        densenet = densenets[b](\n            input_shape=(image_size, image_size, 3),\n            weights='imagenet',\n            include_top=False\n        )\n        densenet.trainable=True\n        model = tf.keras.Sequential([\n            densenet,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            tf.keras.layers.Dropout(dropout),\n            tf.keras.layers.Dense(classes, activation='softmax')\n        ])\n\n    if one_hot: \n        model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n\n    else: \n        model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\n        \n    return model","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-25T21:22:22.948907Z","iopub.status.busy":"2020-11-25T21:22:22.947844Z","iopub.status.idle":"2020-11-26T00:06:44.085489Z","shell.execute_reply":"2020-11-26T00:06:44.084684Z"},"papermill":{"duration":9861.406867,"end_time":"2020-11-26T00:06:44.085623","exception":false,"start_time":"2020-11-25T21:22:22.678756","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import KFold\n\nhistories = []\noof_pred = []; oof_labels = []\n\nkfold = KFold(FOLDS, shuffle = True, random_state = SEED)\n\nfor f, (train_index, val_index) in enumerate(kfold.split(TRAINING_FILENAMES)):\n    \n    #show fold info\n    if DEVICE=='TPU':\n        #hack to clear TPU memory\n        if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n    print('#'*25); print('FOLD',f+1); print('#'*25); print('')\n    print('Getting datasets...'); print('')\n    \n    #convert files to datasets\n    train_ds = get_dataset(list(pd.DataFrame({'TRAINING_FILENAMES': TRAINING_FILENAMES}).loc[train_index]['TRAINING_FILENAMES']),\n                           course_drop=True if AUG_TYPE is 'COURSE_DROP' else False,\n                           mat_aug=True if AUG_TYPE is 'MAT_AUG' else False,\n                           grid_mask=True if AUG_TYPE is 'GRID_MASK' else False,\n                           all_aug=True if AUG_TYPE is 'ATA' else False,\n                           cutmixup=True if AUG_TYPE is 'CUTMIXUP' else False,\n                           one_hot=False, labeled=True, return_image_names=True, repeat=True, shuffle=True)  \n\n    val_ds = get_dataset(list(pd.DataFrame({'TRAINING_FILENAMES': TRAINING_FILENAMES}).loc[val_index]['TRAINING_FILENAMES']),\n                         course_drop=False, mat_aug=False, grid_mask=False, all_aug=False, \n                         cutmixup=False, one_hot=True if AUG_TYPE is 'CUTMIXUP' else False,\n                         labeled=True, return_image_names=False, repeat=False, shuffle=False)\n    \n    sv_loss = tf.keras.callbacks.ModelCheckpoint(f'EFFNET_{f}_{SEED}_{AUG_TYPE}_{IMAGE_SIZE[0]}.h5',\n                                                monitor='val_loss',\n                                                verbose=VERBOSE,\n                                                save_best_only=True,\n                                                save_weights_only=True)\n    \n    sv_acc = tf.keras.callbacks.ModelCheckpoint(f'EFFNET_{f}_{SEED}_{AUG_TYPE}_{IMAGE_SIZE[0]}.h5',\n                                                monitor='val_categorical_accuracy' if AUG_TYPE is 'CUTMIXUP' else 'val_sparse_categorical_accuracy',\n                                                verbose=VERBOSE,\n                                                save_best_only=True,\n                                                save_weights_only=True)\n\n    print('Getting model...'); print(''); print('Training model...'); print('')\n    \n    model = efficientnet(b=4, image_size=IMAGE_SIZE[0], \n                         one_hot=True if AUG_TYPE is 'CUTMIXUP' else False)\n        \n    #model = densenet(b=1, image_size=IMAGE_SIZE[0], \n                         #one_hot=True if AUG_TYPE is 'CUTMIXUP' else False)\n        \n    history = model.fit(train_ds, validation_data=val_ds, callbacks=[lr_callback, sv_loss],\n                        verbose=VERBOSE, steps_per_epoch=STEPS_PER_EPOCH, epochs=EPOCHS,\n    )\n    \n    model.load_weights(f'EFFNET_{f}_{SEED}_{AUG_TYPE}_{IMAGE_SIZE[0]}.h5')\n    model.save(f'EFFNET_{f}_{SEED}_{AUG_TYPE}_{IMAGE_SIZE[0]}_full.h5')\n    histories.append(history)\n    \n    #get OOF predictions\n    oof_labels.append([target.numpy() for img, target in iter(val_ds.unbatch())])\n    x_oof = val_ds.map(lambda image, image_name: image)\n    oof_pred.append(np.argmax(model.predict(x_oof), axis=-1))\n\n    del model; z = gc.collect()\n    \n    if FIRST_FOLD_ONLY:\n        break","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"execution":{"iopub.execute_input":"2020-11-26T00:06:50.331639Z","iopub.status.busy":"2020-11-26T00:06:50.326524Z","iopub.status.idle":"2020-11-26T00:06:50.335325Z","shell.execute_reply":"2020-11-26T00:06:50.335861Z"},"id":"XnOB1oilpxr9","papermill":{"duration":3.090146,"end_time":"2020-11-26T00:06:50.33607","exception":false,"start_time":"2020-11-26T00:06:47.245924","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def plot_learning_curves(histories, one_hot = False): \n    fig, ax = plt.subplots(1, 2, figsize = (20, 10))\n    \n    if one_hot:\n        for i in range(0, 3):\n            ax[0].plot(histories[i].history['categorical_accuracy'], color = 'C0')\n            ax[0].plot(histories[i].history['val_categorical_accuracy'], color = 'C1')\n\n    else:\n        for i in range(0, 3):\n            ax[0].plot(histories[i].history['sparse_categorical_accuracy'], color = 'C0')\n            ax[0].plot(histories[i].history['val_sparse_categorical_accuracy'], color = 'C1')\n\n    for i in range(0, 3):\n        ax[1].plot(histories[i].history['loss'], color = 'C0')\n        ax[1].plot(histories[i].history['val_loss'], color = 'C1')\n\n    ax[0].legend(['train', 'validation'], loc = 'upper left')\n    ax[1].legend(['train', 'validation'], loc = 'upper right')\n    \n    fig.suptitle(\"Model Performance\", fontsize=14)\n    \n    for i in range(0,2):\n        ax[0].set_ylabel('Accuracy')\n        ax[0].set_xlabel('Epoch')\n        ax[1].set_ylabel('Loss')\n        ax[1].set_xlabel('Epoch')\n\n    return plt.show()","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-26T00:06:56.526275Z","iopub.status.busy":"2020-11-26T00:06:56.525179Z","iopub.status.idle":"2020-11-26T00:06:56.955199Z","shell.execute_reply":"2020-11-26T00:06:56.954499Z"},"id":"gX8GRENrpxr-","outputId":"56786033-f8bc-4f8e-dc85-4d779fc2de90","papermill":{"duration":3.517607,"end_time":"2020-11-26T00:06:56.955335","exception":false,"start_time":"2020-11-26T00:06:53.437728","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"if not FIRST_FOLD_ONLY:\n    plot_learning_curves(histories, one_hot=True if AUG_TYPE is 'CUTMIXUP' else False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_true = np.concatenate(oof_labels)\ny_preds = np.concatenate(oof_pred)\n\nprint(classification_report(np.argmax(y_true, axis=1) if AUG_TYPE is 'CUTMIXUP' else y_true, y_preds))\nprint(f\"OOF accuracy score: {accuracy_score(np.argmax(y_true, axis=1) if AUG_TYPE is 'CUTMIXUP' else y_true, y_preds)}\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}