{"cells":[{"metadata":{"id":"lKUdMDFOyaZK"},"cell_type":"markdown","source":"# Intro"},{"metadata":{"id":"X_pu6fU2yePf"},"cell_type":"markdown","source":"This book is a brief exploration of various procedures for calculating predictions from test time augmentations.\n\nThe test time augmentation procedure implemented below results in 42 versions of each original image, based on five crops from four  scales with horizontal flips, plus the original image and its horizontal flip. This procedure is an abbreviated version of the procedure followed in the [Going deeper with convolutions paper](https://arxiv.org/pdf/1409.4842.pdf), which was cited in the recent [Big Transfer paper](https://arxiv.org/pdf/1912.11370.pdf).\n\nThree different prediction procedures are evaluated against baseline predictions for original images, including:\n\n* Argmax of average of softmax probs of each image version\n* Argmax of max of softmax probs of each image version\n* Mode of argmax of softmax probs of each image version\n* Argmax of softmax of average of logits of each image version\n\nTo enable this book to run quickly, predictions for comparison are generated from the relatively small Efficientnet-B0 model (4.1 million parameters), trained on only images from the training set with an original size of 224x224. Training images are augmented with random horizontal flip, scale and crop. The model starts with noisy-student pretrained weights and runs for 30 epochs using sparse categorical cross entropy for the loss function and Adam for the optimizer. The base batch size is 16 (128 across 8 replicas) and the base learning rate is 0.003 (0.0015 following the batch_size / 256 convention).\n\nThe baseline macro f1-score s 0.921. The best score of 0.933 results from averaging softmax probs, a 16% reduction in total error (numbers may vary slightly with different runs of this book)."},{"metadata":{"id":"2aRcKOPwamWo"},"cell_type":"markdown","source":"# Setup"},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"id":"xv8DfDVqeaWd","trusted":true},"cell_type":"code","source":"%matplotlib inline\n%load_ext autoreload\n%autoreload 2","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"id":"5ne95ptIeaWg","trusted":true},"cell_type":"code","source":"from google.cloud import storage\nimport json\nimport math\nimport matplotlib.gridspec as gridspec\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport os\nimport pandas as pd\nfrom pathlib import Path\nfrom sklearn.metrics import classification_report\nimport subprocess\nimport sys\nimport tensorflow as tf\nimport time\nfrom tqdm.notebook import tqdm\n\nfrom tensorflow.keras.backend import dot","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"id":"yRTaszOfeaWj","trusted":true},"cell_type":"code","source":"def porc(c):\n    print_output(run_command(c))\n\ndef print_output(output):\n    \"\"\"Prints output from string.\"\"\"\n    for l in output.split('\\n'):\n        print(l)      \n\ndef run_command(command):\n    \"\"\"Runs command line command as a subprocess returning output as string.\"\"\"\n    STDOUT = subprocess.PIPE\n    process = subprocess.run(command, shell=True, check=False,\n                             stdout=STDOUT, stderr=STDOUT, universal_newlines=True)\n    return process.stdout\n\ndef show_images(imgs, titles=None, hw=(3,3), rc=(4,4)):\n    \"\"\"Show list of images with optional list of titles.\"\"\"\n    h, w = hw\n    r, c = rc\n    fig=plt.figure(figsize=(w*c, h*r))\n    gs1 = gridspec.GridSpec(r, c, fig, hspace=0.2, wspace=0.05)\n    for i in range(r*c):\n        img = imgs[i].squeeze()\n        ax = fig.add_subplot(gs1[i])\n        if titles != None:\n            ax.set_title(titles[i], {'fontsize': 10})\n        plt.imshow(img)\n        plt.axis('off')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"outputId":"88b5bce5-4d9b-492d-a8ad-f5e09c4ffba9","id":"FSnZf1wNeaWl","trusted":true,"executionInfo":{"status":"ok","timestamp":1587494208118,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":4129}},"cell_type":"code","source":"output = run_command('pip freeze | grep efficientnet')\nif output == '':\n    print_output(run_command('pip install efficientnet'))\nelse:\n    print_output(output)\nfrom efficientnet import tfkeras as efn","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"outputId":"2883cf84-81b5-4a3f-8359-cbc656f15627","id":"wCrt4EiAeaWo","trusted":true,"executionInfo":{"status":"ok","timestamp":1587494552993,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":348717}},"cell_type":"code","source":"# raise SystemExit(\"Stop right there!\")\n\nKAGGLE = os.getenv('KAGGLE_KERNEL_RUN_TYPE') != None\nBUCKET = 'flowers-caleb'\nTFRECORD_DIR = f'gs://{BUCKET}'\nDATASET_DIR = Path('flowers-tta')\n\nif KAGGLE:\n    from kaggle_datasets import KaggleDatasets\n    DATASET_DIR = '/kaggle/input'/DATASET_DIR\n    GCS_DATASET_DIR = KaggleDatasets().get_gcs_path(DATASET_DIR.parts[-1])\n    PATH = Path('/kaggle/input/flower-classification-with-tpus')\n    TFRECORD_DIR = KaggleDatasets().get_gcs_path(PATH.parts[-1])\nelse:\n    GOOGLE_DRIVE = Path('/content/drive/My Drive')\n\n    from google.colab import drive\n    drive.mount('/content/drive')\n    \n    os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = f'{GOOGLE_DRIVE}/credentials/google.json'\n\nclient = storage.Client(project='fastai-caleb')\nbucket = client.get_bucket(BUCKET)\n\nSIZES = {s: f'{s}x{s}' for s in [192, 224, 331, 512]}\nAUTO = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"outputId":"69a01fdf-0de8-4958-975e-69a1702e6082","id":"ODEnAvYyZLvB","trusted":true,"executionInfo":{"status":"ok","timestamp":1587494558343,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":353853}},"cell_type":"code","source":"if not KAGGLE:\n    output = run_command('pip freeze | grep gcsfs')\n    if output == '':\n        print_output(run_command('pip install gcsfs'))\n    else:\n        print_output(output)\n\n    output = run_command('pip freeze | kaggle')\n    if output == '':\n        print_output(run_command('pip install kaggle'))\n        KAGGLE_PATH = Path('/root/.kaggle/kaggle.json')\n        KAGGLE_PATH.parent.mkdir(exist_ok=True)\n        kaggle_json = (GOOGLE_DRIVE/'credentials/kaggle.json').read_text()\n        KAGGLE_PATH.write_text(kaggle_json)\n        KAGGLE_PATH.chmod(600)\n    else:\n        print_output(output)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"outputId":"1595a943-4450-4dfc-ab85-6013cbc813a7","id":"LZgmJcKjeaWr","trusted":true,"executionInfo":{"status":"ok","timestamp":1587494564632,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":359951}},"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n    for d in tf.config.list_physical_devices():\n        print(d)","execution_count":null,"outputs":[]},{"metadata":{"id":"I2oGTV7ieaWu"},"cell_type":"markdown","source":"# Dataset Functions"},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"id":"QWCWZaRSeaW0","trusted":true},"cell_type":"code","source":"def augment(example):\n    new_example = example.copy()\n    image = example['image']\n    \n    image = tf.image.random_flip_left_right(image)\n    \n    shape = tf.shape(image)\n    max_resize = tf.cast(tf.cast(shape[-2], tf.float32) * 0.075, tf.int32)\n    image = tf.image.resize(image, tf.shape(image)[1:3] + tf.random.uniform((),0, max_resize, tf.int32))\n    image = tf.image.random_crop(image, shape)\n\n    new_example['image'] = tf.cast(image, tf.uint8)\n    \n    return new_example\n\ndef get_tta_fn(input_size=(224, 224), scale_factor=1.0):\n    resize_factors = (np.array([256., 288., 320., 352.]) - 224) * scale_factor\n    resize_factors = tf.constant(224. / (resize_factors + 224.), tf.float32)\n\n    def get_crops(r):\n        return np.array([[0, 0, r, r], # top-left\n                            [0, 1-r, r, 1], # top-right\n                            [1-r, 0, 1, r], # bottom-left\n                            [1-r, 1-r, 1, 1], # bottom-right\n                            [(1-r)/2,(1-r)/2,1-(1-r)/2,1-(1-r)/2]]) # center\n            \n    boxes = tf.reshape(tf.map_fn(get_crops, resize_factors),(20,4))\n\n    def tta(example):\n        image = example['image']\n        del example['image']\n        new_example = {}\n\n        cropped = tf.image.crop_and_resize(image, boxes, tf.zeros(20, tf.int32),\n                                           input_size)\n        \n        flipped = tf.image.flip_left_right(cropped)\n\n        image = tf.cast(tf.image.resize(image, input_size), tf.uint8)\n\n        new_example['image'] = tf.concat([image, tf.image.flip_left_right(image),\n                                          tf.cast(cropped, tf.uint8),\n                                          tf.cast(flipped, tf.uint8)], axis=0)\n        for k in example:\n            new_example[k] = tf.repeat(example[k], 42) \n\n        return new_example\n\n    return tta\n\ndef get_preprocess_fn(input_size=(224, 224), batch_size=128, norm=None, test=False):\n    \n    def imagenet_norm(image):\n        mean = tf.constant([0.485, 0.456, 0.406])\n        std = tf.constant([0.229, 0.224, 0.225])\n        \n        return (image / tf.constant(255, tf.float32) - mean) / std\n    \n    norm_fn = {'per_image': tf.image.per_image_standardization,\n               'imagenet': imagenet_norm,\n               None: tf.image.per_image_standardization\n              }\n\n    def preprocess(batch):\n        image = tf.image.resize(batch['image'], input_size)\n        image = norm_fn[norm](image)\n\n        if test:\n            return image\n        \n        else:\n            image = tf.reshape(image, (batch_size, *input_size, 3))\n            label = tf.cast(batch['label'], tf.float32)\n            label = tf.reshape(label, (batch_size,))\n                \n            return image, label\n        \n    return preprocess\n\nclasses_filename = DATASET_DIR/'classes.csv' if KAGGLE else f'gs://{BUCKET}/classes.csv' \nCLASSES = tf.constant(pd.read_csv(classes_filename).values.squeeze(), tf.string)\ndef get_parse_fn(split):\n    def parse_fn(example):\n        features = {\"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n                    \"id\": tf.io.FixedLenFeature([], tf.string),\n                    \"class\": tf.io.FixedLenFeature([], tf.int64)}\n                \n        example = tf.io.parse_single_example(example, features)\n\n        example['image'] = tf.image.decode_jpeg(example['image'], channels=3)        \n        example['label'] = tf.cast(example['class'], tf.int32)\n        example['class'] = CLASSES[example['label']]\n        \n        return example\n\n    return parse_fn\n\ndef get_ds(split, img_size=224, batch_size=128, shuffle=False, repeat=False):\n    file_pat = get_file_pat(split, img_size)\n    \n    options = tf.data.Options()\n    options.experimental_deterministic = not shuffle\n    \n    ds = (tf.data.Dataset.list_files(file_pat, shuffle=shuffle)\n          .with_options(options)\n          .interleave(tf.data.TFRecordDataset, num_parallel_calls=AUTO)\n          .map(get_parse_fn(split), num_parallel_calls=AUTO)\n         )\n    \n    if repeat:\n        ds = ds.repeat()\n             \n    return ds.prefetch(AUTO)\n\ndef get_ds_splits(splits, img_size=224, batch_size=128, shuffle=True, repeat=False):\n    \"\"\"splits is a dict of {split: weight} for each split to include\"\"\"\n        \n    ds_dict = {split: get_ds(split, img_size, batch_size, shuffle, repeat) \\\n               for split in splits}\n    \n    ds = tf.data.experimental.sample_from_datasets([ds_dict[s] for s in splits], \\\n                                                   [splits[s] for s in splits])\n        \n    if 'train' in splits:\n        ds = ds.shuffle(4096).batch(batch_size).map(augment, num_parallel_calls=AUTO)\n    \n    else:\n        ds = ds.batch(batch_size)\n    \n    return ds.prefetch(AUTO)\n\ndef get_file_pat(split, img_size):\n    return f'{TFRECORD_DIR}/tfrecords-jpeg-{SIZES[img_size]}/{split}/*.tfrec'","execution_count":null,"outputs":[]},{"metadata":{"id":"VNoqIOV-eaW2","trusted":true},"cell_type":"code","source":"ds = get_ds('val').batch(16)\nds_iter = iter(ds)","execution_count":null,"outputs":[]},{"metadata":{"outputId":"4d239f6c-a404-492c-e865-d8256bb65b81","id":"FDw2tALkeaW4","trusted":true,"executionInfo":{"status":"ok","timestamp":1587501214818,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":4325}},"cell_type":"code","source":"b = next(ds_iter)\nb_aug = augment(b)\nshow_images(b['image'].numpy(), b['class'].numpy().tolist(), hw=(2,2), rc=(2,8))\nshow_images(b_aug['image'].numpy(), b_aug['class'].numpy().tolist(), hw=(2,2), rc=(2,8))","execution_count":null,"outputs":[]},{"metadata":{"id":"Sbe0JyPYPZJn"},"cell_type":"markdown","source":"# Class Weights"},{"metadata":{"_kg_hide-input":true,"id":"6U1o0FB4eaW7","trusted":true},"cell_type":"code","source":"splits = ['train', 'val']\n\ndef split_classes(split):\n    return [b['label'].numpy() for b in get_ds(split)]\n\ndf_split_stats = pd.concat([pd.Series(split_classes(s)).value_counts() for s in splits], axis=1).fillna(0)\ndf_split_stats.columns = splits","execution_count":null,"outputs":[]},{"metadata":{"id":"efz1yO1DeaW9"},"cell_type":"markdown","source":"Training and validation splits have similar distributions with respect to label."},{"metadata":{"outputId":"65f7000a-369c-4e6d-c229-20f1eddbcb0e","id":"xPuFhkZP_JdF","trusted":true,"executionInfo":{"status":"ok","timestamp":1587501255403,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":40961},"_kg_hide-input":true},"cell_type":"code","source":"df_split_stats['total'] = df_split_stats[splits].sum(axis=1)\ndf_split_stats.sort_values('total', ascending=False)[splits].plot(kind='bar', stacked=True, width=1.0, figsize=(15,5))\nplt.xticks(fontsize=8);","execution_count":null,"outputs":[]},{"metadata":{"id":"tswS8mBieFep"},"cell_type":"markdown","source":"Class weights are calculated based on frequency relative to max class frequency."},{"metadata":{"outputId":"8eb6964e-b7cd-44b8-8937-b68bcb477828","id":"Tb_FcdceeaXC","trusted":true,"executionInfo":{"status":"ok","timestamp":1587501256503,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":41748},"_kg_hide-input":true},"cell_type":"code","source":"df_split_stats['weight'] = df_split_stats.sum(axis=1).max() / df_split_stats.sum(axis=1)\nax = df_split_stats.sort_values('total', ascending=False)['weight'] \\\n.plot(kind='bar', figsize=(15,5), title='Class Weights')\nplt.xticks(fontsize=8);","execution_count":null,"outputs":[]},{"metadata":{"id":"kQxNy1LUeaXL"},"cell_type":"markdown","source":"# Training Datasets"},{"metadata":{"outputId":"75eea370-bf0b-4643-c4a5-c513e89b3e8b","id":"sv-TGBFlh0wF","trusted":true,"executionInfo":{"status":"ok","timestamp":1587501257240,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":42124}},"cell_type":"code","source":"img_size = 224\ninput_size = (224, 224)\nbatch_size = 16 * strategy.num_replicas_in_sync\nbase_lr = .003\nreference_lr = batch_size / 256 * base_lr\nlr_start = reference_lr / 25\nlr_end = reference_lr / 100\n\nprint('reference_lr:', reference_lr)\n\ntrain_splits = {'train': 12.7}\nds_train = get_ds_splits(train_splits, img_size=img_size, batch_size=batch_size, \\\n                         shuffle=True, repeat=True)\n\nvalid_splits = {'val': 3.7}\nds_valid = get_ds_splits(valid_splits, img_size=img_size, batch_size=batch_size, \\\n                         shuffle=False)\n\npreprocess = get_preprocess_fn(batch_size=batch_size,\n                               input_size=input_size, norm='imagenet')\n\nds_train_fit = ds_train.map(preprocess, num_parallel_calls=AUTO)\nds_valid_fit = ds_valid.map(preprocess, num_parallel_calls=AUTO)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"outputId":"e28159be-3389-4dc8-971c-080d4bc59801","id":"hIgBG_l7eaXa","trusted":true,"executionInfo":{"status":"ok","timestamp":1587501257468,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":41768}},"cell_type":"code","source":"print('split', '\\t', 'items', '\\t',  'steps')\nsplit_dict = {}\nfor split in splits:\n    items = 0\n    prefix = '/'.join(get_file_pat(split, img_size).split('/')[3:-1])\n    for b in bucket.list_blobs(prefix=prefix):\n        items += int(b.name.split('.')[0][-3:])\n    split_dict[split] = items // batch_size\n    print(split, '\\t', items, '\\t',  items // batch_size)","execution_count":null,"outputs":[]},{"metadata":{"outputId":"9f080e16-1a09-4d46-c5fd-bee83b448416","id":"vwEPW1nHbbBg","trusted":true,"executionInfo":{"status":"ok","timestamp":1587501405358,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":390}},"cell_type":"code","source":"total_steps = int(512 / batch_size * 500 * 1.5) \nsteps_per_epoch = sum([split_dict[split] for split in train_splits])\nval_steps = sum([split_dict[split] for split in valid_splits])\nepochs = total_steps // steps_per_epoch\nprint(total_steps, steps_per_epoch, val_steps, epochs)","execution_count":null,"outputs":[]},{"metadata":{"id":"AcMQ476YeaXE"},"cell_type":"markdown","source":"# Learning Rate Schedule "},{"metadata":{"_kg_hide-input":true,"id":"5k2R2KeneaXJ","trusted":true},"cell_type":"code","source":"# https://docs.fast.ai/callbacks.one_cycle.html\n# https://sgugger.github.io/the-1cycle-policy.html\n\nclass OneCycleScheduler(tf.keras.callbacks.Callback):\n    def __init__(self, epochs_up=5, epochs_across=5, epochs_down=15, lr_max=0.001,\n              lr_start=.00003, lr_end=0.00001, decay=0.93,\n              mo_max=0.95, mo_min=0.85):\n\n        super(OneCycleScheduler, self).__init__()\n\n        def one_cycle(epoch):\n\n            if epoch <= epochs_up:\n                new_lr = (lr_max - lr_start)/2  * (-math.cos((math.pi * epoch) / epochs_up) + 1) + lr_start\n                new_mo = (mo_max - mo_min)/2  * (math.cos((math.pi * epoch) / epochs_up) + 1) + mo_min\n            \n            elif epoch <= (epochs_up + epochs_across):\n                new_lr = lr_max\n                new_mo = mo_min\n            \n            elif epoch <= (epochs_up + epochs_across + epochs_down):\n                down_epoch = epoch - epochs_across - epochs_up\n                new_lr = (lr_max - lr_end)/2  * (math.cos((math.pi * down_epoch) / epochs_down) + 1) + lr_end\n                new_mo = (mo_max - mo_min)/2  * (-math.cos((math.pi * down_epoch) / epochs_down) + 1) + mo_min\n\n            else:\n                new_lr = lr_end * decay**(epoch - epochs_up - epochs_across - epochs_down)\n                new_mo = mo_max\n            \n            return new_lr, new_mo\n\n        self.schedule = one_cycle\n \n    def on_epoch_begin(self, epoch, logs):\n        scheduled_lr, scheduled_mo = self.schedule(epoch)\n        tf.keras.backend.set_value(self.model.optimizer.lr, scheduled_lr)\n        tf.keras.backend.set_value(self.model.optimizer.beta_1, scheduled_mo)\n        print(f'\\nepoch {epoch+1:02d}: learning_rate={scheduled_lr:0.3e}, beta_1={scheduled_mo:0.3e}')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"outputId":"bc47f327-5871-41e9-952b-d4af2f7fd07e","id":"H4-N3VFieaXH","trusted":true,"executionInfo":{"status":"ok","timestamp":1587501436048,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":1078}},"cell_type":"code","source":"one_cycle = OneCycleScheduler(lr_max=reference_lr, lr_start=lr_start, \\\n                              lr_end=lr_end).schedule\nfig,ax = plt.subplots()\nax2=ax.twinx()\nax.set_ylabel('Learning Rate')\nax2.set_ylabel('Momentum')\nplt.title('Learning Rate and Momentum Schedule');\nax.plot([one_cycle(e)[0] for e in range(epochs)], label='lr')\nax2.plot([one_cycle(e)[1] for e in range(epochs)], c='orange', label='beta_1');","execution_count":null,"outputs":[]},{"metadata":{"id":"hf22ad4teaXX"},"cell_type":"markdown","source":"# Model"},{"metadata":{"_kg_hide-output":true,"id":"7Oklwt9weaXY","outputId":"4458d50c-8dac-485a-f2d5-fc12b75ec6c6","trusted":true,"cellView":"code","scroll":true,"scrolled":true,"executionInfo":{"status":"ok","timestamp":1587501451817,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":14407}},"cell_type":"code","source":"with strategy.scope():\n    cnn = efn.EfficientNetB0(weights='noisy-student', include_top=False,\n                             pooling='avg', input_shape=(*input_size, 3))\n    \n    output = tf.keras.layers.Dense(len(CLASSES), activation='softmax')(cnn.output)\n    \n    model = tf.keras.Model(cnn.input, output)\n\n    opt = tf.keras.optimizers.Adam()\n    loss_fn = tf.keras.losses.SparseCategoricalCrossentropy()\n    metrics = [tf.keras.metrics.SparseCategoricalAccuracy()] \n\n    model.compile(loss=loss_fn, optimizer=opt, metrics=metrics)\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"_kg_hide-output":true,"outputId":"b8a2449d-8363-407b-ae25-0ca853dcf582","id":"Sfv80159ZdfP","trusted":true,"executionInfo":{"status":"ok","timestamp":1587501804246,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":348300}},"cell_type":"code","source":"history = model.fit(ds_train_fit,\n                    steps_per_epoch=steps_per_epoch,\n                    epochs=epochs,\n                    validation_data=ds_valid_fit,\n                    validation_steps=val_steps,\n                    validation_freq=5,\n                    callbacks=[OneCycleScheduler(lr_max=reference_lr, \\\n                                                 lr_start=lr_start, \\\n                                                 lr_end=lr_end)],\n                    class_weight=df_split_stats.weight.to_dict()\n                    )","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"outputId":"64864940-f8e2-4c1f-fc8e-d19bb0071498","id":"m-0K6dUVVbrM","trusted":true,"executionInfo":{"status":"ok","timestamp":1587501812333,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":528}},"cell_type":"code","source":"df_hist = pd.DataFrame({m: v for m, v in history.history.items() if 'val' not in m})\n\nfor m in [m for m in history.history if 'val' in m]:\n    s = pd.Series(history.history[m], index=list(range(epochs))[4::5]) \\\n    .reindex(range(epochs), method='ffill')\n    \n    df_hist[m] = s\n\ndf_hist.plot(figsize=(8, 5), title='Training History')\nplt.grid(True)\nplt.gca().set_ylim(0, 1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"QjXjYCcNmlBo"},"cell_type":"markdown","source":"# Predictions"},{"metadata":{"id":"C0NLjTSUtwSq"},"cell_type":"markdown","source":"## Test Time Augmentations"},{"metadata":{"id":"l9AG_U3_fYCB"},"cell_type":"markdown","source":"Here is a description of a test time augmentation procedure from the paper [Going deeper with convolutions](https://arxiv.org/pdf/1409.4842.pdf), which was cited in the recent [Big Transfer paper](https://arxiv.org/pdf/1912.11370.pdf).\n* *During testing, we adopted a more aggressive cropping approach than that of Krizhevsky etal. [9]. Specifically, we resize the image to 4 scales where the shorter dimension (height or width) is 256, 288, 320 and 352 respectively, take the left, center and right square of these resized images (in the case of portrait images, we take the top, center and bottom squares). For each square, we then take the 4 corners and the center 224×224 crop as well as the square resized to 224×224, and their mirrored versions. This results in 4×3×6×2 = 144 crops per image.*\n* *The softmax probabilities are averaged over multiple crops and over all the individual classifiers to obtain the final prediction.*\n\nThe procedure below follows the same general approach, but is somewhat abbreviated since the images in the tfrecord dataset for the competition have already been cropped square. The procedure takes upper-left, upper-right, lower-left, lower-right and center crops for four different resizing levels along with corresponding horizontal flips, the original image and its horizontal flip - 42 versions for each original image."},{"metadata":{"id":"Nyt60dNuMQ3X","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"scale_factor = 0.5 # scales the base level of resize\n\nds_pred_tta = get_ds_splits(valid_splits, img_size, 1, shuffle=False) \\\n.map(get_tta_fn(input_size, scale_factor=scale_factor), num_parallel_calls=AUTO).unbatch() \\\n.batch(batch_size)\n\nds_pred_tta_pp = ds_pred_tta.map(preprocess, num_parallel_calls=AUTO)\n\nresize_factors = (np.array([256., 288., 320., 352.]) - 224) * scale_factor\nresize_factors = 224. / (resize_factors + 224.)\n\ncrops = ['top-left', 'top-right', 'bottom-left', 'bottom-right', 'center']\n\ntitles = [f'{1 / z[0]:0.3f} - {z[1]}' for z in \\\n          zip(np.repeat(resize_factors, len(crops)), crops * len(resize_factors))]\ntitles.extend([f'{t} - flip' for t in titles])\n\nds_tta_iter = ds_pred_tta.unbatch().batch(len(resize_factors) * len(crops) * 2 + 2) \\\n.as_numpy_iterator()","execution_count":null,"outputs":[]},{"metadata":{"outputId":"8c5aee0b-3a37-4cef-f776-0066e8991ed8","id":"tuceNZ8OzHr6","trusted":true,"executionInfo":{"status":"ok","timestamp":1587508041059,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":5476}},"cell_type":"code","source":"b = next(ds_tta_iter)\nshow_images(b['image'][2:], titles, rc=(8,5))","execution_count":null,"outputs":[]},{"metadata":{"id":"-zp66zSkucvq"},"cell_type":"markdown","source":"## Prediction Procedures"},{"metadata":{"id":"d-zd9y9GujEw"},"cell_type":"markdown","source":"The following prediction procedures are evaluated against baseline predictions for the original images only:\n* Argmax of average of softmax probs of each image version\n* Mode of argmax of softmax probs of each image version\n* Argmax of softmax of average of logits of each image version"},{"metadata":{"id":"3dlhIkMpbHO5","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"pred_tta = model.predict(ds_pred_tta_pp)\n\nid_list_tta = []\nlabel_list_tta = []\nclass_list_tta = []\n\nfor b in ds_pred_tta.as_numpy_iterator():\n    id_list_tta.extend(b['id'].squeeze())\n    label_list_tta.extend(b['label'].squeeze())\n    class_list_tta.extend(b['class'].squeeze())","execution_count":null,"outputs":[]},{"metadata":{"id":"kabOdLmwghTM","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_pred_tta = pd.DataFrame(pred_tta)\ndf_pred_tta['id'] = id_list_tta\ndf_pred_tta['label'] = label_list_tta\ndf_pred_tta['tta_version'] = (['orig', 'orig - flip'] + titles) * (val_steps * batch_size)","execution_count":null,"outputs":[]},{"metadata":{"id":"Sw15KnGDjssK"},"cell_type":"markdown","source":"### Baseline"},{"metadata":{"outputId":"beadc393-2b42-447e-9f86-bdb6ec31addd","id":"IPee_AjAkCZi","trusted":true,"executionInfo":{"status":"ok","timestamp":1587508135385,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":99771}},"cell_type":"code","source":"df_pred_tta_base = df_pred_tta[df_pred_tta.tta_version == 'orig'].copy()\ndf_pred_tta_base['label_pred'] = (df_pred_tta_base[range(len(CLASSES))]\n                                   .apply(lambda s: np.argmax(s.values), axis=1))\nf1_report_base = classification_report(df_pred_tta_base.label, \\\n                                       df_pred_tta_base.label_pred, \\\n                                       output_dict=True)\nprint(f1_report_base['macro avg'])","execution_count":null,"outputs":[]},{"metadata":{"id":"QcRj4YdsjHa7"},"cell_type":"markdown","source":"### Argmax of average softmax probabilities"},{"metadata":{"outputId":"93fedd45-ef4e-4748-9fb9-418ebc858c3f","id":"Od8oUIkNi3iV","trusted":true,"executionInfo":{"status":"ok","timestamp":1587508136045,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":100414}},"cell_type":"code","source":"agg_dict = {c: 'mean' for c in range(len(CLASSES))}\nagg_dict['label'] = 'max'\n\ndf_pred_tta_avg = df_pred_tta.groupby('id').agg(agg_dict)\ndf_pred_tta_avg['label_pred'] = (df_pred_tta_avg[range(len(CLASSES))]\n                                   .apply(lambda s: np.argmax(s.values), axis=1))\nf1_report_avg = classification_report(df_pred_tta_avg.label,\\\n                                      df_pred_tta_avg.label_pred, \\\n                                      output_dict=True)\nprint(f1_report_avg['macro avg'])","execution_count":null,"outputs":[]},{"metadata":{"id":"QcRj4YdsjHa7"},"cell_type":"markdown","source":"### Argmax of max softmax probabilities"},{"metadata":{"outputId":"93fedd45-ef4e-4748-9fb9-418ebc858c3f","id":"Od8oUIkNi3iV","trusted":true,"executionInfo":{"status":"ok","timestamp":1587508136045,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":100414}},"cell_type":"code","source":"agg_dict = {c: 'max' for c in range(len(CLASSES))}\nagg_dict['label'] = 'max'\n\ndf_pred_tta_max = df_pred_tta.groupby('id').agg(agg_dict)\ndf_pred_tta_max['label_pred'] = (df_pred_tta_max[range(len(CLASSES))]\n                                   .apply(lambda s: np.argmax(s.values), axis=1))\nf1_report_max = classification_report(df_pred_tta_max.label,\n                                      df_pred_tta_max.label_pred,\n                                      output_dict=True)\nprint(f1_report_max['macro avg'])","execution_count":null,"outputs":[]},{"metadata":{"id":"S9AkKIo0oQHp"},"cell_type":"markdown","source":"### Mode of argmax of softmax probabilities"},{"metadata":{"outputId":"5e351b5c-2c37-40f5-98a6-7b75461ff794","id":"fvzqIflesNrA","trusted":true,"executionInfo":{"status":"ok","timestamp":1587508150285,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":114638}},"cell_type":"code","source":"agg_dict = {'label': 'max',  'label_pred': lambda s: s.value_counts().index[0]}\n\ndf_pred_tta_mode = df_pred_tta.copy()\ndf_pred_tta_mode['label_pred'] = (df_pred_tta_mode[range(len(CLASSES))]\n                                   .apply(lambda s: np.argmax(s.values), axis=1))\n\ndf_pred_tta_mode = df_pred_tta_mode[['id', 'label', 'label_pred']]\ndf_pred_tta_mode = df_pred_tta_mode.groupby('id').agg(agg_dict)\nf1_report_mode = classification_report(df_pred_tta_mode.label, \\\n                                       df_pred_tta_mode.label_pred, \\\n                                       output_dict=True)\nprint(f1_report_mode['macro avg'])","execution_count":null,"outputs":[]},{"metadata":{"id":"is6QnpbEsOu5"},"cell_type":"markdown","source":"### Argmax of softmax of average logits"},{"metadata":{"id":"XxJgcxxNsWCq","trusted":true},"cell_type":"code","source":"with strategy.scope():\n    pred_model_logits = tf.keras.Model(model.input, \\\n                                       model.output.op.inputs[0])\nlogits = pred_model_logits.predict(ds_pred_tta_pp)","execution_count":null,"outputs":[]},{"metadata":{"id":"QkX9uOLXyLMK","trusted":true},"cell_type":"code","source":"def softmax(x):\n    e_x = np.exp(x - np.max(x))\n    return e_x / e_x.sum()","execution_count":null,"outputs":[]},{"metadata":{"id":"cNOnvupHyVNg","trusted":true},"cell_type":"code","source":"df_pred_tta_logits = pd.DataFrame(logits)\ndf_pred_tta_logits['id'] = id_list_tta\ndf_pred_tta_logits['label'] = label_list_tta","execution_count":null,"outputs":[]},{"metadata":{"outputId":"b8e5d00f-4f37-4f3a-9d48-603cb1658c81","id":"JL_MspMTwKAo","trusted":true,"executionInfo":{"status":"ok","timestamp":1587508194851,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":159172}},"cell_type":"code","source":"agg_dict = {c: 'mean' for c in range(len(CLASSES))}\nagg_dict['label'] = 'max'\n\ndf_pred_tta_logits_avg = df_pred_tta_logits.groupby('id').agg(agg_dict)\ndf_pred_tta_logits_avg['label_pred'] = (df_pred_tta_logits_avg[range(len(CLASSES))]\n                                   .apply(lambda s: softmax(s), axis=1)\n                                   .apply(lambda s: np.argmax(s.values), axis=1)) \nf1_report_logits_avg = classification_report(df_pred_tta_logits_avg.label, \\\n                                             df_pred_tta_logits_avg.label_pred, \\\n                                             output_dict=True)\nprint(f1_report_logits_avg['macro avg'])","execution_count":null,"outputs":[]},{"metadata":{"id":"CORWru2d4h4m"},"cell_type":"markdown","source":"## Summary"},{"metadata":{"id":"fwOYsfttxSA4"},"cell_type":"markdown","source":"The baseline macro f1-score s 0.921. The best score of 0.933 results from averaging softmax probs, a 16% reduction in total error (numbers may vary slightly with different runs of this book)."},{"metadata":{"outputId":"92d32bd9-1680-4f1b-da0a-fc691df21cff","id":"dmUG8alh1DWt","trusted":true,"executionInfo":{"status":"ok","timestamp":1587508194975,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":159283},"_kg_hide-input":true},"cell_type":"code","source":"procs = ['Base', 'Argmax(Mean(Softmax))', 'Argmax(Max(Softmax))', 'Mode(Argmax(Softmax))', 'Argmax(Softmax(Mean(Logits))']\ntta_results = [f1_report_base, f1_report_avg, f1_report_max, f1_report_mode, f1_report_logits_avg]\n\nf1_scores = [res['macro avg']['f1-score'] for res in tta_results]\n\nax = pd.DataFrame(zip(procs, f1_scores), columns=['procedure', 'f1-score']).set_index('procedure').plot(kind='bar', figsize=(10,5))\n\nfor i in ax.patches:\n    x = i.get_x()\n    y = i.get_height()\n    ax.text(x+.1,y+.001,f'{y:0.4f}',fontsize=12)\n\nax.set_ylabel('Macro F1-Score')\nax.set_title('Macro F1-Scores for TTA Procedures')\nax.set_ylim((.91,.95));","execution_count":null,"outputs":[]},{"metadata":{"id":"l49k0h5vmLWF"},"cell_type":"markdown","source":"## Additional Experiments"},{"metadata":{"id":"q6XiwrMWmTI_"},"cell_type":"markdown","source":"One thing I was also curious about was how the different tta versions may be contributing to overall prediction performance. The chart below shows the macro f1-score for each tta version. This is sort of interesting - it basically shows that performance declines as the scaling factor goes up and that the center crops perform better than the corner ones do."},{"metadata":{"outputId":"c26e0848-be41-4b08-98b4-9ca1d80c5a35","id":"1XEtm_jKnHE0","executionInfo":{"status":"ok","timestamp":1587508208349,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":172644},"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tta_ver_f1_scores = []\nfor t in df_pred_tta.tta_version.unique():\n    df_pred_tta_ver = df_pred_tta[df_pred_tta.tta_version == t].copy()\n    df_pred_tta_ver['label_pred'] = (df_pred_tta_ver[range(len(CLASSES))]\n                                    .apply(lambda s: np.argmax(s.values), axis=1))\n    f1_report_ver = classification_report(df_pred_tta_ver.label, \\\n                                          df_pred_tta_ver.label_pred, \\\n                                          output_dict=True)\n    f1_report_ver['macro avg']['tta_version'] = t\n    tta_ver_f1_scores.append(f1_report_ver['macro avg'])\ndf_tta_pred_ver = pd.DataFrame(tta_ver_f1_scores).set_index('tta_version')\nax = df_tta_pred_ver['f1-score'].plot(kind='bar', figsize=(15,5), title='Macro F1-Score by TTA Version')\nax.set_ylim((0.8, 0.95));","execution_count":null,"outputs":[]},{"metadata":{"id":"z1xy6mxhrMXr"},"cell_type":"markdown","source":"Let's see if performance improves under each of the following scenarios:\n* Original and flip\n* Original, flip and center crops\n* Original and flip for the first scaling level\n\nTo keep things simple I'm just going to argmax the average probabilities. Predictions for the originals and all tta versions are included for reference."},{"metadata":{"outputId":"f3171686-a4f3-4dc0-aaa5-51beea532e02","id":"UpQtfF6er6nu","executionInfo":{"status":"ok","timestamp":1587508367870,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":2680},"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"scenarios = {'orig': ['orig'],\n             'orig + flip': ['orig', 'orig - flip'],\n             'orig + flip + centers': ['orig', 'orig - flip'] + [v for v in titles if 'center' in v],\n             'orig + flip + 1st scale': ['orig', 'orig - flip'] + [v for v in titles if '1.143' in v],\n             'all': ['orig', 'orig - flip'] + titles,\n        }\n\ndef get_tta_ver_score(scenario, versions=['orig']):\n    agg_dict = {c: 'mean' for c in range(len(CLASSES))}\n    agg_dict['label'] = 'max'\n\n    df_pred_tta_ver = df_pred_tta[df_pred_tta.tta_version.isin(versions)].copy()\n    df_pred_tta_ver = df_pred_tta_ver.groupby('id').agg(agg_dict)\n    df_pred_tta_ver['label_pred'] = (df_pred_tta_ver[range(len(CLASSES))]\n                                    .apply(lambda s: np.argmax(s.values), axis=1))\n    f1_report_ver = classification_report(df_pred_tta_ver.label, \\\n                                            df_pred_tta_ver.label_pred, \\\n                                            output_dict=True)\n    f1_report_ver['macro avg']['scenario'] = scenario\n    return f1_report_ver['macro avg']\n\nax = pd.DataFrame([get_tta_ver_score(s, vers) for s, vers in scenarios.items()]).set_index('scenario')['f1-score'].plot(kind='bar', figsize=(10,5))\n\nfor i in ax.patches:\n    x = i.get_x()\n    y = i.get_height()\n    ax.text(x+.1,y+.001,f'{y:0.4f}',fontsize=12)\n\nax.set_ylabel('Macro F1-Score')\nax.set_title('Macro F1-Scores by Scenario')\nax.set_ylim((.9,.95));","execution_count":null,"outputs":[]},{"metadata":{"id":"J5MnENfQeaYA"},"cell_type":"markdown","source":"# Push Kernel"},{"metadata":{"_kg_hide-input":true,"scrolled":true,"id":"fx8d8IaAeaYA","trusted":true},"cell_type":"code","source":"if False:\n    if not KAGGLE:\n        DATASET_DIR.mkdir(exist_ok=True)\n              \n        data = {\"licenses\": [{\"name\": \"CC0-1.0\"}],\n                \"id\": f\"calebeverett/{str(DATASET_DIR)}\", \n                \"title\": str(DATASET_DIR)}\n        \n        with open(f'{DATASET_DIR}/dataset-metadata.json', 'w') as f:\n            json.dump(data, f)\n        \n        for f in ['classes.csv']:\n          bucket.blob(f).download_to_filename(f'{DATASET_DIR}/{f}')\n\n        print_output(run_command(f'kaggle datasets create -r tar -p {str(DATASET_DIR)}'))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"outputId":"1baf580c-c7e2-461c-8f95-f18d13c3b2af","id":"nrLySMd6eaYE","trusted":true,"executionInfo":{"status":"ok","timestamp":1587501806522,"user":{"userId":"10356454804103861543","displayName":"Caleb Everett","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi5mP1o_Kw8il9U4xZqVImymAm5OkgrTS6nHvCiVQ=s64"},"user_tz":420,"elapsed":2255}},"cell_type":"code","source":"if False:\n    if not KAGGLE:\n\n        run_command('cp \"drive/My Drive/Colab Notebooks/Comparison of TTA Prediction Procedures.ipynb\" .')\n\n        data = {'id': 'calebeverett/comparison-of-tta-prediction-procedures',\n                      'title': 'Comparison of TTA Prediction Procedures',\n                      'code_file': 'Comparison of TTA Prediction Procedures.ipynb',\n                      'language': 'python',\n                      'kernel_type': 'notebook',\n                      'is_private': 'true',\n                      'enable_gpu': 'false',\n                      'enable_internet': 'true',\n                      'dataset_sources': [f\"calebeverett/{str(DATASET_DIR)}\"],\n                      'competition_sources': ['flower-classification-with-tpus'],\n                     ' kernel_sources': []}\n        \n        with open('kernel-metadata.json', 'w') as f:\n            json.dump(data, f)\n\n        print_output(run_command('kaggle kernels push'))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","name":"python3"},"toc":{"nav_menu":{},"toc_window_display":false,"toc_section_display":true,"skip_h1_title":true,"base_numbering":1,"title_cell":"Table of Contents","number_sections":true,"title_sidebar":"Contents","sideBar":true,"toc_position":{},"toc_cell":false},"celltoolbar":"Edit Metadata","colab":{"machine_shape":"hm","provenance":[{"timestamp":1587412897564,"file_id":"1Os1cgaeTlgQ4YIuqXbQ4hM9sd2pnqXDE"}],"collapsed_sections":[],"name":"Comparison of TTA Prediction Procedures.ipynb","toc_visible":true},"accelerator":"TPU"},"nbformat":4,"nbformat_minor":4}