{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About this kernel, about this competition\n\n## Intro\n\nThis competition will be over in about 2 days and it has been my first Kaggle competition. I am rather a beginner in ML and I want to thank Kaggle for this great opportunity \nto learn about tensorflow, classification and tpu's.\n\nI started with a public ensemble kernel from Wojtek Rosa and tried a lot of basic hyperparameter tuning. As new discussion entries appeared I could learn a lot about augmentation techniques, under/oversampling, optimizers and other stuff. Kudos to the nice people, that fed the community\nwith their knowlege. I will mention the most important contributions in the later sections.\n\n## 0) TPU stuff\n\nTPUs are impressive and @mgoernergoogle made it easy to understand the basic code, which is necessary to start with tpus.\n\nhttps://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu\n\nLater he provided a kernel which implemented a custom training loop, that could speed up learning up to 20%. Unfortunately tensorflow 2.1 showed up to be unstable when training 512x512 sized images - this should be fixed in tf 2.2, which has been released 3 days ago. but did not find the way into the Kaggle environment as I write this.\n\nhttps://www.kaggle.com/mgornergoogle/custom-training-loop-with-100-flowers-on-tpu\n\n\n## 1) Datasets\n\nWhat could be found in the the beginning was an unbalanced dataset of flowers with 104 classes, which has been nicely assembled from 5 public flower datasets by Martin Goerner.\n\nAs the competition went on, people incorporated one or more of these public datasets in their training, published them later and it showed that using those datasets could greatly improve LB scores. Thanks to Heng CherKeng, Kirill Blinov and all the others for their contributions.\n\nhttps://www.kaggle.com/c/flower-classification-with-tpus/discussion/140866\n\nhttps://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec\n\nIn the last days there has even been a little discussion whether these datasets are allowed to be used. \n\nhttps://www.kaggle.com/c/flower-classification-with-tpus/discussion/148329\n\n\n\n## 2) Augmentations\n\nChris Deotte contributed greatly to this topic, providing notebooks that showed us an implementation Gridmask, CutMix and MixUp augmentations along with his spatial affine transformations. I tried them all and found it very interesting and also introduced me to learn about label smoothing (another technique to handle unbalanced datasets with one-hot encoded class labels).\n\nhttps://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n\nhttps://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu\n\nhttps://www.kaggle.com/yihdarshieh/make-chris-deotte-s-data-augmentation-faster\n\nhttps://www.kaggle.com/yihdarshieh/batch-implementation-of-more-data-augmentations\n\nhttps://www.kaggle.com/xiejialun/gridmask-data-augmentation-with-tensorflow\n\n\nThanks to Michał Szachniewicz who implemented AugMix to run under tensorflow 2.x. A pitty that the experiments did not produce nice results.\n\nhttps://www.kaggle.com/szacho/augmix-data-augmentation-on-tpu\n\nI found it interesting, how good the rather simple cutout augmentation worked - see the random_blockout() function from a competitor below.\n\n\nI also stumbled about AutoAug, a technique used by Google researchers and in AutoML for classification and now even for object detection, to find the best fitting augmentation parameters for a given dataset. AutoAug can be found in the tensorflow repository on github, but is implemented in tensorflow 1.x and I did not have the time to invest in that.\n\n\n## 3) Models and Techniques\n\n### 3.1 Models\n\nState of the art is the usage of the Effcientnet set of models. Theses models are trained on imagenet and noisy-student - both variations of the weights are available in the Keras version on github. Some people combined one or two Efficientnet model with other models in an ensemble, like it is done in this kernel.\n\nWojtek Rosa provided this starter kernel: https://www.kaggle.com/wrrosa/tpu-enet-b7-densenet\n\n\n### 3.2 Optimizers\n\nThere is a lot of research going on in this field and computer scientists are proposing a lot of new optimizers these days.\nI started with Adam and did some experiments, especially with the so called Ranger optimizer (a combination of RectifiedAdam and Lookahead - they can be found in the tensorflow addons library).\n\nAll in all I did not find success using these, maybe because they converge slower and there is limited training time in this competition -  so I went back to plain Adam.\n\n\n### 3.3 Learning Rate and other parameters\n\nWhen one is finetuning a model (train all weights of a pretrained model) one should implement a rampup phase for some epochs with a lower learning rate, so one doesn't break the pretrained features. The starter notebook provides a LearningRateScheduler with exponential decay.\n\nAn alternative would be the usage of a cosine decaying learning rate, which I implemented below in this notebook. I did not try cyclic learning rates, which would have been interesting as well. Btw. - the ranger optimizer likes high flat learning rates in the beginning and cosine annealing.\n\n> On TPU the initial batch size could be doubled with 512x512 sized images, which really was a big improvement (16 * strategy.num_replicas_in_sync * 2)\n\n> I could get a bit better results multiplying the proposed learning rate schedule from the starter kernel by 1.2\n\n\n\n\n### 3.4 Class Weights\n\nClass weights are a method where one can tell the optimizer to underweight the influence of overrepresented classes. A short piece of code in shown below, but I did not use it at last, because it showed, that the losses are getting smaller more slowly. Maybe more epochs would show that this method leads to a good model, but in this competition we are restricted to a runtime of 3 hours and this is not effective. Further there is doubt whether class weights do work at all in tf2.1 on tpu.\n\n\n### 3.5 Oversampling/Undersampling\n\nFor an unbalanced dataset people have found success in training with data, where one filters out examples of the overpresented classes or one extends the dataset with (modified) copies of the underrepresented examples. I did not get lucky with it in this competition.\n\nhttps://www.kaggle.com/yihdarshieh/tutorial-oversample\n\n\n### 3.6 Progressive Resizing\n\nDuring the competition I read about progressive resizing (to train a model with a smaller image size first and then again with a larger image size) but then a notebook which implemented this using the fastai library, brought me back to this idea in the last days, so I implemented it in this kernel.\n\nhttps://www.kaggle.com/kurianbenoy/classifying-flowers-with-fastaiv2-0-96\n\n\n### 3.7 Custom Training Loop\n\nAs mentioned above the custom training loop from https://www.kaggle.com/mgornergoogle/custom-training-loop-with-100-flowers-on-tpu can save about 20% training time.\n\n\n### 3.8 KFolds\n\nUsing KFolds is the idea of putting together training and validation data in one set and then splitting this set differently K-times, train K models and then aggregate the predictions of these K models.\n\nhttps://www.kaggle.com/ragnar123/4-kfold-densenet201\n\n\n### 3.9 TTA\n\nTTA (test time augmentations) is the idea to augment the test data several times and aggregate the predictions on these data. I did not find success with this, but many successful competitors use it. There is a nice notebook from Caleb about this technique:\n\nhttps://www.kaggle.com/calebeverett/comparison-of-tta-prediction-procedures\n\n\n### 3.10 Pseudo labeling test data\n\nI did not try, but well doing competitors probably do.\n\nSome time ago Chris Deotte provided a nice summary how it is done:\n\nhttps://www.kaggle.com/cdeotte/pseudo-labeling-qda-0-969\n\n\n### Using Mish activations\n\nEfficientnet uses the relatively new swish activation function. In the last days I came across an alternative which seems to do better - mish()\n\nAs exchanging activations functions in Keras on the fly seems to be difficult, I am not sure if I can try something in that direction in the next days.\n\n\n## Final words\n\nMy best result so far (before the last weekend) has been training a 4 model ensemble for 13 epochs with the 512x512 size images with over 40000 images and no TTA (LB score 0.975) and I am rather sure that training for more epochs and using more training images would improve the score a lot. Probably one has to use 224x224 images,\n\nFor the next weekend I tend to try smaller images sizes with more epochs and images for fun, because the usage of the external datasets is probably not allowed for the final LB run.\nMaybe some TTA experiments, if time allows.\n\n**Update**\n\nI am sorry, I cannot remember whos kernel I copied - but the 4 model one is working great - thx.\nTraining the 4 model ensemble with all external training data pushed me to a LB score of 0.9837 with a training time of 2h20 - this should be a good base for further experiments :)\n\n**Update**\n\nGot an 0.984+ score with training 224x224 images.\n\n**Update**\n\nCould not test mish() - but training time seems to be 10% slower on EN models with my simple patch.\n\nCould not test TTA - I think I simply did it wrong. No way to climb the LB score without these :)\n\nTried to run 48 epochs with a custom training loop, but it seems to use more then 3 hours.... what a pitty :)\n\nI really like to try these improvements next week...\n\n\nThis notebook will serve as a summary of some of the knowledge I built up and should be able to reach a LB score of 0.967 with the base dataset in one way or another.\nOne should try different models and run it with validation calculations to find the best alpha for ensembling and then submit it training on both (train and val) datasets.\n\nThanks for the fish :)   (to everybody, who doesn't know this quote, pls google Douglas Adams)\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import math, re, os, time\nimport tensorflow as tf, tensorflow.keras.backend as K\nimport numpy as np\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nimport efficientnet.tfkeras as efn\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\nfrom tensorflow.keras.applications import DenseNet201\nfrom collections import namedtuple\n#from sklearn.model_selection import KFold\nprint(\"Tensorflow version \" + tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Configurations","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n\n# Detect hardware, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\nif tpu:\n    BATCH_SIZE_MULT = 2\nelse:\n    BATCH_SIZE_MULT = 1\n\n# Data access\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('flower-classification')\nprint(GCS_DS_PATH)\n\n# Configuration\n#IMAGE_SIZE = [512, 512]\nIMAGE_SIZE = [331, 331]\n#IMAGE_SIZE = [192, 192]\n\nimg_size=IMAGE_SIZE[0]\n\n# pls check all EPOCHS value changes, if you want to train for higher scores\nEPOCHS = 3 #14\n\n# bigger batch size is really useful\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync * BATCH_SIZE_MULT # doubling the batch_size rocks\n\n# flag for TTA predictions\nDO_TTA = False\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not tpu:\n    MIXED_PRECISION = False\n    XLA_ACCELERATE = False\nelse:\n    MIXED_PRECISION = True\n    XLA_ACCELERATE = True\n\nif MIXED_PRECISION:\n    from tensorflow.keras.mixed_precision import experimental as mixed_precision\n    if tpu: policy = tf.keras.mixed_precision.experimental.Policy('mixed_bfloat16')\n    else: policy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\n    mixed_precision.set_policy(policy)\n    print('Mixed precision enabled')\n\nif XLA_ACCELERATE: \n    if not tpu: # I cannot remember why, probably got a problem without this\n        tf.config.optimizer.set_jit(True)\n        print('Accelerated Linear Algebra enabled')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Custom LR schedule","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Standard learning rate function","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"LR_START = 0.00001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001 #0.00001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = .8\n\n#pushing this value often helps a bit (eg 1.2)\nLR_MULTIPLIER = 1.0\n\n@tf.function\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr * LR_MULTIPLIER\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=True)\n\nrng = [i for i in range(EPOCHS)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Learning rate function for a follow up training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# this is very basic and not optimized\nLR_START2 = 0.00001\nLR_MAX2 = 0.000025 * strategy.num_replicas_in_sync\nLR_MIN2 = 0.00001 #0.00001\nLR_RAMPUP_EPOCHS2 = 5\nLR_SUSTAIN_EPOCHS2 = 0\nLR_EXP_DECAY2 = .8\n\nLR_MULTIPLIER = 1.0\n\n@tf.function\ndef lrfn2(epoch):\n    if epoch < LR_RAMPUP_EPOCHS2:\n        lr = (LR_MAX2 - LR_START2) / LR_RAMPUP_EPOCHS2 * epoch + LR_START2\n    elif epoch < LR_RAMPUP_EPOCHS2 + LR_SUSTAIN_EPOCHS2:\n        lr = LR_MAX2\n    else:\n        lr = (LR_MAX2 - LR_MIN2) * LR_EXP_DECAY2**(epoch - LR_RAMPUP_EPOCHS2 - LR_SUSTAIN_EPOCHS2) + LR_MIN2\n    return lr\n    \nlr_callback2 = tf.keras.callbacks.LearningRateScheduler(lrfn2, verbose=True)\n\nrng = [i for i in range(EPOCHS)]\ny = [lrfn2(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Learning rate schedule with cosine anneal","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"@tf.function\ndef lrfnCosineDecay(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    # cosine anneal\n    else:\n        lr = LR_MIN + (LR_MAX - LR_MIN) * (1 + math.cos(math.pi * epoch / EPOCHS)) / 2\n    return lr * LR_MULTIPLIER\n\nrng = [i for i in range(EPOCHS)]\ny = [lrfnCosineDecay(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))\n\nlr_callbackCosineDecay = tf.keras.callbacks.LearningRateScheduler(lrfnCosineDecay, verbose=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_PATH_SELECT = { # available image sizes\n    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'\n}\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') # predictions on this dataset should be submitted for the competition\n\n# watch out for overfitting!\nSKIP_VALIDATION = False\nif SKIP_VALIDATION:\n    TRAINING_FILENAMES = TRAINING_FILENAMES + VALIDATION_FILENAMES\n\n# in the beginning I tried to remove suspicious samples - a relict\n#VALIDATION_MISMATCHES_IDS = ['55a883e16','f4ec48685','2023d3cac','f8eab6777','741999f79','861282b96','28594d9ce','bab3ef1f5','617a30d60','4571b9509','6a3a28a06','9b8f2f5bd','293c37e25','7472eb523','0bf0b39b3','c846d8649','9ee42218f','f4ec48685']\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Unhide to see `CLASSES`:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"CLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 102","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Helper Functions\n## Visualization","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"# numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch, predictions=None, figsize  = 13.0):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE =  figsize\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n\ndef display_confusion_matrix(cmat, score, precision, recall):\n    plt.figure(figsize=(15,15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    plt.show()\n    \ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Datasets Functions","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n\ndef read_labeled_id_tfrecord(example):\n    LABELED_ID_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_ID_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    idnum =  example['id']\n    return image, label, idnum # returns a dataset of (image, label, idnum) triples\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_id_tfrecord  if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n        \n    return dataset\n\ndef load_dataset_with_id(filenames, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_id_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset\n\ndef data_augment(image, label):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement in the next function (below),\n    # this happens essentially for free on TPU. Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n    image = tf.image.random_flip_left_right(image)\n    #image = tf.image.random_brightness(image, 0.05, seed=None)\n    #image = tf.image.random_contrast(image, 0.8, 1.2, seed=None)\n    \n    #random cut\n    image= random_blockout(image)\n    \n    return image, label \n\n\ndef get_training_dataset(do_aug=True,do_repeat=True):\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    \n    #dataset = dataset.filter(lambda image, label, idnum: tf.reduce_sum(tf.cast(idnum == VALIDATION_MISMATCHES_IDS, tf.int32))==0)\n    dataset = dataset.map(lambda image, label, idnum: [image, label])\n    \n    # add an additional argument to this function with False default and call it with True if you want to try undersampling\n    #if do_undersample:\n    #    dataset = dataset.filter(undersample_filter)\n    \n    if do_repeat:\n        dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    \n    if do_aug:\n        dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n        #dataset = dataset.map(cropandresize, num_parallel_calls=AUTO)\n        \n    if do_repeat:\n        dataset = dataset.shuffle(2048)\n        dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)\n        dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\n\ndef get_validation_dataset(ordered=False, repeated=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    \n    #dataset = dataset.filter(lambda image, label, idnum: tf.reduce_sum(tf.cast(idnum == VALIDATION_MISMATCHES_IDS, tf.int32))==0)\n    dataset = dataset.map(lambda image, label, idnum: [image, label])\n    \n    if repeated:\n        dataset = dataset.repeat()\n        dataset = dataset.shuffle(2048)\n        \n    dataset = dataset.batch(BATCH_SIZE, drop_remainder=repeated)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset_with_id(ordered=False):\n    dataset = load_dataset_with_id(VALIDATION_FILENAMES, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_test_dataset(ordered=False, do_aug=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    \n    if do_aug:\n        dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n        dataset = dataset.map(transform, num_parallel_calls=AUTO)\n        \n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef count_data_items(filenames):\n    #dataset = load_dataset(filenames,labeled = False)\n    #dataset = dataset.map(lambda image, idnum: idnum)\n    #dataset = dataset.filter(lambda idnum: tf.reduce_sum(tf.cast(idnum == VALIDATION_MISMATCHES_IDS, tf.int32))==0)\n    #uids = next(iter(dataset.batch(26000))).numpy().astype('U') \n    #return len(np.unique(uids))    \n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\ndef int_div_round_up(a, b):\n    return (a + b - 1) // b\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = (1 - SKIP_VALIDATION) * count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nVALIDATION_STEPS = int_div_round_up(NUM_VALIDATION_IMAGES, BATCH_SIZE)\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Augmentations","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 1) Chris Deottes affine transforms","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n    # returns 3x3 transformmatrix which transforms indicies\n        \n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    shear = math.pi * shear / 180.\n    \n    # ROTATION MATRIX\n    c1 = tf.math.cos(rotation)\n    s1 = tf.math.sin(rotation)\n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    rotation_matrix = tf.reshape( tf.concat([c1,s1,zero, -s1,c1,zero, zero,zero,one],axis=0),[3,3] )\n        \n    # SHEAR MATRIX\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)\n    shear_matrix = tf.reshape( tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3] )    \n    \n    # ZOOM MATRIX\n    zoom_matrix = tf.reshape( tf.concat([one/height_zoom,zero,zero, zero,one/width_zoom,zero, zero,zero,one],axis=0),[3,3] )\n    \n    # SHIFT MATRIX\n    shift_matrix = tf.reshape( tf.concat([one,zero,height_shift, zero,one,width_shift, zero,zero,one],axis=0),[3,3] )\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), K.dot(zoom_matrix, shift_matrix))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def transform(image,label):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated, sheared, zoomed, and shifted\n    \n    DIM = IMAGE_SIZE[0]\n    XDIM = DIM%2 #fix for size 331\n    \n    rot = 15. * tf.random.normal([1],dtype='float32')\n    shr = 5. * tf.random.normal([1],dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1],dtype='float32')/10.\n    w_zoom = 1.0 + tf.random.normal([1],dtype='float32')/10.\n    #w_zoom = h_zoom\n    h_shift = 16. * tf.random.normal([1],dtype='float32') \n    w_shift = 16. * tf.random.normal([1],dtype='float32') \n  \n    # GET TRANSFORMATION MATRIX\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    # LIST DESTINATION PIXEL INDICES\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(m,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES           \n    idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image,tf.transpose(idx3))\n    #print('transform')\n        \n\n    return tf.reshape(d,[DIM,DIM,3]),label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Unused augmentation (maybe for TTA)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#cropandresize\ndef cropandresize(image,label):\n\n    #box[0] = [0,0,int(IMAGE_SIZE[0]/2),int(IMAGE_SIZE[1]/2)]\n    #box[1] = [int(IMAGE_SIZE[0]/2),0,int(IMAGE_SIZE[0]/2),int(IMAGE_SIZE[1]/2)]\n    #box[2] = [0,int(IMAGE_SIZE[0]/2),int(IMAGE_SIZE[0]/2),int(IMAGE_SIZE[1]/2)]\n    #box[3] = [int(IMAGE_SIZE[0]/2),int(IMAGE_SIZE[0]/2),int(IMAGE_SIZE[0]/2),int(IMAGE_SIZE[1]/2)]\n    #box[4] = [int(IMAGE_SIZE[0]/4),int(IMAGE_SIZE[0]/4),int(IMAGE_SIZE[0]/2),int(IMAGE_SIZE[1]/2)]\n    \n    rnd = tf.random.uniform(shape=[], minval=0, maxval=7, dtype=tf.int64) \n    \n    if rnd == 0:\n        image = tf.image.crop_to_bounding_box(image, 0, 0, int(2*IMAGE_SIZE[0]/3),int(2*IMAGE_SIZE[0]/3))\n    elif rnd == 1:\n        image = tf.image.crop_to_bounding_box(image, int(IMAGE_SIZE[0]/3), 0, int(2*IMAGE_SIZE[0]/3),int(2*IMAGE_SIZE[0]/3))\n    elif rnd == 2:\n        image = tf.image.crop_to_bounding_box(image, 0, int(IMAGE_SIZE[0]/3), int(2*IMAGE_SIZE[0]/3),int(2*IMAGE_SIZE[0]/3))\n    elif rnd == 3:\n        image = tf.image.crop_to_bounding_box(image, int(IMAGE_SIZE[0]/3), int(IMAGE_SIZE[0]/3), int(2*IMAGE_SIZE[0]/3),int(2*IMAGE_SIZE[0]/3))\n    elif rnd == 4:\n        image = tf.image.crop_to_bounding_box(image, int(IMAGE_SIZE[0]/6), int(IMAGE_SIZE[0]/6), int(2*IMAGE_SIZE[0]/3),int(2*IMAGE_SIZE[0]/3))\n    else:\n        image =  image\n    \n    #image = tf.image.resize(image, size=[IMAGE_SIZE[0],IMAGE_SIZE[0]])\n    return tf.image.resize(image, size=[IMAGE_SIZE[0],IMAGE_SIZE[0]]),label     #tf.reshape(image,[IMAGE_SIZE[0],IMAGE_SIZE[0],3]),label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Used augmentation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#simple and effective\n\ndef random_blockout(img, sl=0.1, sh=0.2, rl=0.4):\n\n    h, w, c = img_size, img_size, 3\n    origin_area = tf.cast(h*w, tf.float32)\n\n    e_size_l = tf.cast(tf.round(tf.sqrt(origin_area * sl * rl)), tf.int32)\n    e_size_h = tf.cast(tf.round(tf.sqrt(origin_area * sh / rl)), tf.int32)\n\n    e_height_h = tf.minimum(e_size_h, h)\n    e_width_h = tf.minimum(e_size_h, w)\n\n    erase_height = tf.random.uniform(shape=[], minval=e_size_l, maxval=e_height_h, dtype=tf.int32)\n    erase_width = tf.random.uniform(shape=[], minval=e_size_l, maxval=e_width_h, dtype=tf.int32)\n\n    erase_area = tf.zeros(shape=[erase_height, erase_width, c])\n    erase_area = tf.cast(erase_area, tf.uint8)\n\n    pad_h = h - erase_height\n    pad_top = tf.random.uniform(shape=[], minval=0, maxval=pad_h, dtype=tf.int32)\n    pad_bottom = pad_h - pad_top\n\n    pad_w = w - erase_width\n    pad_left = tf.random.uniform(shape=[], minval=0, maxval=pad_w, dtype=tf.int32)\n    pad_right = pad_w - pad_left\n\n    erase_mask = tf.pad([erase_area], [[0,0],[pad_top, pad_bottom], [pad_left, pad_right], [0,0]], constant_values=1)\n    erase_mask = tf.squeeze(erase_mask, axis=0)\n    erased_img = tf.multiply(tf.cast(img,tf.float32), tf.cast(erase_mask, tf.float32))\n\n    return tf.cast(erased_img, img.dtype)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### Peek at training data\n\ntraining_dataset = get_training_dataset(do_aug=False)\ntraining_dataset = training_dataset.unbatch().batch(20)\ntrain_batch = iter(training_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### Run this cell again for next set of images\ndisplay_batch_of_images(next(train_batch))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Class weights\n\nHow to calc them:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import datetime\nimport tqdm\nimport json\nfrom collections import Counter\nimport gc\ngc.enable()\n\ndef get_training_dataset_raw():\n\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True, ordered=False)\n    return dataset\n\n\nraw_training_dataset = get_training_dataset_raw()\n\nlabel_counter = Counter()\nfor images, labels, id in raw_training_dataset:\n    label_counter.update([labels.numpy()])\n\ndel raw_training_dataset    \n\nTARGET_NUM_PER_CLASS = 122\n\ndef get_weight_for_class(class_id):\n    counting = label_counter[class_id]\n    \n    weight = TARGET_NUM_PER_CLASS / counting\n    \n    return weight\n\n# one version for tpu one for gpu - tf2.1 bug\nweight_per_class = {class_id: get_weight_for_class(class_id) for class_id in range(104)}\nweight_per_class_wo_id = {get_weight_for_class(class_id) for class_id in range(104)}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(weight_per_class)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Undersampling","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# not used here - see get_taining_data()\nUNDERSAMPLE_CLASS_IDS0 = [103]           # 30%\nUNDERSAMPLE_CLASS_IDS1 = [67, 4]         # 35%\nUNDERSAMPLE_CLASS_IDS2 = [49,13,0]         # 50%\nUNDERSAMPLE_CLASS_IDS3 = [53,48,73,47,102] # 50%\n\ndef undersample_filter(image, label):\n    rnd = tf.random.uniform([1], minval=0, maxval=1, dtype='float32', seed=0)\n    \n    res = tf.math.equal(tf.reduce_sum(tf.cast(label == UNDERSAMPLE_CLASS_IDS0, tf.int32)), 0)\n    if not res and rnd >= 0.7:\n        return False\n    \n    res = tf.math.equal(tf.reduce_sum(tf.cast(label == UNDERSAMPLE_CLASS_IDS1, tf.int32)), 0)\n    if not res and rnd >= 0.65:\n        return False\n    \n    res = tf.math.equal(tf.reduce_sum(tf.cast(label == UNDERSAMPLE_CLASS_IDS2, tf.int32)), 0)\n    if not res and rnd >= 0.5:\n        return False\n    \n    res = tf.math.equal(tf.reduce_sum(tf.cast(label == UNDERSAMPLE_CLASS_IDS3, tf.int32)), 0)\n    if not res and rnd >= 0.4:\n        return False\n\n    return True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training Model","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Load Model into TPU","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Need this line so Google will recite some incantations\n# for Turing to magically load the model onto the TPU\nwith strategy.scope():\n    enet = efn.EfficientNetB7(\n        input_shape=(None,None,3), #setting the shape to None allows multiple input sizes\n        weights='noisy-student',\n        include_top=False\n    )\n\n    model = tf.keras.Sequential([\n        enet,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        #tf.keras.layers.Dense(2000, activation='relu'), # extra layer or dropout did not improve the model \n        tf.keras.layers.Dense(len(CLASSES), activation='softmax',dtype='float32')\n    ])\n        \nmodel.compile(\n    optimizer=tf.keras.optimizers.Adam(lr=0.0001),\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy']\n)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"** additional callbacks could be added, plus the way to add class_weights to training **","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#scheduler = tf.keras.callbacks.ReduceLROnPlateau(patience=3, verbose=1)\n#earlystopping = tf.keras.callbacks.EarlyStopping(monitor='val_sparse_categorical_accuracy', min_delta=0.001, patience=4, verbose=1, mode='auto', baseline=None, restore_best_weights=True)\n\nhistory = model.fit(\n    get_training_dataset(), \n    steps_per_epoch=STEPS_PER_EPOCH,\n    epochs=EPOCHS,\n    callbacks=[lr_callback], #earlystopping],\n    validation_data=None if SKIP_VALIDATION else get_validation_dataset()\n    #class_weight=weight_per_class\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# save the weights and restart training with different image size and learning rate for progressive resizing\n#model.save_weights('11.h5')\n\nIMAGE_SIZE = [512, 512] #changing input image sizes on the fly\nimg_size=IMAGE_SIZE[0]\nEPOCHS = 3 # 18\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(\n    get_training_dataset(), \n    steps_per_epoch=STEPS_PER_EPOCH,\n    epochs=EPOCHS,\n    callbacks=[lr_callback2], #earlystopping],\n    validation_data=None if SKIP_VALIDATION else get_validation_dataset()\n    #class_weight=weight_per_class\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not SKIP_VALIDATION:\n    display_training_curves(history.history['loss'], history.history['val_loss'], 'loss', 211)\n    display_training_curves(history.history['sparse_categorical_accuracy'], history.history['val_sparse_categorical_accuracy'], 'accuracy', 212)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#when we run a prediction on valid now, we can del the model to save memory and use the prediction later\n#del model\n#gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training model 2 - using the custom training loop:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_SIZE = [331, 331]\nimg_size=IMAGE_SIZE[0]\nEPOCHS = 3 #40\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### this would be the standard way\nwith strategy.scope():\n    rnet = efn.EfficientNetB6(\n        input_shape=(None,None,3), #(IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n        #weights='imagenet',\n        weights='noisy-student',\n        include_top=False\n    )\n\n    model2 = tf.keras.Sequential([\n        rnet,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax',dtype='float32')\n    ])\n        \nmodel2.compile(\n    optimizer=tf.keras.optimizers.Adam(lr=0.0001),\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy']\n)\nmodel2.summary()","execution_count":null},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### this would be the standard way - part2\nhistory2 = model2.fit(\n    get_training_dataset(), \n    steps_per_epoch=STEPS_PER_EPOCH,\n    epochs=EPOCHS, \n    callbacks=[lr_callback],\n    validation_data=None if SKIP_VALIDATION else get_validation_dataset()\n    #class_weight=weight_per_class_wo_id\n)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Martin Goerners custom training loop code","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    pretrained_model = efn.EfficientNetB5(weights='noisy-student', include_top=False ,input_shape=(None,None,3)) #[*IMAGE_SIZE, 3])\n    #pretrained_model.trainable = True # False = transfer learning, True = fine-tuning\n    \n    model2 = tf.keras.Sequential([\n        pretrained_model,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax', dtype='float32') # setting dtype='float32' is necessary for mixed precision usage\n    ])\n    model2.summary()\n    \n    # Instiate optimizer with learning rate schedule\n    class LRSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n        def __call__(self, step):\n            return lrfn(epoch=step//STEPS_PER_EPOCH)\n    optimizer = tf.keras.optimizers.Adam(learning_rate=LRSchedule())\n        \n    # this also works but is not very readable\n    #optimizer = tf.keras.optimizers.Adam(learning_rate=lambda: lrfn(tf.cast(optimizer.iterations, tf.float32)//STEPS_PER_EPOCH))\n    \n    # Instantiate metrics\n    train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy()\n    valid_accuracy = tf.keras.metrics.SparseCategoricalAccuracy()\n    train_loss = tf.keras.metrics.Sum()\n    valid_loss = tf.keras.metrics.Sum()\n    \n    # Loss\n    # The recommendation from the Tensorflow custom training loop  documentation is:\n    # loss_fn = lambda a,b: tf.nn.compute_average_loss(tf.keras.losses.sparse_categorical_crossentropy(a,b), global_batch_size=BATCH_SIZE)\n    # https://www.tensorflow.org/tutorials/distribute/custom_training#define_the_loss_function\n    # This works too and shifts all the averaging to the training loop which is easier:\n    loss_fn = tf.keras.losses.sparse_categorical_crossentropy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"STEPS_PER_TPU_CALL = int(NUM_TRAINING_IMAGES/BATCH_SIZE) #99 original\nVALIDATION_STEPS_PER_TPU_CALL = 29  # random?!\n\n@tf.function\ndef train_step(data_iter):\n    def train_step_fn(images, labels):\n        with tf.GradientTape() as tape:\n            probabilities = model2(images, training=True)\n            loss = loss_fn(labels, probabilities)\n        grads = tape.gradient(loss, model2.trainable_variables)\n        optimizer.apply_gradients(zip(grads, model2.trainable_variables))\n        \n        #update metrics\n        train_accuracy.update_state(labels, probabilities)\n        train_loss.update_state(loss)\n        \n    # this loop runs on the TPU\n    for _ in tf.range(STEPS_PER_TPU_CALL):\n        strategy.experimental_run_v2(train_step_fn, next(data_iter))\n\n@tf.function\ndef valid_step(data_iter):\n    def valid_step_fn(images, labels):\n        probabilities = model2(images, training=False)\n        loss = loss_fn(labels, probabilities)\n        \n        # update metrics\n        valid_accuracy.update_state(labels, probabilities)\n        valid_loss.update_state(loss)\n        \n    # this loop runs on the TPU\n    for _ in tf.range(VALIDATION_STEPS_PER_TPU_CALL):\n        strategy.experimental_run_v2(valid_step_fn, next(data_iter))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"start_time = epoch_start_time = time.time()\n\n# distribute the datset according to the strategy\ntrain_dist_ds = strategy.experimental_distribute_dataset(get_training_dataset())\n# Hitting End Of Dataset exceptions is a problem in this setup. Using a repeated validation set instead.\n# This will introduce a slight inaccuracy because the validation dataset now has some repeated elements.\nvalid_dist_ds = strategy.experimental_distribute_dataset(get_validation_dataset(repeated=True))\n\nprint(\"Training steps per epoch:\", STEPS_PER_EPOCH, \"in increments of\", STEPS_PER_TPU_CALL)\nprint(\"Validation images:\", NUM_VALIDATION_IMAGES,\n      \"Batch size:\", BATCH_SIZE,\n      \"Validation steps:\", NUM_VALIDATION_IMAGES//BATCH_SIZE, \"in increments of\", VALIDATION_STEPS_PER_TPU_CALL)\nprint(\"Repeated validation images:\", int_div_round_up(NUM_VALIDATION_IMAGES, BATCH_SIZE*VALIDATION_STEPS_PER_TPU_CALL)*VALIDATION_STEPS_PER_TPU_CALL*BATCH_SIZE-NUM_VALIDATION_IMAGES)\nHistory = namedtuple('History', 'history')\nhistory = History(history={'loss': [], 'val_loss': [], 'sparse_categorical_accuracy': [], 'val_sparse_categorical_accuracy': []})\n\nepoch = 0\ntrain_data_iter = iter(train_dist_ds) # the training data iterator is repeated and it is not reset\n                                      # for each validation run (same as model.fit)\nvalid_data_iter = iter(valid_dist_ds) # the validation data iterator is repeated and it is not reset\n                                      # for each validation run (different from model.fit whre the\n                                      # recommendation is to use a non-repeating validation dataset)\n\nstep = 0\nepoch_steps = 0\nwhile True:\n    \n    # run training step\n    train_step(train_data_iter)\n    epoch_steps += STEPS_PER_TPU_CALL\n    step += STEPS_PER_TPU_CALL\n    print('=', end='', flush=True)\n\n    # validation run at the end of each epoch\n    if (step // STEPS_PER_EPOCH) > epoch:\n        print('|', end='', flush=True)\n        \n        # validation run\n        valid_epoch_steps = 0\n        for _ in range(int_div_round_up(NUM_VALIDATION_IMAGES, BATCH_SIZE*VALIDATION_STEPS_PER_TPU_CALL)):\n            valid_step(valid_data_iter)\n            valid_epoch_steps += VALIDATION_STEPS_PER_TPU_CALL\n            print('=', end='', flush=True)\n\n        # compute metrics\n        history.history['sparse_categorical_accuracy'].append(train_accuracy.result().numpy())\n        history.history['val_sparse_categorical_accuracy'].append(valid_accuracy.result().numpy())\n        history.history['loss'].append(train_loss.result().numpy() / (BATCH_SIZE*epoch_steps))\n        history.history['val_loss'].append(valid_loss.result().numpy() / (BATCH_SIZE*valid_epoch_steps))\n        \n        # report metrics\n        epoch_time = time.time() - epoch_start_time\n        print('\\nEPOCH {:d}/{:d}'.format(epoch+1, EPOCHS))\n        print('time: {:0.1f}s'.format(epoch_time),\n              'loss: {:0.4f}'.format(history.history['loss'][-1]),\n              'accuracy: {:0.4f}'.format(history.history['sparse_categorical_accuracy'][-1]),\n              'val_loss: {:0.4f}'.format(history.history['val_loss'][-1]),\n              'val_acc: {:0.4f}'.format(history.history['val_sparse_categorical_accuracy'][-1]),\n              'lr: {:0.4g}'.format(lrfn(epoch)),\n              'steps/val_steps: {:d}/{:d}'.format(epoch_steps, valid_epoch_steps), flush=True)\n        \n        # set up next epoch\n        epoch = step // STEPS_PER_EPOCH\n        epoch_steps = 0\n        epoch_start_time = time.time()\n        train_accuracy.reset_states()\n        valid_accuracy.reset_states()\n        valid_loss.reset_states()\n        train_loss.reset_states()\n        if epoch >= EPOCHS:\n            break\n\noptimized_ctl_training_time = time.time() - start_time\nprint(\"OPTIMIZED CTL TRAINING TIME: {:0.1f}s\".format(optimized_ctl_training_time))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### history2 not defined here\nif not SKIP_VALIDATION:\n    display_training_curves(history2.history['loss'], history2.history['val_loss'], 'loss', 211)\n    display_training_curves(history2.history['sparse_categorical_accuracy'], history2.history['val_sparse_categorical_accuracy'], 'accuracy', 212)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**B6 with 512x512 gets oom**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Finding best alpha\nOur final model is just mix of two presented above. In the first commit it was arithmetic mean (alpha = 0.5). Note that using validation data as training will fit your model with accuracy equal 1.0.\nThus formula presented below of linear combination of models will work only with validation data:\n\nprob = alpha  prob(model) + (1 - alpha)  prob(model2)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_SIZE = [512, 512] \nimg_size=IMAGE_SIZE[0] \nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec') \nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not SKIP_VALIDATION:\n    cmdataset = get_validation_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and labels, order matters.\n    images_ds = cmdataset.map(lambda image, label: image)\n    labels_ds = cmdataset.map(lambda image, label: label).unbatch()\n    cm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch\n    m = model.predict(images_ds)    \n    m2 = model2.predict(images_ds)\n    scores = []\n    for alpha in np.linspace(0,1,100):\n        cm_probabilities = alpha*m+(1-alpha)*m2\n        cm_predictions = np.argmax(cm_probabilities, axis=-1)\n        scores.append(f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro'))\n        \n    print(\"Correct   labels: \", cm_correct_labels.shape, cm_correct_labels)\n    print(\"Predicted labels: \", cm_predictions.shape, cm_predictions)\n    plt.plot(scores)\n    best_alpha = np.argmax(scores)/100\n    cm_probabilities = best_alpha*m+(1-best_alpha)*m2\n    cm_predictions = np.argmax(cm_probabilities, axis=-1)\nelse:\n    best_alpha = 0.5 #0.44","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(best_alpha)\n\n# there should be code in the competition discussion or a sample model which shows how to implement this for 3 or more models\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Mismatches on a validation data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#best_alpha=0.60","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not SKIP_VALIDATION:\n    \n    cmdataset_with_id = get_validation_dataset_with_id(ordered=True)\n    ids_ds = cmdataset_with_id.map(lambda image, label, idnum: idnum).unbatch()\n    ids = next(iter(ids_ds.batch(NUM_VALIDATION_IMAGES))).numpy().astype('U') # get everything as one batch\n\n    val_batch = iter(cmdataset.unbatch().batch(1))\n    noip = sum(cm_predictions!=cm_correct_labels)\n    print('Number of incorrect predictions: ' + str(noip) + ' ('+str(round(noip/NUM_VALIDATION_IMAGES*100,1))+'%)')\n    for fi in range(NUM_VALIDATION_IMAGES):\n        x = next(val_batch)\n        if cm_predictions[fi] != cm_correct_labels[fi]:\n            print(\"Image id: '\" + ids[fi] + \"'\")\n            display_batch_of_images(x,np.array([cm_predictions[fi]]),figsize = 4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Confusion matrix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# better colors for the cmat would be nice \nif not SKIP_VALIDATION:\n    cmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))\n    score = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\n    precision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\n    recall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\n    #cmat = (cmat.T / cmat.sum(axis=1)).T # normalized\n    display_confusion_matrix(cmat, score, precision, recall)\n    print('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predictions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if not DO_TTA:\n    test_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n\n    print('Computing predictions...')\n\n    test_images_ds = test_ds.map(lambda image, idnum: image)\n    probabilities = best_alpha*model.predict(test_images_ds) + (1-best_alpha)*model2.predict(test_images_ds)\n    predictions = np.argmax(probabilities, axis=-1)\n    print(predictions)\n\n    print('Generating submission.csv file...')\n    test_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\n    test_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\n    np.savetxt('submission.csv', np.rec.fromarrays([test_ids, predictions]), fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')\n    \nelse:\n    \n    print('Computing predictions with TTA ...')\n    preds_tta = []\n    \n    for i in range(8):\n        # did not have time to work out the TTA code - my implemented augmentations are probably too much\n        test_ds = get_test_dataset(ordered=True, do_aug=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n        test_images_ds = test_ds.map(lambda image, idnum: image)\n        probabilities = best_alpha*model.predict(test_images_ds) + (1-best_alpha)*model2.predict(test_images_ds)\n        predictions = np.argmax(probabilities, axis=-1)\n        preds_tta.append(predictions)\n    \n    final_pred = np.mean(preds_tta, axis=0)\n    \n    print(final_pred)\n\n    print('Generating submission.csv file...')\n    test_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\n    test_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\n    np.savetxt('submission.csv', np.rec.fromarrays([test_ids, final_pred]), fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')\n\nprint('Done')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}