{"cells":[{"metadata":{},"cell_type":"markdown","source":"# TPU CLASSIFICATION OF FLOWERS"},{"metadata":{},"cell_type":"markdown","source":"There are over 5,000 species of mammals, 10,000 species of birds, 30,000 species of fish – and astonishingly, over 400,000 different types of flowers.\n\nIn this competition, you’re challenged to build a machine learning model that identifies the type of flowers in a dataset of images (for simplicity, we’re sticking to just over 100 types)."},{"metadata":{},"cell_type":"markdown","source":"### Files\n\nThis competition is different in that images are provided in TFRecord format. The TFRecord format is a container format frequently used in Tensorflow to group and shard data data files for optimal training performace. Each file contains the id, label (the class of the sample, for training data) and img (the actual pixels in array form) information for many images. \n\ntrain/*.tfrec - training samples, including labels.\nval/*.tfrec - pre-split training samples w/ labels intended to help with checking your model's performance on TPU. The split was stratified across labels.\ntest/*.tfrec - samples without labels - you'll be predicting what classes of flowers these fall into.\nsample_submission.csv - a sample submission file in the correct format\n\nid - a unique ID for each sample.\n\nlabel - (in training data) the class of flower represented by the sample"},{"metadata":{},"cell_type":"markdown","source":"## Acknowledgement\n\nhttps://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu\n\nhttps://www.kaggle.com/sebastiankoenig/flower-classification-ensemble\n\nhttps://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu\n\nhttps://www.kaggle.com/phunghieu/flowers-with-tpu-ensembling-models"},{"metadata":{},"cell_type":"markdown","source":"Introduced the LR Scheduling in version 16. Until then my score was touching only .90. In version 19, it jumped to .96 with kfold\nVersion 20 - Ensembling of models Efficient Net B7, Resnet152, Inception Resnet V2 and Densenet201\n\nVersion 21 - Efficient Net B7 with (Kfold) 5 folds\n\nVersion 23 - External dataset  for training\n\n[External DS tfrec](https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec)\n\n[Thread1](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/148329)\n\n[Thread2](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/140866)\n\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#!pip install keras-rectified-adam","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import efficientnet.tfkeras as efn\nfrom tensorflow.keras.applications import InceptionResNetV2\nfrom tensorflow.keras.applications import ResNet152V2\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\n#import keras_radam\n#from keras_radam import RAdam","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n#for dirname, _, filenames in os.walk('/kaggle/input'):\n    #for filename in filenames:\n        #print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import display\nfrom IPython.core.interactiveshell import InteractiveShell\n#InteractiveShell.ast_node_interactivity = \"all\"\nfrom sklearn.model_selection import KFold\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import math, re\nimport tensorflow as tf\nimport numpy as np\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nprint(\"Tensorflow version \" + tf.__version__)\nAUTO = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Data access\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('flower-classification-with-tpus')\n# Configuration\nIMAGE_SIZE = [512,512]\n#PATH2 = KaggleDatasets().get_gcs_path('tf-flower-photo-tfrec')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### External datasets not allowed , so going back to previous version(kernel)"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nExternal dataset is disallowed in this competition\n#When using external DS\nGCS_PATH_SELECT = { # available image sizes\n    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'\n}\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\nTRAINING_FILENAMES = tf.io.gfile.glob(PATH2 + '/imagenet/tfrecords-jpeg-512x512/*.tfrec')\nTRAINING_FILENAMES = TRAINING_FILENAMES + tf.io.gfile.glob(PATH2 + '/inaturalist_1/tfrecords-jpeg-512x512/*.tfrec')\nTRAINING_FILENAMES = TRAINING_FILENAMES + tf.io.gfile.glob(PATH2 + '/openimage/tfrecords-jpeg-512x512/*.tfrec')\nTRAINING_FILENAMES = TRAINING_FILENAMES + tf.io.gfile.glob(PATH2 + '/oxford_102/tfrecords-jpeg-512x512/*.tfrec')\nTRAINING_FILENAMES = TRAINING_FILENAMES + tf.io.gfile.glob(PATH2 + '/tf_flowers/tfrecords-jpeg-512x512/*.tfrec')\n\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') # predictions on this dataset should be submitted for the competition\n\n# watch out for overfitting!\nSKIP_VALIDATION = False\nif SKIP_VALIDATION:\n    TRAINING_FILENAMES = TRAINING_FILENAMES + VALIDATION_FILENAMES\n    \n'''  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nGCS_PATH_SELECT = { # available image sizes\n    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'\n}\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') # predictions on this dataset should be submitted for the competition\n\n# watch out for overfitting!\nSKIP_VALIDATION = False\nif SKIP_VALIDATION:\n    TRAINING_FILENAMES = TRAINING_FILENAMES + VALIDATION_FILENAMES\n    \n ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"len(TRAINING_FILENAMES)\nlen(VALIDATION_FILENAMES)\nlen(TEST_FILENAMES)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CLASSES = ['pink primrose',\n           'hard-leaved pocket orchid',\n           'canterbury bells', \n           'sweet pea',    \n           'wild geranium',  \n           'tiger lily',       \n           'moon orchid',     \n           'bird of paradise',\n           'monkshood',     \n           'globe thistle',         # 00 - 09\n           'snapdragon', \n           \"colt's foot\",      \n           'king protea',   \n           'spear thistle',\n           'yellow iris',     \n           'globe-flower',  \n           'purple coneflower',  \n           'peruvian lily',   \n           'balloon flower',  \n           'giant white arum lily', # 10 - 19\n           'fire lily',  \n           'pincushion flower',  \n           'fritillary',    \n           'red ginger', \n           'grape hyacinth', \n           'corn poppy',     \n           'prince of wales feathers',\n           'stemless gentian',\n           'artichoke',       \n           'sweet william',         # 20 - 29\n           'carnation',   \n           'garden phlox',     \n           'love in the mist',\n           'cosmos',       \n           'alpine sea holly',\n           'ruby-lipped cattleya',\n           'cape flower',        \n           'great masterwort', \n           'siam tulip',      \n           'lenten rose',           # 30 - 39\n           'barberton daisy',\n           'daffodil',        \n           'sword lily',     \n           'poinsettia',   \n           'bolero deep blue',  \n           'wallflower',       \n           'marigold',         \n           'buttercup',       \n           'daisy',        \n           'common dandelion',      # 40 - 49\n           'petunia',     \n           'wild pansy',        \n           'primula',        \n           'sunflower',     \n           'lilac hibiscus',  \n           'bishop of llandaff', \n           'gaura',              \n           'geranium',       \n           'orange dahlia',  \n           'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata',\n           'japanese anemone', \n           'black-eyed susan',\n           'silverbush',  \n           'californian poppy',\n           'osteospermum',   \n           'spring crocus',  \n           'iris',       \n           'windflower',   \n           'tree poppy',            # 60 - 69\n           'gazania',    \n           'azalea',   \n           'water lily', \n           'rose',          \n           'thorn apple',   \n           'morning glory',  \n           'passion flower',  \n           'lotus',           \n           'toad lily',      \n           'anthurium',             # 70 - 79\n           'frangipani', \n           'clematis',      \n           'hibiscus',      \n           'columbine',   \n           'desert-rose',     \n           'tree mallow',   \n           'magnolia',       \n           'cyclamen ',      \n           'watercress',     \n           'canna lily',            # 80 - 89\n           'hippeastrum ', \n           'bee balm',       \n           'pink quill',     \n           'foxglove',    \n           'bougainvillea', \n           'camellia',      \n           'mallow',          \n           'mexican petunia', \n           'bromelia',         \n           'blanket flower',        # 90 - 99\n           'trumpet creeper', \n           'blackberry lily',   \n           'common tulip',    \n           'wild rose']                                                                                                                                               # 100 - 102","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MIXED_PRECISION = False\nXLA_ACCELERATE = False\n\nif MIXED_PRECISION:\n    from tensorflow.keras.mixed_precision import experimental as mixed_precision\n    if tpu: policy = tf.keras.mixed_precision.experimental.Policy('mixed_bfloat16')\n    else: policy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\n    mixed_precision.set_policy(policy)\n    print('Mixed precision enabled')\n\nif XLA_ACCELERATE:\n    tf.config.optimizer.set_jit(True)\n    print('Accelerated Linear Algebra enabled')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## DETECT ACCELERATOR"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LR SCHEDULING"},{"metadata":{"trusted":true},"cell_type":"code","source":"EPOCHS = 20","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Learning rate schedule for TPU, GPU and CPU.\n# Using an LR ramp up because fine-tuning a pre-trained model.\n# Starting with a high LR would break the pre-trained weights.\n\nLR_START = 0.00001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 4\nLR_SUSTAIN_EPOCHS = 0 \nLR_EXP_DECAY = .8\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n\nrng = [i for i in range(50 if EPOCHS<50 else EPOCHS)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## PREPROCESSING & LOADIN"},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset\n\ndef data_augment(image, label, seed=2020):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement in the next function (below),\n    # this happens essentially for free on TPU. Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n    image = tf.image.random_flip_left_right(image, seed=seed)\n#     image = tf.image.random_flip_up_down(image, seed=seed)\n#     image = tf.image.random_brightness(image, 0.1, seed=seed)\n    \n#     image = tf.image.random_jpeg_quality(image, 85, 100, seed=seed)\n#     image = tf.image.resize(image, [530, 530])\n#     image = tf.image.random_crop(image, [512, 512], seed=seed)\n    #image = tf.image.random_saturation(image, 0, 2)\n    return image, label   \n\ndef get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_training_dataset(dataset, do_aug=True):\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.batch(AUG_BATCH)\n    if do_aug: dataset = dataset.map(transform, num_parallel_calls=AUTO) # note we put AFTER batching\n    dataset = dataset.unbatch()\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\n\ndef get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_train_valid_datasets():\n    dataset = load_dataset(TRAINING_FILENAMES + VALIDATION_FILENAMES, labeled=True)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## SET PARAMETERS"},{"metadata":{"trusted":true},"cell_type":"code","source":"#EPOCHS = 15\nBATCH_SIZE = 8 * strategy.num_replicas_in_sync\nAUG_BATCH = BATCH_SIZE\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))\n#AUTO = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n\ndef display_confusion_matrix(cmat, score, precision, recall):\n    plt.figure(figsize=(15,15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='BuPu')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    plt.show()\n    \ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Peek at training data\ndataset = load_dataset(TRAINING_FILENAMES, labeled=True)\ntraining_dataset = get_training_dataset(dataset, do_aug=False)\ntraining_dataset = training_dataset.unbatch().batch(20)\ntrain_batch = iter(training_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Training data shapes:\")\nfor image, label in get_training_dataset(dataset, do_aug=False).take(2):\n    print(image.numpy().shape, label.numpy().shape)\n    train_image = image.numpy()\n    train_label = label.numpy()\nprint(\"Training data label examples:\", label.numpy())\nprint(\"Validation data shapes:\")\nfor image, label in get_validation_dataset().take(2):\n    print(image.numpy().shape, label.numpy().shape)\nprint(\"Validation data label examples:\", label.numpy())\nprint(\"Test data shapes:\")\nfor image, idnum in get_test_dataset().take(2):\n    print(image.numpy().shape, idnum.numpy().shape)\nprint(\"Test data IDs:\", idnum.numpy().astype('U')) # U=unicode string","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Peek at training data\nvalid_dataset = get_validation_dataset()\nvalid_dataset = valid_dataset.unbatch().batch(20)\nval_batch = iter(valid_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Visualization**"},{"metadata":{},"cell_type":"markdown","source":"## Training set"},{"metadata":{"trusted":true},"cell_type":"code","source":"# run this cell again for next set of images\ndisplay_batch_of_images(next(train_batch))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Validation set"},{"metadata":{"trusted":true},"cell_type":"code","source":"# run this cell again for next set of images\ndisplay_batch_of_images(next(val_batch))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# peer at test data\ntest_dataset = get_test_dataset()\ntest_dataset = test_dataset.unbatch().batch(20)\ntest_batch = iter(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test set"},{"metadata":{"trusted":true},"cell_type":"code","source":"# run this cell again for next set of images\ndisplay_batch_of_images(next(test_batch))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## CutMix"},{"metadata":{"trusted":true},"cell_type":"code","source":"def onehot(image,label):\n    CLASSES = 104\n    return image,tf.one_hot(label,CLASSES)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def cutmix(image, label, PROBABILITY = 1.0):\n    # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n    # output - a batch of images with cutmix applied\n    DIM = IMAGE_SIZE[0]\n    CLASSES = 104\n    \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n        # DO CUTMIX WITH PROBABILITY DEFINED ABOVE\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.int32)\n        # CHOOSE RANDOM IMAGE TO CUTMIX WITH\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n        # CHOOSE RANDOM LOCATION\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        b = tf.random.uniform([],0,1) # this is beta dist with alpha=1.0\n        WIDTH = tf.cast( DIM * tf.math.sqrt(1-b),tf.int32) * P\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n        # MAKE CUTMIX IMAGE\n        one = image[j,ya:yb,0:xa,:]\n        two = image[k,ya:yb,xa:xb,:]\n        three = image[j,ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        img = tf.concat([image[j,0:ya,:,:],middle,image[j,yb:DIM,:,:]],axis=0)\n        imgs.append(img)\n        # MAKE CUTMIX LABEL\n        a = tf.cast(WIDTH*WIDTH/DIM/DIM,tf.float32)\n        if len(label.shape)==1:\n            lab1 = tf.one_hot(label[j],CLASSES)\n            lab2 = tf.one_hot(label[k],CLASSES)\n        else:\n            lab1 = label[j,]\n            lab2 = label[k,]\n        labs.append((1-a)*lab1 + a*lab2)\n            \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image2,label2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"row = 6; col = 4;\nrow = min(row,AUG_BATCH//col)\nall_elements = get_training_dataset(load_dataset(TRAINING_FILENAMES),do_aug=False).unbatch()\naugmented_element = all_elements.repeat().batch(AUG_BATCH).map(cutmix)\n\nfor (img,label) in augmented_element:\n    plt.figure(figsize=(15,int(15*row/col)))\n    for j in range(row*col):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def mixup(image, label, PROBABILITY = 1.0):\n    # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n    # output - a batch of images with mixup applied\n    DIM = IMAGE_SIZE[0]\n    CLASSES = 104\n    \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n        # DO MIXUP WITH PROBABILITY DEFINED ABOVE\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.float32)\n        # CHOOSE RANDOM\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n        a = tf.random.uniform([],0,1)*P # this is beta dist with alpha=1.0\n        # MAKE MIXUP IMAGE\n        img1 = image[j,]\n        img2 = image[k,]\n        imgs.append((1-a)*img1 + a*img2)\n        # MAKE CUTMIX LABEL\n        if len(label.shape)==1:\n            lab1 = tf.one_hot(label[j],CLASSES)\n            lab2 = tf.one_hot(label[k],CLASSES)\n        else:\n            lab1 = label[j,]\n            lab2 = label[k,]\n        labs.append((1-a)*lab1 + a*lab2)\n            \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image2,label2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def transform(image,label):\n    # THIS FUNCTION APPLIES BOTH CUTMIX AND MIXUP\n    DIM = IMAGE_SIZE[0]\n    CLASSES = 104\n    SWITCH = 0.5\n    CUTMIX_PROB = 0.666\n    MIXUP_PROB = 0.666\n    # FOR SWITCH PERCENT OF TIME WE DO CUTMIX AND (1-SWITCH) WE DO MIXUP\n    image2, label2 = cutmix(image, label, CUTMIX_PROB)\n    image3, label3 = mixup(image, label, MIXUP_PROB)\n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n        P = tf.cast( tf.random.uniform([],0,1)<=SWITCH, tf.float32)\n        imgs.append(P*image2[j,]+(1-P)*image3[j,])\n        labs.append(P*label2[j,]+(1-P)*label3[j,])\n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n    image4 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label4 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image4,label4","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"row = 6; col = 4;\nrow = min(row,AUG_BATCH//col)\nall_elements = get_training_dataset(load_dataset(TRAINING_FILENAMES),do_aug=False).unbatch()\naugmented_element = all_elements.repeat().batch(AUG_BATCH).map(mixup)\n\nfor (img,label) in augmented_element:\n    plt.figure(figsize=(15,int(15*row/col)))\n    for j in range(row*col):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Display 33%/33%/33% CutMix/MixUp/None"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"row = 6; col = 4;\nrow = min(row,AUG_BATCH//col)\nall_elements = get_training_dataset(load_dataset(TRAINING_FILENAMES),do_aug=False).unbatch()\naugmented_element = all_elements.repeat().batch(AUG_BATCH).map(transform)\n\nfor (img,label) in augmented_element:\n    plt.figure(figsize=(15,int(15*row/col)))\n    for j in range(row*col):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nfrom keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau\nweight_path=\"{}_tpu_weights.h5\".format('flower_classify')\ncheckpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1, \n                             save_best_only=True, mode='min', save_weights_only = True)\n\n\nreduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.8, patience=3, verbose=1, mode='auto', \n                                   min_delta=0.01, cooldown=3, min_lr=0.0001)\nearly = EarlyStopping(monitor=\"val_loss\", \n                      mode=\"min\",restore_best_weights=True, \n                      patience=10) # probably needs to be more patient\ncallbacks_list = [checkpoint, early, reduceLROnPlat]\n'''","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ** EfficientNetB7"},{"metadata":{"trusted":true},"cell_type":"code","source":"def call_model_efn():\n    enet = efn.EfficientNetB7(\n        input_shape=(512, 512, 3),\n        weights = 'noisy-student', #weights='imagenet',\n        include_top=False\n    )\n\n    enet.trainable = True\n\n    model = tf.keras.Sequential([\n        enet,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n    \n    #opt = RAdam(total_steps=5000, warmup_proportion=0.1, min_lr=1e-5)\n    \n    model.compile(\n        optimizer='adam',\n        loss = 'sparse_categorical_crossentropy',\n        metrics=['sparse_categorical_accuracy']\n        #tf.keras.optimizers.Adam(lr=1e-3),\n        #loss = 'sparse_categorical_crossentropy',\n        #metrics=['sparse_categorical_accuracy']\n    )\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Kfold CV  [reference](https://www.machinecurve.com/index.php/2020/02/18/how-to-use-k-fold-cross-validation-with-keras/)"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\ntrain_inputs, train_labels = batch_to_numpy_images_and_labels(next(train_batch))\nval_inputs, val_labels = batch_to_numpy_images_and_labels(next(val_batch))\n\ninputs = np.concatenate((train_inputs, val_inputs), axis=0)\ntargets = np.concatenate((train_labels, val_labels), axis=0)\n\n#print (inputs, targets)\n'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#weights_model='/kaggle/input/tpu-flowers-kfold-cv/flower_classify_tpu_weights.h5'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nnum_folds = 2\n\n# Define per-fold score containers\nacc_per_fold = []\nloss_per_fold = []\n\n\n# Define the K-fold Cross Validator\nkfold = KFold(n_splits=num_folds, shuffle=True, random_state = 42)\n\n# K-fold Cross Validation model evaluation\nfold_no = 1\nfor train, val in kfold.split(inputs, targets):\n    with strategy.scope():\n        enet_model = call_model_efn()\n    enet_model.summary()\n    #enet_model.load_weights(weights_model)\n    # Generate a print\n    print('------------------------------------------------------------------------')\n    print(f'Training for fold {fold_no} ...')\n    print('------------------------------------------------------------------------')\n    # scheduler = tf.keras.callbacks.ReduceLROnPlateau(patience=3, verbose=1)\n    #lr_schedule = tf.keras.callbacks.LearningRateScheduler(reduceLROnPlat, verbose=1)\n    history_enet = enet_model.fit(\n        #get_training_dataset(),#get_train_valid_datasets(),\n        get_training_dataset(load_dataset(TRAINING_FILENAMES),do_aug=False),\n        steps_per_epoch=STEPS_PER_EPOCH,\n        epochs=EPOCHS, \n        callbacks = [lr_callback],\n        #callbacks=[checkpoint, early, reduceLROnPlat],\n        validation_data = get_validation_dataset(),\n        verbose=2\n    )\n    # Generate generalization metrics\n    scores = enet_model.evaluate(get_validation_dataset())\n    print(f'Score for fold {fold_no}: {enet_model.metrics_names[0]} of {scores[0]}; {enet_model.metrics_names[1]} of {scores[1]*100}%')\n    acc_per_fold.append(scores[1] * 100)\n    loss_per_fold.append(scores[0])\n    # Increase fold number\n    fold_no = fold_no + 1\n    \n    \n# == Provide average scores ==\nprint('\\n------------------------------------------------------------------------')\nprint('Score per fold')\nfor i in range(0, len(acc_per_fold)):\n  print('------------------------------------------------------------------------')\n  print(f'> Fold {i+1} - Loss: {loss_per_fold[i]} - Accuracy: {acc_per_fold[i]}%')\nprint('------------------------------------------------------------------------')\nprint('Average scores for all folds:')\nprint(f'> Accuracy: {np.mean(acc_per_fold)} (+- {np.std(acc_per_fold)})')\nprint(f'> Loss: {np.mean(loss_per_fold)}')\nprint('------------------------------------------------------------------------')    \n'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n#Skip this cell if you use k-fold\n#use for ensemble\nwith strategy.scope():\n    enet_model = call_model_efn()\nenet_model.summary()\n\nhistory_enet = enet_model.fit(\n    #get_training_dataset(),#get_train_valid_datasets(),\n    get_training_dataset(load_dataset(TRAINING_FILENAMES),do_aug=False),\n    steps_per_epoch=STEPS_PER_EPOCH,\n    epochs=EPOCHS, \n    #callbacks=[checkpoint, early, reduceLROnPlat],\n    callbacks = [lr_callback],\n    validation_data = get_validation_dataset(),\n    verbose=2\n)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])\n    \ndisplay_training_curves(history_enet.history['loss'], history_enet.history['val_loss'], 'loss', 211)\ndisplay_training_curves(history_enet.history['sparse_categorical_accuracy'], history_enet.history['val_sparse_categorical_accuracy'], 'accuracy', 212)    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Confusion Matrix"},{"metadata":{"trusted":true},"cell_type":"code","source":"cmdataset = get_validation_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and labels, order matters.\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch\ncm_probabilities = enet_model.predict(images_ds)\ncm_predictions = np.argmax(cm_probabilities, axis=-1)\nprint(\"Correct   labels: \", cm_correct_labels.shape, cm_correct_labels)\nprint(\"Predicted labels: \", cm_predictions.shape, cm_predictions)\n\ncmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))\nscore = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\nprecision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\nrecall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\ncmat = (cmat.T / cmat.sum(axis=1)).T # normalized\ndisplay_confusion_matrix(cmat, score, precision, recall)\nprint('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Efficient Net Predictions and Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nenet_probabilities = enet_model.predict(test_images_ds)\nenet_predictions = np.argmax(enet_probabilities, axis=-1)\nprint(enet_predictions)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nprint('Generating submission.csv file...')\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\nnp.savetxt('submission.csv', np.rec.fromarrays([test_ids, enet_predictions]), fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')\n'''","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Efficientnet  Visual Validations"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\n# run this cell again for next set of images\nimages, labels = next(val_batch)\nenet_probabilities = enet_model.predict(images)\nenet_predictions = np.argmax(enet_probabilities, axis=-1)\ndisplay_batch_of_images((images, labels), enet_predictions)\n'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#EPOCHS = 20","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ** Restnet152V2"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef call_model_rn152v2():\n    rnet = ResNet152V2(\n        input_shape=(512,512,3),\n        weights = 'imagenet', #weights='imagenet',\n        include_top=False\n    )\n\n    rnet.trainable = True\n\n    model = tf.keras.Sequential([\n        rnet,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n    \n    #opt = RAdam(total_steps=5000, warmup_proportion=0.1, min_lr=1e-5)\n    \n    model.compile(\n        #optimizer=tf.optimizers.RectifiedAdam(),\n        #tf.keras.optimizers.Adam(lr=1e-5),\n        optimizer = 'adam',\n        loss = 'sparse_categorical_crossentropy',\n        metrics=['sparse_categorical_accuracy']\n    )\n    return model\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nwith strategy.scope():\n    rnet_model = call_model_rn152v2()\nrnet_model.summary()\n\nhistory_resnet = rnet_model.fit(\n    #get_training_dataset(),#get_train_valid_datasets(),\n    get_training_dataset(load_dataset(TRAINING_FILENAMES),do_aug=False),\n    steps_per_epoch=STEPS_PER_EPOCH,\n    epochs=EPOCHS, \n    #callbacks=[checkpoint, early, reduceLROnPlat],\n    callbacks = [lr_callback], \n    validation_data = get_validation_dataset(),\n    verbose=2\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndisplay_training_curves(history_resnet.history['loss'], history_resnet.history['val_loss'], 'loss', 211)\ndisplay_training_curves(history_resnet.history['sparse_categorical_accuracy'], history_resnet.history['val_sparse_categorical_accuracy'], 'accuracy', 212) \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ncmdataset = get_validation_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and labels, order matters.\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\nrm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch\nrm_probabilities = rnet_model.predict(images_ds)\nrm_predictions = np.argmax(rm_probabilities, axis=-1)\nprint(\"Correct   labels: \", rm_correct_labels.shape, cm_correct_labels)\nprint(\"Predicted labels: \", rm_predictions.shape, cm_predictions)\n\nrmat = confusion_matrix(rm_correct_labels, rm_predictions, labels=range(len(CLASSES)))\nscore = f1_score(rm_correct_labels, rm_predictions, labels=range(len(CLASSES)), average='macro')\nprecision = precision_score(rm_correct_labels, rm_predictions, labels=range(len(CLASSES)), average='macro')\nrecall = recall_score(rm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\nrmat = (rmat.T / rmat.sum(axis=1)).T # normalized\ndisplay_confusion_matrix(rmat, score, precision, recall)\nprint('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Resnet152V2 Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntest_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nrnet_probabilities = rnet_model.predict(test_images_ds)\nrnet_predictions = np.argmax(rnet_probabilities, axis=-1)\nprint(rnet_predictions)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ** InceptionResNetV2"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef call_model_IRv2():\n    irnet = InceptionResNetV2(\n        input_shape=(512,512,3),\n        weights = 'imagenet', #weights='imagenet',\n        include_top=False\n    )\n\n    irnet.trainable = True\n\n    model = tf.keras.Sequential([\n        irnet,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n    \n    #opt = RAdam(total_steps=5000, warmup_proportion=0.1, min_lr=1e-5)\n    \n    model.compile(\n        #optimizer=tf.optimizers.RectifiedAdam(),\n        #tf.keras.optimizers.Adam(lr=1e-5),\n        optimizer = 'adam',\n        loss = 'sparse_categorical_crossentropy',\n        metrics=['sparse_categorical_accuracy']\n    )\n    return model\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nwith strategy.scope():\n    irnet_model = call_model_IRv2()\nirnet_model.summary()\n\nhistory_incepresnet = irnet_model.fit(\n    #get_training_dataset(),#get_train_valid_datasets(),\n    get_training_dataset(load_dataset(TRAINING_FILENAMES),do_aug=False),\n    steps_per_epoch=STEPS_PER_EPOCH,\n    epochs=EPOCHS, \n    callbacks = [lr_callback],\n    #callbacks=[checkpoint, early, reduceLROnPlat],\n    validation_data = get_validation_dataset(),\n    verbose=2\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndisplay_training_curves(history_incepresnet.history['loss'], history_incepresnet.history['val_loss'], 'loss', 211)\ndisplay_training_curves(history_incepresnet.history['sparse_categorical_accuracy'], history_incepresnet.history['val_sparse_categorical_accuracy'], 'accuracy', 212)   \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ncmdataset = get_validation_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and labels, order matters.\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch\ncm_probabilities = irnet_model.predict(images_ds)\ncm_predictions = np.argmax(cm_probabilities, axis=-1)\nprint(\"Correct   labels: \", cm_correct_labels.shape, cm_correct_labels)\nprint(\"Predicted labels: \", cm_predictions.shape, cm_predictions)\n\ncmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))\nrscore = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\nrprecision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\nrrecall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\ncmat = (cmat.T / cmat.sum(axis=1)).T # normalized\ndisplay_confusion_matrix(cmat, rscore, rprecision, rrecall)\nprint('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(rscore, rprecision, rrecall))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inception Resnet V2 Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntest_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nirnet_probabilities = irnet_model.predict(test_images_ds)\nirnet_predictions = np.argmax(irnet_probabilities, axis=-1)\nprint(irnet_predictions)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"## ** DenseNet 201"},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.applications import DenseNet201","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef call_model_Dnet201():\n    dnet = DenseNet201(\n        input_shape=(512,512,3),\n        weights='imagenet',\n        include_top=False\n    )\n\n    dnet.trainable = True\n\n    model = tf.keras.Sequential([\n        dnet,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n    \n    #opt = RAdam(total_steps=5000, warmup_proportion=0.1, min_lr=1e-5)\n    \n    model.compile(\n        #optimizer=tf.optimizers.RectifiedAdam(),\n        #tf.keras.optimizers.Adam(),\n        optimizer = 'adam',\n        loss = 'sparse_categorical_crossentropy',\n        metrics=['sparse_categorical_accuracy']\n    )\n    return model\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nwith strategy.scope():\n    dnet_model = call_model_Dnet201()\ndnet_model.summary()\n\nhistory_dnet = dnet_model.fit(\n    #get_training_dataset(),#get_train_valid_datasets(),\n    get_training_dataset(load_dataset(TRAINING_FILENAMES),do_aug=False),\n    steps_per_epoch=STEPS_PER_EPOCH,\n    epochs=EPOCHS, \n    callbacks = [lr_callback],\n    #callbacks=[checkpoint, early, reduceLROnPlat],\n    validation_data = get_validation_dataset(),\n    verbose=2\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndisplay_training_curves(history_dnet.history['loss'], history_dnet.history['val_loss'], 'loss', 211)\ndisplay_training_curves(history_dnet.history['sparse_categorical_accuracy'], history_dnet.history['val_sparse_categorical_accuracy'], 'accuracy', 212) \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ncmdataset = get_validation_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and labels, order matters.\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch\ncm_probabilities = dnet_model.predict(images_ds)\ncm_predictions = np.argmax(cm_probabilities, axis=-1)\nprint(\"Correct   labels: \", cm_correct_labels.shape, cm_correct_labels)\nprint(\"Predicted labels: \", cm_predictions.shape, cm_predictions)\n\ncmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))\nrscore = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\nrprecision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\nrrecall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\ncmat = (cmat.T / cmat.sum(axis=1)).T # normalized\ndisplay_confusion_matrix(cmat, rscore, rprecision, rrecall)\nprint('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(rscore, rprecision, rrecall))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Densenet Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntest_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\ndnet_probabilities = dnet_model.predict(test_images_ds)\ndnet_predictions = np.argmax(dnet_probabilities, axis=-1)\nprint(dnet_predictions)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"probabilities = (enet_probabilities + dnet_probabilities + irnet_probabilities + rnet_probabilities)/4\nensemble_predict = np.argmax(probabilities, axis=-1)\nprint(ensemble_predict)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ** Ensemble all EffNetB7,  Resnet 152 V2, InceptionResnetV2, Densenet 201 for submission "},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nprobabilities = np.mean(\n    [\n        enet_probabilities,\n        dnet_probabilities\n        ,irnet_probabilities\n        ,rnet_probabilities\n    ],\n    axis=0\n)\n\nensemble_predict = np.argmax(probabilities, axis=-1)\nprint(ensemble_predict)\n'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nprint('Generating submission.csv file...')\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\n\n#ensemble_predict = (np.mean [enet_predictions , rnet_predictions, irnet_predictions,dnet_predictions ], axis = 0)\nnp.savetxt('submission.csv', np.rec.fromarrays([test_ids, ensemble_predict]), fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')\nprint ('Submission Ensemble saved.....')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}