{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Flower classification with TPUs\n\n## Problem\nThis kernel is to show how to use TPUs to classify images of 104 types of flowers. In this competition we need to classify the flowers based on their images drawn from five different public datasets. Some classes are very narrow, containing only a particular sub-type of flower (e.g. pink primroses) while other classes contain many sub-types (e.g. wild roses).\n\n## Data\nThe images are provided in the competition dataset \"flower-classification-with-tpus\" in TFRecord format. This particular format is frequently used in Tensorflow to group and shard data files. Each file contains an id, label(class of flower) and img(pixels in array form).\n\n1. train/*.tfrec - training samples with labels.\n2. val/*.tfrec    - validation samples with labels.\n3. test/*.tfrec   - test samples without labels.\n\n## Overview\n\nThe details of the competition can be find in the following link :\n> [Flower Classification with TPUs](https://www.kaggle.com/c/flower-classification-with-tpus/overview)> \n\nThe kernel includes basic EDA(Exploratory Data Analysis) and models to classify the flowers to their correct category.\n\n## Solution\n\nI've used an ensemble model of DensetNet201 and EfficientNet-B7.\n\n* A model of DensetNet201 pre-trained on the \"imagenet\" dataset with a softmax activation function.\n* A model of EfficientNet-B7 pre-trained on the \"noisy-student\" dataset with a softmax activation function.\n* Adam optimizer for learning rate optimization in both the models.\n* Sparse Categorical Crossentropy as the loss function. \n\n## Notes\n\n* There were multiple external datasets which were created during the competition. But unfortunately some of the test images were present in  these public external dataset. As a result of this, it was notified that external datasets were to be used only if these datasets were clean and free of any test image. Hence, I've not used the external dataset in this kernel. However, I'll attach some very useful kernels which shows the usage of external dataset. \n\n* You will see that the validation data is being integrated with the training data in the kernel. This is being done to show that including validation data can help in better performance in the leaderboard. However, this should not be done in real time projects as it can give you wrong sense of model performance. You can set the variable \"USE_VALIDATION_DATA\" to True/False to see the difference in the performances.\n\n## Acknowledgment\n\nI'd like to thank authors of the below kernels. These kernels were very helpful in understanding how to work with TPUs in this Playground Competition :\n\n\n1. [Getting started with 100+ flowers on TPU by Martin Görner](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu)\n2. [TPU flowers - First Love by Roman Weilguny](https://www.kaggle.com/romanweilguny/tpu-flowers-first-love)\n3. [gold medalRotation Augmentation GPU/TPU - [0.96+] by Chris Deotte](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96)\n4. [EfficientNet-With-All-5-Imagesets-S1 by Haveri](https://www.kaggle.com/haveri/efficientnet-with-all-5-imagesets-s1)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Dependencies","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install -q efficientnet\nimport math, re, os\nimport tensorflow as tf\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport efficientnet.tfkeras as efn\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\nprint(\"Tensorflow version \" + tf.__version__)\nAUTO = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TPU Configuration\n\nPlease read the below document given by Kaggle for better understanding of the below configuration:\n\n[Tensor Processing Units (TPUs)](https://www.kaggle.com/docs/tpu)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TPU variables initialization","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Data access ...\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('flower-classification-with-tpus')\n\nIMAGE_SIZE = [512, 512] # at this size, a GPU will run out of memory. Use the TPU\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n\nGCS_PATH_SELECT = { # available image sizes\n    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'\n}\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') # predictions on this dataset should be submitted for the competition\n\nCLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 102","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using the validation data for training to inch out last marginal performance improvement\n\n* USE_VALIDATION_DATA = True  -> Validation data will not be included in Training data\n* USE_VALIDATION_DATA = False -> Validation data will be included in Training data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"USE_VALIDATION_DATA = True\nif not USE_VALIDATION_DATA:\n    TRAINING_FILENAMES = TRAINING_FILENAMES + VALIDATION_FILENAMES","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataset Utilities","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset\n\ndef data_augment(image, label):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement in the next function (below),\n    # this happens essentially for free on TPU. Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n    image = tf.image.random_flip_left_right(image)\n    #image = tf.image.random_saturation(image, 0, 2)\n    return image, label   \n\ndef get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataset visualization","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":false,"_kg_hide-input":false},"cell_type":"code","source":"# data dump\nprint(\"Training data shapes:\")\nfor image, label in get_training_dataset().take(1):\n    print(image.numpy().shape, label.numpy().shape)\nprint(\"Training data label examples:\", label.numpy())\nprint(\"n\")\nprint(\"Validation data shapes:\")\nfor image, label in get_validation_dataset().take(1):\n    print(image.numpy().shape, label.numpy().shape)\nprint(\"Validation data label examples:\", label.numpy())\nprint(\"n\")\nprint(\"Test data shapes:\")\nfor image, idnum in get_test_dataset().take(1):\n    print(image.numpy().shape, idnum.numpy().shape)\nprint(\"Test data IDs:\", idnum.numpy().astype('U')) # U=unicode string","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Visualization utilities","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Visualization utility functions\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n\ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n    \n# Visualize model predictions\ndef dataset_to_numpy_util(dataset, N):\n    dataset = dataset.unbatch().batch(N)\n    for images, labels in dataset:\n        numpy_images = images.numpy()\n        numpy_labels = labels.numpy()\n        break;  \n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    label = np.argmax(label, axis=-1)\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], str(correct), ', shoud be ' if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower_eval(image, title, subplot, red=False):\n    plt.subplot(subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    plt.title(title, fontsize=14, color='red' if red else 'black')\n    return subplot+1\n\ndef display_9_images_with_predictions(images, predictions, labels):\n    subplot=331\n    plt.figure(figsize=(13,13))\n    for i, image in enumerate(images):\n        title, correct = title_from_label_and_target(predictions[i], labels[i])\n        subplot = display_one_flower_eval(image, title, subplot, not correct)\n        if i >= 8:\n            break;\n              \n    plt.tight_layout()\n    plt.subplots_adjust(wspace=0.1, hspace=0.1)\n    plt.show()\n    \ndef display_confusion_matrix(cmat, score, precision, recall):\n    plt.figure(figsize=(15,15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    plt.show()\n    \ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Visualization","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Training Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Peek at training data\ntraining_dataset = get_training_dataset()\ntraining_dataset = training_dataset.unbatch().batch(20)\ntrain_batch = iter(training_dataset)\ndisplay_batch_of_images(next(train_batch))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Validation Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Peek at validation data\nvalidation_dataset = get_validation_dataset()\nvalidation_dataset = validation_dataset.unbatch().batch(20)\nvalidation_batch = iter(validation_dataset)\ndisplay_batch_of_images(next(validation_batch))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Test Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Peek at test data\ntest_dataset = get_test_dataset()\ntest_dataset = test_dataset.unbatch().batch(20)\ntest_batch = iter(test_dataset)\ndisplay_batch_of_images(next(test_batch))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualization - Label Distribution","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = next(iter(training_dataset.unbatch().map(lambda image, label: label).batch(NUM_TRAINING_IMAGES))).numpy()\n\ny_valid = next(iter(validation_dataset.unbatch().map(lambda image, label: label).batch(NUM_VALIDATION_IMAGES))).numpy()\n\ntrain_agg = np.asarray([[label, (y_train == index).sum()] for index, label in enumerate(CLASSES)])\nvalid_agg = np.asarray([[label, (y_valid == index).sum()] for index, label in enumerate(CLASSES)])\n\nfig, (ax1, ax2) = plt.subplots(2, 1, figsize=(24, 64))\n\nax1 = sns.barplot(x=train_agg[...,1], y=train_agg[...,0], order=CLASSES, ax=ax1)\nax1.set_title('Train', fontsize=30)\nax1.tick_params(labelsize=16)\n\nax2 = sns.barplot(x=valid_agg[...,1], y=valid_agg[...,0], order=CLASSES, ax=ax2)\nax2.set_title('Validation', fontsize=30)\nax2.tick_params(labelsize=16)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Models","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### DenseNet201 pre-trained with *imagenet* dataset","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"with strategy.scope():\n    dn201 = tf.keras.applications.DenseNet201(weights='imagenet', include_top=False, input_shape=[*IMAGE_SIZE, 3])\n    dn201.trainable = True # Full Training\n    \n    model_densenet201 = tf.keras.Sequential([\n        dn201,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n        \nmodel_densenet201.compile(\n    optimizer = tf.keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, amsgrad=False),\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy']\n)\nmodel_densetnet201.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### EfficientNet-B7 pre-trained with *noisy-student* dataset","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"with strategy.scope():\n    enb7 = efn.EfficientNetB7(weights='noisy-student', include_top=False, input_shape=[*IMAGE_SIZE, 3])\n    enb7.trainable = True # Full Training\n    \n    model_efficientnetb7 = tf.keras.Sequential([\n        enb7,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n        \nmodel_efficientnetb7.compile(\n    optimizer = tf.keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, amsgrad=False),\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy']\n)\nmodel_efficientnetb7.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Using Learning Rate Schedular and Early Stopping Callbacks.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"LR_START = 0.0001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 4\nLR_SUSTAIN_EPOCHS = 4\nLR_EXP_DECAY = .8\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=True)\n\nes_callback = tf.keras.callbacks.EarlyStopping(min_delta=0, patience=10, verbose=1, mode='auto', restore_best_weights=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Training the DenseNet201 model for 30 epochs","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"if USE_VALIDATION_DATA:\n    history_densenet201 = model_densenet201.fit(get_training_dataset(), steps_per_epoch=STEPS_PER_EPOCH, epochs=30, validation_data=get_validation_dataset(), callbacks = [lr_callback, es_callback])\nelse:\n    history_densenet201 = model_densenet201.fit(get_training_dataset(), steps_per_epoch=STEPS_PER_EPOCH, epochs=30, callbacks = [lr_callback]).history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Training the EfficientNet-B7 model for 40 epochs","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"if USE_VALIDATION_DATA:\n    history_efficientnetb7 = model_efficientnetb7.fit(get_training_dataset(), steps_per_epoch=STEPS_PER_EPOCH, epochs=40, validation_data=get_validation_dataset(), callbacks = [lr_callback, es_callback])\nelse:\n    history_efficientnetb7 = model_efficientnetb7.fit(get_training_dataset(), steps_per_epoch=STEPS_PER_EPOCH, epochs=40, callbacks = [lr_callback])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Loss Graphs","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if USE_VALIDATION_DATA:\n    display_training_curves(history_densenet201.history['loss'], history_densenet201.history['val_loss'], 'loss', 211)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if USE_VALIDATION_DATA:\n    display_training_curves(history_efficientnetb7.history['loss'], history_efficientnetb7.history['val_loss'], 'loss', 211)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if USE_VALIDATION_DATA:    \n    cmdataset = get_validation_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and labels, order matters.\n    images_ds = cmdataset.map(lambda image, label: image)\n    labels_ds = cmdataset.map(lambda image, label: label).unbatch()\n    cm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch\n    cm_probabilities = model_densenet201.predict(images_ds) * 0.4 + model_efficientnetb7.predict(images_ds) * 0.6\n    cm_predictions = np.argmax(cm_probabilities, axis=-1)\n    print(\"Correct   labels: \", cm_correct_labels.shape, cm_correct_labels)\n    print(\"Predicted labels: \", cm_predictions.shape, cm_predictions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Confusion Matrix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if USE_VALIDATION_DATA: \n    cmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))\n    score = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\n    precision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\n    recall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\n    display_confusion_matrix(cmat, score, precision, recall)\n    print('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prediction","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = model_densenet201.predict(test_images_ds) * 0.4 + model_efficientnetb7.predict(test_images_ds) * 0.6\npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)\n\nprint('Generating submission.csv file...')\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\nnp.savetxt('submission.csv', np.rec.fromarrays([test_ids, predictions]), fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')\n!head submission.csv","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualization for prediction of Validation Dataset\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if USE_VALIDATION_DATA:\n    validation_dataset = validation_dataset.unbatch().batch(1)\n    batch = iter(validation_dataset)\n    images, labels = next(batch)\n    probabilities = model_densenet201.predict(images) * 0.4 + model_efficientnetb7.predict(images) * 0.6\n    predictions = np.argmax(probabilities, axis=-1)\n    display_batch_of_images((images, labels), predictions)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}