{"cells":[{"metadata":{},"cell_type":"markdown","source":"**To run this sample on Google Cloud Platform with various accelerator setups:**\n 1. Download this notebook\n 1. Create a Cloud AI Platform Notebook VM with your choice of accelerator.\n   * V100 GPU ([AI Platform Notebook UI](https://console.cloud.google.com/ai-platform/notebooks) > New Instance > Tensorflow 2.2 > Customize > V100 x1)\n   * 4x V100 GPU ([AI Platform Notebook UI](https://console.cloud.google.com/ai-platform/notebooks) > New Instance > Tensorflow 2.2 > Customize > V100 x 4)\n   * 8x V100 GPU ([AI Platform Notebook UI](https://console.cloud.google.com/ai-platform/notebooks) > New Instance > Tensorflow 2.2 > Customize > V100 x 8)\n   * TPU v3-8 (use `create-tpu-deep-learning-vm.sh` script from [this page](create-tpu-deep-learning-vm.sh) with `--tpu-type v3-8`)\n   * TPU v3-32 pod (use `create-tpu-deep-learning-vm.sh` script from [this page](create-tpu-deep-learning-vm.sh) with `--tpu-type v3-32`)\n 1. Get the data from Kaggle. The easiest is to run the cell below on Kaggle and copy the name of the GCS bucket where the dataset is cached. This bucket is a cache and will expire after a couple of days but it should be enough to run the notebook. Optionnally, for best performance, copy the data to your own bucket located in the same region as your TPU.\n 1. adjust the import and the `GCS_PATH` in the cell below."},{"metadata":{"trusted":true},"cell_type":"code","source":"# When not running on Kaggle, comment out this import\nfrom kaggle_datasets import KaggleDatasets\n# When not running on Kaggle, set a fixed GCS path here\nGCS_PATH = KaggleDatasets().get_gcs_path('flower-classification-with-tpus')\nprint(GCS_PATH)","execution_count":null,"outputs":[]},{"metadata":{"id":"89B27-TGiDNB"},"cell_type":"markdown","source":"## Imports"},{"metadata":{"id":"9u3d4Z7uQsmp","outputId":"fc0dbfa1-04cd-472a-93b2-b7d4931fc18e","trusted":true},"cell_type":"code","source":"import re, sys, math, logging\nimport tensorflow as tf\nimport tensorflow_hub as hub\nimport numpy as np\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\nfrom matplotlib import pyplot as plt\nprint(\"Tensorflow version \" + tf.__version__)\ntf.get_logger().setLevel(logging.ERROR)","execution_count":null,"outputs":[]},{"metadata":{"id":"mPo10cahZXXQ"},"cell_type":"markdown","source":"## TPU or GPU detection\nTPUClusterResolver() automatically detects a connected TPU on all Gooogle's\nplatforms: Colaboratory, AI Platform (ML Engine), Kubernetes and Deep Learning\nVMs provided the TPU_NAME environment variable is set on the VM."},{"metadata":{"id":"FpvUOuC3j27n","outputId":"5329c9e4-539d-403c-d2e5-754d04a7fe6a","trusted":true},"cell_type":"code","source":"try: # detect TPU\n    tpu = None\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect() # TPU detection\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept ValueError: # detect GPU(s) and enable mixed precision\n    strategy = tf.distribute.MirroredStrategy() # works on GPU and multi-GPU\n    policy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\n    tf.config.optimizer.set_jit(True) # XLA compilation\n    tf.keras.mixed_precision.experimental.set_policy(policy)\n    print('Mixed precision enabled')\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\n# mixed precision\n# On TPU, bfloat16/float32 mixed precision is automatically used in TPU computations.\n# Enabling it in Keras also stores relevant variables in bfloat16 format (memory optimization).\n# This additional optimization was not used for TPUs in this sample.\n# On GPU, specifically V100, mixed precision must be enabled for hardware TensorCores to be used.\n# XLA compilation must be enabled for this to work. (On TPU, XLA compilation is the default and cannot be turned off)","execution_count":null,"outputs":[]},{"metadata":{"id":"w9S3uKC_iXY5"},"cell_type":"markdown","source":"## Configuration"},{"metadata":{"id":"M3G-2aUBQJ-H","outputId":"2261e4f0-57a6-4b02-99b2-8441cbb1b1b2","trusted":true},"cell_type":"code","source":"IMAGE_SIZE = [512, 512]\nEPOCHS = 10\n\nif tpu:\n    BATCH_SIZE = 32 * strategy.num_replicas_in_sync\nelse:\n    BATCH_SIZE = 4 * strategy.num_replicas_in_sync\n\nGCS_PATH_SELECT = { # available image sizes\n    192: GCS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_PATH + '/tfrecords-jpeg-512x512'\n}\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec')\n\nCLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 102\n\nAUTO = tf.data.experimental.AUTOTUNE\n\n# Learning rate settings\nif strategy.num_replicas_in_sync > 8:\n    # settings for TPUv3-32\n    LR_MAX = 0.00006 * strategy.num_replicas_in_sync\n    LR_RAMPUP_EPOCHS = 3\n    LR_EXP_DECAY = .75\nelif strategy.num_replicas_in_sync == 4:\n    # settings for 4xV100\n    LR_MAX = 0.0001 * strategy.num_replicas_in_sync\n    LR_RAMPUP_EPOCHS = 4\n    LR_EXP_DECAY = .8\nelse:\n    # settings for TPUv3-8, V100, 8xV100\n    LR_MAX = 0.0001 * strategy.num_replicas_in_sync\n    LR_RAMPUP_EPOCHS = 3\n    LR_EXP_DECAY = .8\nLR_START = 0.00001\nLR_MIN = 0.00001\nLR_SUSTAIN_EPOCHS = 0\n\n@tf.function\ndef lr_fn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n\nprint(\"Learning rate schedule:\")\nrng = [i for i in range(EPOCHS)]\ny = [lr_fn(x) for x in rng]\nplt.plot(rng, [lr_fn(x) for x in rng])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"OzKIJyEJxYSP"},"cell_type":"markdown","source":"## Display utilities"},{"metadata":{"cellView":"form","id":"MPkvHdAYNt9J","trusted":true},"cell_type":"code","source":"# numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n    if type(data) == tuple:\n        images, labels = data\n        numpy_labels = labels.numpy()\n    else:\n        images = data\n        numpy_labels = None\n    numpy_images = images.numpy()\n    if numpy_labels is None or numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    #plt.tight_layout() ## buggy in this version of matplotlib ##\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n\ndef display_confusion_matrix(cmat, score, precision, recall):\n    plt.figure(figsize=(15,15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    plt.show()\n    \ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        #plt.tight_layout()  ## buggy in this version of matplotlib ##\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","execution_count":null,"outputs":[]},{"metadata":{"id":"kvPXiovhi3ZZ"},"cell_type":"markdown","source":"## Dataset"},{"metadata":{"id":"LtAVr-4CP1rp","outputId":"85b2fdf5-d65f-41f8-8a3a-b81e121ed203","trusted":true},"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3) # imgae format: uint8 [0,255]\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset\n\ndef data_augment(image, label):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement in the next function (below),\n    # this happens essentially for free on TPU. Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n    image = tf.image.random_flip_left_right(image)\n    #image = tf.image.random_saturation(image, 0, 2)\n    return image, label   \n\ndef get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nVALIDATION_STEPS = -(-NUM_VALIDATION_IMAGES // BATCH_SIZE) # The \"-(-//)\" trick rounds up instead of down :-)\nTEST_STEPS = -(-NUM_TEST_IMAGES // BATCH_SIZE)             # The \"-(-//)\" trick rounds up instead of down :-)\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{"id":"e9b9FXjpxYSV","trusted":true},"cell_type":"code","source":"# Peek at training data\ntraining_dataset = get_training_dataset()\ntraining_dataset = training_dataset.unbatch().batch(20)\ntrain_batch = iter(training_dataset)","execution_count":null,"outputs":[]},{"metadata":{"id":"xb-b4PRz-V6O","outputId":"93bda053-1da0-44b7-ffa1-d0f54e184c81","trusted":true},"cell_type":"code","source":"# run this cell again for next set of images\ndisplay_batch_of_images(next(train_batch))","execution_count":null,"outputs":[]},{"metadata":{"id":"ALtRUlxhw8Vt"},"cell_type":"markdown","source":"## Model"},{"metadata":{"id":"UqLr4cQlxYSa","outputId":"bb97f3ac-a042-41eb-8396-aba79c09e92b","trusted":true},"cell_type":"code","source":"%%time\nwith strategy.scope():\n    # this option makes loading models from TFHub directly work on TPU\n    load_locally = tf.saved_model.LoadOptions(experimental_io_device='/job:localhost')\n    pretrained_model = hub.KerasLayer('https://tfhub.dev/tensorflow/efficientnet/b6/feature-vector/1',\n                                      trainable=True, load_options=load_locally,\n                                      dtype=tf.float32) # float32 here so that mixed precision works    \n    model = tf.keras.Sequential([\n        # the expected image format for all TFHub image models is float32 in [0,1) range\n        tf.keras.layers.Lambda(lambda data: tf.image.convert_image_dtype(data, tf.float32), input_shape=[*IMAGE_SIZE, 3]),\n        pretrained_model,\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax', dtype=tf.float32)  # float32 here so that mixed precision works\n    ])\n        \n    model.compile(\n        optimizer='adam',\n        loss = 'sparse_categorical_crossentropy',\n        metrics=['sparse_categorical_accuracy'],\n        steps_per_execution=32\n    )\n    \n    model.summary()","execution_count":null,"outputs":[]},{"metadata":{"id":"dMfenMQcxAAb"},"cell_type":"markdown","source":"## Training"},{"metadata":{"id":"M-ID7vP5mIKs","outputId":"1403675f-3853-4aed-94ea-bcc1b68026e7","trusted":true},"cell_type":"code","source":"%%time\nlr_callback = tf.keras.callbacks.LearningRateScheduler(lr_fn)\nhistory = model.fit(get_training_dataset(), steps_per_epoch=STEPS_PER_EPOCH, epochs=EPOCHS,\n                    #validation_data=get_validation_dataset(), validation_steps=VALIDATION_STEPS,\n                    callbacks=[lr_callback])","execution_count":null,"outputs":[]},{"metadata":{"id":"VngeUBIdyJ1T","trusted":true},"cell_type":"code","source":"#display_training_curves(history.history['loss'], history.history['val_loss'], 'loss', 211)\n#display_training_curves(history.history['sparse_categorical_accuracy'], history.history['val_sparse_categorical_accuracy'], 'accuracy', 212)","execution_count":null,"outputs":[]},{"metadata":{"id":"MKFMWzh0Yxsq"},"cell_type":"markdown","source":"## Confusion matrix"},{"metadata":{"id":"2X07EKsqK_RW","outputId":"4d97852e-9734-4581-e274-55634feaebd8","trusted":true},"cell_type":"code","source":"cmdataset = get_validation_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and labels, order matters.\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch\nif strategy.num_replicas_in_sync > 8:\n    images_ds = images_ds.unbatch().batch(128) # bug on TPU v3-32, predictions won't work without this\ncm_probabilities = model.predict(images_ds, steps=VALIDATION_STEPS)\ncm_predictions = np.argmax(cm_probabilities, axis=-1)\nprint(\"Correct   labels: \", cm_correct_labels.shape, cm_correct_labels)\nprint(\"Predicted labels: \", cm_predictions.shape, cm_predictions)","execution_count":null,"outputs":[]},{"metadata":{"id":"qzCCDL1CZFx6","outputId":"acb343ab-a49f-4bd9-a040-9e824aa2b88a","trusted":true},"cell_type":"code","source":"cmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))\nscore = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\nprecision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\nrecall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\ncmat = (cmat.T / cmat.sum(axis=1)).T # normalized\ndisplay_confusion_matrix(cmat, score, precision, recall)\nprint('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))","execution_count":null,"outputs":[]},{"metadata":{"id":"mLaw59ynxYSl"},"cell_type":"markdown","source":"## Visual validation"},{"metadata":{"id":"x4sTeoUdxYSl","trusted":true},"cell_type":"code","source":"dataset = get_validation_dataset()\ndataset = dataset.unbatch().batch(20)\nbatch = iter(dataset)","execution_count":null,"outputs":[]},{"metadata":{"id":"UNj8aecJxYSn","outputId":"0244ef95-e9df-4408-e030-0bae104a1dc6","trusted":true},"cell_type":"code","source":"# run this cell again for next set of images\nimages, labels = next(batch)\nprobabilities = model.predict(images)\npredictions = np.argmax(probabilities, axis=-1)\ndisplay_batch_of_images((images, labels), predictions)","execution_count":null,"outputs":[]},{"metadata":{"id":"SVY1pBg5ydH-"},"cell_type":"markdown","source":"## License"},{"metadata":{"id":"hleIN5-pcr0N"},"cell_type":"markdown","source":"\n\n---\n\n\nauthor: Martin Gorner<br>\ntwitter: @martin_gorner\n\n\n---\n\n\nCopyright 2020 Google LLC\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n\n\n---\n\n\nThis is not an official Google product but sample code provided for an educational purpose\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}