{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":""},"kaggle":{"accelerator":"tpu1vmV38","dataSources":[{"sourceId":21154,"databundleVersionId":1243559,"sourceType":"competition"}],"dockerImageVersionId":31091,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Welcome\n### Welcome to a fun adventure with TPUs and flower classification 🌻🌸🌹.\nIn this fun notebook we will go step by step and create a deep learning model to perform flower classification on 104 different species!\n\n<div class=\"alert alert-block alert-info\"> 📌 If you find this notebook interesting, please upvote it, it means a lot to me and it keeps me motivated to improve it as shown in my long list of items below 😀</div><br>  \n\nThis notebook builds heavily on Ryan's awesome notebook as part of the Deep Learning Kaggle course on Computer Vision at: https://www.kaggle.com/ryanholbrook/create-your-first-submission\n\nI explore 25 combinations of transfer learning and hyperparameter tuning and compare their performance/val accuracy, where did the val loss and accuracy perform best, f1 score as well as precision and recall.\nLook out for my personal notes, labeled as Note ... 😀, as we take this journey together...\n\nMy Personal Plan of Action:\n- Understand how TPUs work and how to use them ✅\n- Explore transfer learning with 10+ models pretrained on either imagenet or noisy-student and evaluate their performance ✅\n- Explore training large CNN models from scratch and evaluate their performance ✅\n- Explore 10+ hyperparameter tuning methods and evaluate their performance ✅\n- Explore 25+ combinations of models and tuning methods above and evaluate their performance ✅\n- Ensemble models with loaded weights and evaluate their performance ✅\n- Build a great looking vizualization that captures and highlights model + tuning performance ✅\n- Be generous with comments, either as markdown or in code so anyone can follow along ✅\n- Respond to each comment on this notebook and learn from each other ✅\n- Meet and interact with kagglers online ✅\n- Reach out and thank other Kagglers for their amazing notebooks, share feedback and learn from each other ✅\n- Promote and share our cool online community [Deep Learning Adventures](https://www.meetup.com/Deep-Learning-Adventures) ✅\n- Have fun while building cool models ✅\n- Climb up the leaderboard with new explorations ✅\n- Plot model performance report in 3D 😎 ✅\n- Start and maintain [a discussion thread around this notebook](https://www.kaggle.com/c/tpu-getting-started/discussion/209865) ✅\n- Visualize different data augmentation methods ✅\n- Visualize incorrect predictions ✅\n- Explore Test Time Augmentation (TTA) ✅\n- Find and use more training data, increasing the variety our models are exposed to from 12K training images to 68K ✅\n- Convert a few models from .h5 to TensorFlow Lite using [TensorFlow Lite converter](https://www.tensorflow.org/lite/convert) ✅\n- Deploy a few models models on mobile and IoT devices\n- Explore other dimensions 224x224 images\n- Explore models from TF Hub or Model Garden\n- Explore Mixed precision\n- Explore ELI5 and model explainability\n- Read a paper or two on different data augmentation methods for computer vision\n- Earn 5+ votes/bronze medal for this notebook 🥉 ✅\n- Earn 20+ votes/silver medal for this notebook 🥈 ✅\n- Earn 50+ votes/gold medal for this notebook - your help is needed here 😀⬅️⬅️⬅️","metadata":{}},{"cell_type":"code","source":"from IPython.display import Image\nImage(filename=\"../input/images/Petals to the Metal 31.png\", width=1200, height=1000)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"Image(filename=\"../input/images/Petals to the Metal 32.png\", width=1200, height=1000)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"Image(filename=\"../input/images/Petals to the Metal 28.png\", width=1200, height=1000)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"Image(filename=\"../input/images/3d_plot3.png\", width=1200, height=1000)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"Image(filename=\"../input/images/3d_plot4.png\", width=1200, height=1000)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"Image(filename=\"../input/images/3d_plot5.png\", width=1200, height=1000)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"Image(filename=\"../input/images/Petals to the Metal 10.png\", width=1200, height=1000) ","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Weights for 12K images\nImage(filename=\"../input/images/Petals to the Metal 25.png\", width=1200, height=1000) ","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Weights for 70K images\nImage(filename=\"../input/images/Petals to the Metal 26.png\", width=1200, height=1000) ","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"Image(filename=\"../input/images/Petals to the Metal 22.png\", width=1200, height=1000) ","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"If you would like to learn more as well as join a larger data science community 🎉, feel free to join us at: https://www.meetup.com/Deep-Learning-Adventures/events/275438349\nAll our sessions are recorded 😃 and available on YouTube at: http://bit.ly/dla-kaggle-courses","metadata":{}},{"cell_type":"code","source":"Image(filename=\"../input/images/Deep Learning Adventures.png\", width=1200, height=1000) ","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n# Introduction #\n\nWelcome to the [**Petals to the Metal**](https://www.kaggle.com/c/tpu-getting-started) competition! In this competition, you’re challenged to build a machine learning model to classify 104 types of flowers based on their images.\n\nIn this tutorial notebook, you'll learn how to build an image classifier in Keras and train it on a [Tensor Processing Unit (TPU)](https://www.kaggle.com/docs/tpu). At the end, you'll have a complete project you can build off of with ideas of your own.\n\n<blockquote style=\"margin-right:auto; margin-left:auto; background-color: #ebf9ff; padding: 1em; margin:24px;\">\n    <strong>Fork This Notebook!</strong><br>\nCreate your own editable copy of this notebook by clicking on the <strong>Copy and Edit</strong> button in the top right corner.\n</blockquote>","metadata":{}},{"cell_type":"markdown","source":"## Note 0 😀\n### Upgrade TensorFlow for improved performance and ability to transfer learn from more models in tf.keras.applications\n### ⚠️ Issue: Code will need to be refactored due to this upgrade ⚠️","metadata":{}},{"cell_type":"code","source":"#!pip install -U tensorflow==2.4.0","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Step 1: Imports #\n\nWe begin by importing several Python packages.","metadata":{}},{"cell_type":"code","source":"!pip install seaborn --upgrade\nimport seaborn as sns\n\nimport matplotlib.pyplot as plt\nfrom matplotlib import cm\nimport math, re, os\nimport pandas as pd\nimport numpy as np\nimport random\nimport plotly.express as px\n\nimport tensorflow as tf\nprint(\"Tensorflow version \" + tf.__version__)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Step 2: Distribution Strategy #\n\nA TPU has eight different *cores* and each of these cores acts as its own accelerator. (A TPU is sort of like having eight GPUs in one machine.) We tell TensorFlow how to make use of all these cores at once through a **distribution strategy**. Run the following cell to create the distribution strategy that we'll later apply to our model.","metadata":{}},{"cell_type":"markdown","source":"## Note 1 😀\n### 1. TPUs are network-connected accelerators and you must first locate them on the network. This is what TPUClusterResolver() does.\n\n### 2. Two additional lines of boilerplate and you can define a TPUStrategy. This object contains the necessary distributed training code that will work on TPUs with their 8 compute cores (see hardware section below).\n\n### 3. Finally, you use the TPUStrategy by instantiating your model in the scope of the strategy. This creates the model on the TPU. Model size is constrained by the TPU RAM only, not by the amount of memory available on the VM running your Python code. Model creation and model training use the usual Keras APIs.\n\n### Source: https://www.kaggle.com/docs/tpu","metadata":{}},{"cell_type":"code","source":"# Detect TPU, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() #See Note 1.1 above 😀\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu) #See Note 1.2 above 😀\n    tf.tpu.experimental.initialize_tpu_system(tpu) #See Note 1.2 above 😀\n    strategy = tf.distribute.experimental.TPUStrategy(tpu) #See Note 1.3 above 😀\nelse:\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We'll use the distribution strategy when we create our neural network model. Then, TensorFlow will distribute the training among the eight TPU cores by creating eight different *replicas* of the model, one for each core.\n\n# Step 3: Loading the Competition Data #\n\n## Get GCS Path ##\n\nWhen used with TPUs, datasets need to be stored in a [Google Cloud Storage bucket](https://cloud.google.com/storage/). You can use data from any public GCS bucket by giving its path just like you would data from `'/kaggle/input'`. The following will retrieve the GCS path for this competition's dataset.","metadata":{}},{"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\n\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('tpu-getting-started')\nprint(GCS_DS_PATH) # what do gcs paths look like?","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"You can use data from any public dataset here on Kaggle in just the same way. If you'd like to use data from one of your private datasets, see [here](https://www.kaggle.com/docs/tpu#tpu3pt5).","metadata":{}},{"cell_type":"markdown","source":"## Load Data ##\n\nWhen used with TPUs, datasets are often serialized into [TFRecords](https://www.kaggle.com/ryanholbrook/tfrecords-basics). This is a format convenient for distributing data to each of the TPUs cores. We've hidden the cell that reads the TFRecords for our dataset since the process is a bit long. You could come back to it later for some guidance on using your own datasets with TPUs.","metadata":{}},{"cell_type":"markdown","source":"## Note 2 😀\n### 1. TPUs are equipped with 128GB of high-speed memory allowing larger batches, larger models and also larger training inputs. In the code below, we can use 512x512 px input images, also provided in the dataset, and see the TPU v3-8 handle them easily.\n### 2. num_parallel_reads=AUTO instructs the API to read from multiple files if available. It figures out how many automatically.\n### 3. experimental_deterministic = False disables data order enforcement. We will be shuffling the data anyway so order is not important. With this setting the API can use any TFRecord as soon as it is streamed in.\n### Source: https://www.kaggle.com/docs/tpu","metadata":{}},{"cell_type":"code","source":"IMAGE_SIZE = [512, 512] #See Note 2.1 above 😀\n\nGCS_PATH = GCS_DS_PATH + '/tfrecords-jpeg-512x512'\nAUTO = tf.data.experimental.AUTOTUNE #See Note 2.2 above 😀\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') \n\nCLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 103\n\n\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed. #See Note 2.3 above 😀\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files. #See Note 2.2 above 😀\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Note 9 😀\n## Use additional data, tuning6, private dataset\n## Inspired by Dmitry's notebook [here](https://www.kaggle.com/dmitrynokhrin/densenet201-aug-additional-data) and Araik's notebook [here](https://www.kaggle.com/atamazian/fc-ensemble-external-data-effnet-densenet)\n## See also [external data and how to use them](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/140866) and [Kirill's tf_flower_photo_tfrec dataset](https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec)","metadata":{}},{"cell_type":"code","source":"GCS_DS_PATH_EXT = KaggleDatasets().get_gcs_path('tf-flower-photo-tfrec')\n\n# External data\nGCS_PATH_SELECT_EXT = {\n    192: '/tfrecords-jpeg-192x192',\n    224: '/tfrecords-jpeg-224x224',\n    331: '/tfrecords-jpeg-331x331',\n    512: '/tfrecords-jpeg-512x512'\n}\nGCS_PATH_EXT = GCS_PATH_SELECT_EXT[IMAGE_SIZE[0]]\n\nIMAGENET_FILES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/imagenet' + GCS_PATH_EXT + '/*.tfrec')\nINATURELIST_FILES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/inaturalist' + GCS_PATH_EXT + '/*.tfrec')\nOPENIMAGE_FILES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/openimage' + GCS_PATH_EXT + '/*.tfrec')\nOXFORD_FILES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/oxford_102' + GCS_PATH_EXT + '/*.tfrec')\nTENSORFLOW_FILES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/tf_flowers' + GCS_PATH_EXT + '/*.tfrec')\n\nADDITIONAL_TRAINING_FILENAMES = IMAGENET_FILES + INATURELIST_FILES + OPENIMAGE_FILES + OXFORD_FILES + TENSORFLOW_FILES  \n\nTRAINING_FILENAMES = TRAINING_FILENAMES + ADDITIONAL_TRAINING_FILENAMES","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Note 8 😀\n## Perform data augmentation, tuning4\n## Inspired by Dmitry's notebook [here](https://www.kaggle.com/dmitrynokhrin/densenet201-aug-additional-data)","metadata":{}},{"cell_type":"code","source":"#tuning4\nSEED = 2020\n\ndef random_blockout(img, sl=0.1, sh=0.2, rl=0.4):\n    p=random.random()\n    if p>=0.25:\n        w, h, c = IMAGE_SIZE[0], IMAGE_SIZE[1], 3\n        origin_area = tf.cast(h*w, tf.float32)\n\n        e_size_l = tf.cast(tf.round(tf.sqrt(origin_area * sl * rl)), tf.int32)\n        e_size_h = tf.cast(tf.round(tf.sqrt(origin_area * sh / rl)), tf.int32)\n\n        e_height_h = tf.minimum(e_size_h, h)\n        e_width_h = tf.minimum(e_size_h, w)\n\n        erase_height = tf.random.uniform(shape=[], minval=e_size_l, maxval=e_height_h, dtype=tf.int32)\n        erase_width = tf.random.uniform(shape=[], minval=e_size_l, maxval=e_width_h, dtype=tf.int32)\n\n        erase_area = tf.zeros(shape=[erase_height, erase_width, c])\n        erase_area = tf.cast(erase_area, tf.uint8)\n\n        pad_h = h - erase_height\n        pad_top = tf.random.uniform(shape=[], minval=0, maxval=pad_h, dtype=tf.int32)\n        pad_bottom = pad_h - pad_top\n\n        pad_w = w - erase_width\n        pad_left = tf.random.uniform(shape=[], minval=0, maxval=pad_w, dtype=tf.int32)\n        pad_right = pad_w - pad_left\n\n        erase_mask = tf.pad([erase_area], [[0,0],[pad_top, pad_bottom], [pad_left, pad_right], [0,0]], constant_values=1)\n        erase_mask = tf.squeeze(erase_mask, axis=0)\n        erased_img = tf.multiply(tf.cast(img,tf.float32), tf.cast(erase_mask, tf.float32))\n\n        return tf.cast(erased_img, img.dtype)\n    else:\n        return tf.cast(img, img.dtype)\n\n    \ndef data_augment_v2(image, label):\n    # Thanks to the dataset.prefetch(AUTO) statement in the next function (below), this happens essentially for free on TPU. \n    # Data pipeline code is executed on the \"CPU\" part of the TPU while the TPU itself is computing gradients.\n    \n    flag = random.randint(1,3)\n    coef_1 = random.randint(70, 90) * 0.01\n    coef_2 = random.randint(70, 90) * 0.01\n    \n    if flag == 1:\n        image = tf.image.random_flip_left_right(image, seed=SEED)\n    elif flag == 2:\n        image = tf.image.random_flip_up_down(image, seed=SEED)\n    else:\n        image = tf.image.random_crop(image, [int(IMAGE_SIZE[0]*coef_1), int(IMAGE_SIZE[0]*coef_2), 3],seed=SEED)\n        \n    image = random_blockout(image)\n    \n    return image, label ","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Perform data augmentation, tuning7\n## Inspired by Xuanzhi Huang and Rahul Paul's notebook [here](https://www.kaggle.com/xuanzhihuang/flower-classification-densenet-201)","metadata":{}},{"cell_type":"code","source":"import tensorflow_addons as tfa\n\n# Randomly make some changes to the images and return the new images and labels\ndef data_augment_v3(image, label):\n        \n    # Set seed for data augmentation\n    seed = 100\n    \n    # Randomly resize and then crop images\n    image = tf.image.resize(image, [720, 720])\n    image = tf.image.random_crop(image, [512, 512, 3], seed = seed)\n\n    # Randomly reset brightness of images\n    image = tf.image.random_brightness(image, 0.6, seed = seed)\n    \n    # Randomly reset saturation of images\n    image = tf.image.random_saturation(image, 3, 5, seed = seed)\n        \n    # Randomly reset contrast of images\n    image = tf.image.random_contrast(image, 0.3, 0.5, seed = seed)\n\n    # Randomly reset hue of images, but this will make the colors really weird, which we think will not happen\n    # in common photography\n    # image = tf.image.random_hue(image, 0.5, seed = seed)\n    \n    # Blur images\n    image = tfa.image.mean_filter2d(image, filter_shape = 10)\n    \n    # Randomly flip images\n    image = tf.image.random_flip_left_right(image, seed = seed)\n    image = tf.image.random_flip_up_down(image, seed = seed)\n    \n    # Fail to rotate and transform images due to some bug in TensorFlow\n    # angle = random.randint(0, 180)\n    # image = tfa.image.rotate(image, tf.constant(np.pi * angle / 180))\n    # image = tfa.image.transform(image, [1.0, 1.0, -250, 0.0, 1.0, 0.0, 0.0, 0.0])\n    \n    return image, label","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Create Data Pipelines ##\n\nIn this final step we'll use the `tf.data` API to define an efficient data pipeline for each of the training, validation, and test splits.","metadata":{}},{"cell_type":"code","source":"def data_augment(image, label):\n    # Thanks to the dataset.prefetch(AUTO) statement in the next function (below), this happens essentially for free on TPU. \n    # Data pipeline code is executed on the \"CPU\" part of the TPU while the TPU itself is computing gradients.\n    image = tf.image.random_flip_left_right(image)\n    #image = tf.image.random_saturation(image, 0, 2)\n    return image, label   \n\ndef get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO) #tuning4\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))\n","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Original Data: Dataset: 12753 training images, 3712 validation images, 7382 unlabeled test images\n### Additional Data: Dataset: 68094 training images, 3712 validation images, 7382 unlabeled test images","metadata":{}},{"cell_type":"markdown","source":"This next cell will create the datasets that we'll use with Keras during training and inference. Notice how we scale the size of the batches to the number of TPU cores.","metadata":{}},{"cell_type":"markdown","source":"## Note 3 😀\n### 1. To go fast on a TPU, increase the batch size. The rule of thumb is to use batches of 128 elements per core (ex: batch size of 128*8=1024 for a TPU with 8 cores). At this size, the 128x128 hardware matrix multipliers of the TPU (see hardware section below) are most likely to be kept busy. You start seeing interesting speedups from a batch size of 8 per core though. In the sample above, the batch size is scaled with the core count through this line of code:\n### BATCH_SIZE = 16 * tpu_strategy.num_replicas_in_sync\n### Source: https://www.kaggle.com/docs/tpu","metadata":{}},{"cell_type":"code","source":"strategy.num_replicas_in_sync","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"16 * strategy.num_replicas_in_sync","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Define the batch size. This will be 16 with TPU off and 128 (=16*8) with TPU on\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync #See Note 3.1 above 😀\n\nds_train = get_training_dataset()\nds_valid = get_validation_dataset()\nds_test = get_test_dataset()\n\nprint(\"Training:\", ds_train)\nprint (\"Validation:\", ds_valid)\nprint(\"Test:\", ds_test)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"These datasets are `tf.data.Dataset` objects. You can think about a dataset in TensorFlow as a *stream* of data records. The training and validation sets are streams of `(image, label)` pairs.","metadata":{}},{"cell_type":"code","source":"np.set_printoptions(threshold=15, linewidth=80)\n\nprint(\"Training data shapes:\")\nfor image, label in ds_train.take(3):\n    print(image.numpy().shape, label.numpy().shape) #See Note 3.1 above 😀\nprint(\"Training data label examples:\", label.numpy())","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The test set is a stream of `(image, idnum)` pairs; `idnum` here is the unique identifier given to the image that we'll use later when we make our submission as a `csv` file.","metadata":{}},{"cell_type":"code","source":"print(\"Test data shapes:\")\nfor image, idnum in ds_test.take(3):\n    print(image.numpy().shape, idnum.numpy().shape) #See Note 3.1 above 😀\nprint(\"Test data IDs:\", idnum.numpy().astype('U')) # U=unicode string","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Step 4: Explore Data #\n\nLet's take a moment to look at some of the images in the dataset.","metadata":{}},{"cell_type":"code","source":"from matplotlib import pyplot as plt\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case,these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n        # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], \n                                'OK' if correct else 'NO', \n                                u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch, predictions=None, display_mismatches_only=False):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        if display_mismatches_only:\n            if predictions[i] != label:\n                subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n        else:        \n            subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n\n\ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])\n\ndef display_training_curves_v2(training, validation, learning_rate_list, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title, color='b')\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.', 'learning rate'])        \n    \n    ax2 = ax.twinx()\n    ax2.plot(learning_rate_list, 'g-')\n    ax2.set_ylabel('learning rate', color='g')","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"You can display a single batch of images from a dataset with another of our helper functions. The next cell will turn the dataset into an iterator of batches of 20 images.","metadata":{}},{"cell_type":"code","source":"ds_iter = iter(ds_train.unbatch().batch(20))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Use the Python `next` function to pop out the next batch in the stream and display it with the helper function.","metadata":{}},{"cell_type":"code","source":"one_batch = next(ds_iter)\ndisplay_batch_of_images(one_batch)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"By defining `ds_iter` and `one_batch` in separate cells, you only need to rerun the cell above to see a new batch of images.","metadata":{}},{"cell_type":"markdown","source":"## tuning7, show a sample of data augmented","metadata":{}},{"cell_type":"code","source":"row = 3\ncol = 4\nall_elements = get_training_dataset().unbatch()\none_element = tf.data.Dataset.from_tensors(next(iter(all_elements)))\n# Map the images to the data augmentation function for image processing\naugmented_element = one_element.repeat().map(data_augment).batch(row * col)\n\nfor (img, label) in augmented_element:\n    plt.figure(figsize = (15, int(15 * row / col)))\n    for j in range(row * col):\n        plt.subplot(row, col, j + 1)\n        plt.axis('off')\n        plt.imshow(img[j, ])\n    plt.show()\n    break","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## tuning7, show a sample of data augmented v2","metadata":{}},{"cell_type":"code","source":"# Map the images to the data augmentation function for image processing\naugmented_element = one_element.repeat().map(data_augment_v2).batch(row * col)\n\nfor (img, label) in augmented_element:\n    plt.figure(figsize = (15, int(15 * row / col)))\n    for j in range(row * col):\n        plt.subplot(row, col, j + 1)\n        plt.axis('off')\n        plt.imshow(img[j, ])\n    plt.show()\n    break","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## tuning7, show a sample of data augmented v3","metadata":{}},{"cell_type":"code","source":"# Map the images to the data augmentation function for image processing\naugmented_element = one_element.repeat().map(data_augment_v3).batch(row * col)\n\nfor (img, label) in augmented_element:\n    plt.figure(figsize = (15, int(15 * row / col)))\n    for j in range(row * col):\n        plt.subplot(row, col, j + 1)\n        plt.axis('off')\n        plt.imshow(img[j, ])\n    plt.show()\n    break","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Step 5: Define Model #\n\nNow we're ready to create a neural network for classifying images! We'll use what's known as **transfer learning**. With transfer learning, you reuse part of a pretrained model to get a head-start on a new dataset.\n\nFor this tutorial, we'll to use a model called **VGG16** pretrained on [ImageNet](http://image-net.org/)). Later, you might want to experiment with [other models](https://www.tensorflow.org/api_docs/python/tf/keras/applications) included with Keras. ([Xception](https://www.tensorflow.org/api_docs/python/tf/keras/applications/Xception) wouldn't be a bad choice.)\n\nThe distribution strategy we created earlier contains a [context manager](https://docs.python.org/3/reference/compound_stmts.html#with), `strategy.scope`. This context manager tells TensorFlow how to divide the work of training among the eight TPU cores. When using TensorFlow with a TPU, it's important to define your model in a `strategy.scope()` context.","metadata":{}},{"cell_type":"code","source":"[*IMAGE_SIZE, 3]","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Note 4 😀\n## Let's transfer learn from different neural network architectures from tf.keras.applications and keep track of their performance\n## Source: https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications\n## Note that TF 2.4 has many more models available for transfer learning\n## Source: https://www.tensorflow.org/api_docs/python/tf/keras/applications","metadata":{}},{"cell_type":"code","source":"', '.join(tf.keras.applications.__dir__())","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Note 5 😀\n## Use ModelCheckpoint to keep track of the \"best\" model during training, according to ```monitor='val_loss'```","metadata":{}},{"cell_type":"code","source":"# Model weights are saved at the end of every epoch, if it's the best seen so far during model.fit\ncheckpoint_filepath = \"Petals_to_the_Metal-70K_images-trainable_True-MobileNetV2.h5\" #\"Petals_to_the_Metal-70K_images-trainable_True-DenseNet201.h5\"\n\ncheckpoint = tf.keras.callbacks.ModelCheckpoint(\n    filepath=checkpoint_filepath,\n    save_weights_only=True,\n    monitor='val_loss',\n    mode='min',\n    save_best_only=True\n)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Note 6 😀\n## Use EarlyStopping to stop the training when there is no improvement in ```monitor='val_loss'``` for ```patience=3``` consecutive epochs","metadata":{}},{"cell_type":"code","source":"# This callback will stop the training when there is no improvement in the validation loss for three consecutive epochs. \nearly_stopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Note 10 😀\n## [Track learning rate during training](https://stackoverflow.com/questions/49127214/keras-how-to-output-learning-rate-onto-tensorboard)\nNotFoundError: Container worker does not exist. (Could not find resource: worker/_AnonymousVar8064)\n\tEncountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors.","metadata":{}},{"cell_type":"code","source":"NotFoundError = \"\"\"\nclass LRTensorBoard(TensorBoard):\n    def __init__(self, log_dir, **kwargs):  # add other arguments to __init__ if you need\n        super().__init__(log_dir=log_dir, **kwargs)\n\n    def on_epoch_end(self, epoch, logs=None):\n        logs = logs or {}\n        logs.update({'lr': K.eval(self.model.optimizer.lr)})\n        super().on_epoch_end(epoch, logs)\n\nlr_tracking = LRTensorBoard(log_dir=\"./lr_tracking\")\n\"\"\"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## [Writing your own callbacks](https://www.tensorflow.org/guide/keras/custom_callback)\n## Not needed","metadata":{}},{"cell_type":"code","source":"class LearningRateTracking(tf.keras.callbacks.Callback):\n    def on_epoch_end(self, epoch, logs=None):\n        keys = list(logs.keys())\n        print(\"End epoch {} of training; got log keys: {}\".format(epoch, keys))\n        \n        #logs = logs or {}\n        #logs.update({'lr': K.eval(self.model.optimizer.lr)}) #optimizer._decayed_lr('float32').numpy()\n        #return \n\n#lr_tracking = LearningRateTracking()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"use_efficientnet = False #tuning9\nif use_efficientnet:\n    !pip install -q efficientnet\n    from efficientnet.tfkeras import EfficientNetB7","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Calculate weight for each class #tuning11\n### Inspired by [Flower Classification DenseNet 201](https://www.kaggle.com/xuanzhihuang/flower-classification-densenet-201)","metadata":{}},{"cell_type":"code","source":"weight_per_class = True\n\nif weight_per_class:\n    from collections import Counter\n    import gc\n\n    gc.enable()\n\n    def get_training_dataset_raw():\n        dataset = load_dataset(TRAINING_FILENAMES, labeled = True, ordered = False)\n        return dataset\n\n    raw_training_dataset = get_training_dataset_raw()\n\n    label_counter = Counter()\n    for images, labels in raw_training_dataset:\n        label_counter.update([labels.numpy()])\n\n    del raw_training_dataset    \n\n    TARGET_NUM_PER_CLASS = 122 #?\n\n    def get_weight_for_class(class_id):\n        counting = label_counter[class_id]\n        weight = TARGET_NUM_PER_CLASS / counting\n        return weight\n\n    weight_per_class = {class_id: get_weight_for_class(class_id) for class_id in range(104)}","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if weight_per_class:\n    data = pd.DataFrame.from_dict(weight_per_class, orient='index', columns=['class_weight'])\n    plt.figure(figsize=(30, 9))\n\n    #barplot color based on value\n    bplot = sns.barplot(x=data.index, y='class_weight', data=data, palette= cm.Blues(data['class_weight']*0.15));\n    for p in bplot.patches:\n        bplot.annotate(format(p.get_height(), '.1f'), \n                       (p.get_x() + p.get_width() / 2., p.get_height()), \n                       ha = 'center', va = 'center', \n                       xytext = (0, 9), \n                       textcoords = 'offset points')\n    plt.xlabel(\"Class\", size=14)\n    plt.ylabel(\"Class weight (inverse of %)\", size=14)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Skip to Model Ensemble/Note 11 😀 or look at previous versions of this notebook for transfer learning or end to end training","metadata":{}},{"cell_type":"code","source":"using_ensemble_models = False","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if not using_ensemble_models:\n    with strategy.scope():\n        #pretrained_model = tf.keras.applications.VGG16\n        #pretrained_model = tf.keras.applications.DenseNet201\n        #pretrained_model = tf.keras.applications.InceptionResNetV2\n        #pretrained_model = tf.keras.applications.InceptionV3\n        #pretrained_model = tf.keras.applications.MobileNet\n        #pretrained_model = tf.keras.applications.MobileNetV2\n        #pretrained_model = tf.keras.applications.NASNetMobile\n        #pretrained_model = tf.keras.applications.ResNet50\n        #pretrained_model = tf.keras.applications.ResNet101V2\n        #pretrained_model = tf.keras.applications.VGG19\n        #pretrained_model = tf.keras.applications.Xception\n        #pretrained_model = tf.keras.applications.DenseNet201 \n        #pretrained_model = EfficientNetB7\n\n        pretrained_model = tf.keras.applications.MobileNetV2(\n            include_top=False ,\n            weights='imagenet', #tuning10 weights='noisy-student' instead of 'imagenet'\n                                #Self-training with Noisy Student improves ImageNet classification https://arxiv.org/abs/1911.04252) \n            #pooling='avg', #tuning1\n            input_shape=[*IMAGE_SIZE, 3]\n        )\n\n        pretrained_model.trainable = True #tuning8 pretrained_model.trainable = True\n\n        model = tf.keras.Sequential([\n            pretrained_model, #Base pretrained on ImageNet to extract features from images\n\n            tf.keras.layers.GlobalAveragePooling2D(), ##Attach a new head to act as a classifier\n            #tf.keras.layers.Dropout(0.3), #tuning3\n            tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n        ])","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The `'sparse_categorical'` versions of the loss and metrics are appropriate for a classification task with more than two labels, like this one.","metadata":{}},{"cell_type":"code","source":"if not using_ensemble_models:\n    model.compile(\n        optimizer='nadam', #tuning2 optimizer='nadam',\n        loss = 'sparse_categorical_crossentropy',\n        metrics=['sparse_categorical_accuracy'],\n    )","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if not using_ensemble_models:\n    model.summary()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#if not using_ensemble_models:\ntf.keras.utils.plot_model(model, show_shapes=True)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Step 6: Training #\n\n## Learning Rate Schedule ##\n\nWe'll train this network with a special learning rate schedule.","metadata":{}},{"cell_type":"code","source":"if not using_ensemble_models:\n    # Define training epochs\n    EPOCHS = 30\n\n    # Define the batch size. This will be 16 with TPU off and 128 (=16*8) with TPU on\n    BATCH_SIZE = 16 * strategy.num_replicas_in_sync #See Note 3.1 above 😀\n\n    STEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if not using_ensemble_models:\n    # Learning Rate Schedule for Fine Tuning #\n    def exponential_lr(epoch,\n                       start_lr = 0.00001, min_lr = 0.00001, max_lr = 0.00005 * strategy.num_replicas_in_sync, #tuning1\n                       rampup_epochs = 5, sustain_epochs = 0,\n                       exp_decay = 0.75): #tuning1\n\n        def lr(epoch, start_lr, min_lr, max_lr, rampup_epochs, sustain_epochs, exp_decay):\n            # linear increase from start to rampup_epochs\n            if epoch < rampup_epochs:\n                lr = ((max_lr - start_lr) /\n                      rampup_epochs * epoch + start_lr)\n            # constant max_lr during sustain_epochs\n            elif epoch < rampup_epochs + sustain_epochs:\n                lr = max_lr\n            # exponential decay towards min_lr\n            else:\n                lr = ((max_lr - min_lr) *\n                      exp_decay**(epoch - rampup_epochs - sustain_epochs) +\n                      min_lr)\n            return lr\n        return lr(epoch,\n                  start_lr,\n                  min_lr,\n                  max_lr,\n                  rampup_epochs,\n                  sustain_epochs,\n                  exp_decay)\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(exponential_lr, verbose=True)\n\n    rng = [i for i in range(EPOCHS)]\n    y = [exponential_lr(x) for x in rng]\n    plt.plot(rng, y)\n    print(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Fit Model ##\n\nAnd now we're ready to train the model. After defining a few parameters, we're good to go!","metadata":{}},{"cell_type":"code","source":"if not using_ensemble_models:\n    history = model.fit(\n        ds_train,\n        validation_data=ds_valid,\n        epochs=EPOCHS,\n        steps_per_epoch=STEPS_PER_EPOCH,\n        callbacks=[lr_callback, checkpoint], # Model weights are saved at the end of every epoch, if it's the best seen so far\n        #workers = 3 #tuning5 https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras\n        class_weight = weight_per_class #tuning11\n    )","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This next cell shows how the loss and metrics progressed during training. Thankfully, it converges!","metadata":{}},{"cell_type":"markdown","source":"## Model performance","metadata":{}},{"cell_type":"code","source":"if not using_ensemble_models:\n    display_training_curves_v2( \n        history.history['loss'],\n        history.history['val_loss'],\n        history.history['lr'],\n        'loss',\n        211,\n    )\n\n    display_training_curves_v2(\n        history.history['sparse_categorical_accuracy'],\n        history.history['val_sparse_categorical_accuracy'],\n        history.history['lr'],\n        'accuracy',\n        212,\n    )","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"zoom_after = 20\nif not using_ensemble_models:\n    display_training_curves(\n        history.history['loss'][zoom_after:],\n        history.history['val_loss'][zoom_after:],\n        'loss',\n        211,\n    )\n\n    display_training_curves(\n        history.history['sparse_categorical_accuracy'][zoom_after:],\n        history.history['val_sparse_categorical_accuracy'][zoom_after:],\n        'accuracy',\n        212,\n    )","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Note 5 continued 😀\n## The model weights (that are considered the best) are loaded into the model.","metadata":{}},{"cell_type":"code","source":"checkpoint_filepath","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if not using_ensemble_models:\n    model.load_weights(checkpoint_filepath)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Note 12 😀\n## Convert a few models from .h5 to TensorFlow Lite using [TensorFlow Lite converter](https://www.tensorflow.org/lite/convert)\n## Deploy a few models models on mobile and IoT devices","metadata":{}},{"cell_type":"code","source":"model.summary()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(checkpoint_filepath)\ntflite_model_name = checkpoint_filepath.replace('.h5', '.tflite')\ntflite_model_name","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Convert the model\nconverter = tf.lite.TFLiteConverter.from_keras_model(model)\ntflite_model = converter.convert()\n\n# Save the model\nwith open(tflite_model_name, 'wb') as f:\n    f.write(tflite_model)\n    \nprint('TFLiteConversion completed successfully \\U0001F680')  ","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Note 11 😀\n## Use an ensemble of top 2 performers EfficientNetB7 and DenseNet201\n## See [my model dataset](https://www.kaggle.com/georgezoto/models) as well as [Dmitry's notebook](https://www.kaggle.com/dmitrynokhrin/start-with-ensemble-v2) and [Rosa's original notebook](https://www.kaggle.com/wrrosa/tpu-enet-b7-densenet)","metadata":{}},{"cell_type":"code","source":"def get_pretrained_model(model_name, image_dataset_weights, trainable=True):\n    pretrained_model= model_name(\n        include_top=False ,\n        weights=image_dataset_weights, #tuning10 weights='noisy-student' instead of 'imagenet'\n                                       #Self-training with Noisy Student improves ImageNet classification https://arxiv.org/abs/1911.04252) \n        input_shape=[*IMAGE_SIZE, 3]\n    )\n\n    pretrained_model.trainable = trainable #tuning8 pretrained_model.trainable = True\n    \n    model = tf.keras.Sequential([\n        pretrained_model, \n        tf.keras.layers.GlobalAveragePooling2D(), \n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n    \n    return model","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if using_ensemble_models:\n    with strategy.scope():\n        model_EB7 = get_pretrained_model(EfficientNetB7, 'noisy-student', trainable=True)\n\n    model_EB7.load_weights('../input/models/Petals_to_the_Metal-70K_images-trainable_True-EfficientNetB7.h5')    ","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if using_ensemble_models:\n    model_EB7.summary()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if using_ensemble_models:\n    with strategy.scope():\n        model_D201 = get_pretrained_model(tf.keras.applications.DenseNet201, 'imagenet', trainable=True)\n\n    model_D201.load_weights('../input/models/Petals_to_the_Metal-70K_images-trainable_True-DenseNet201.h5')  ","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if using_ensemble_models:\n    model_D201.summary()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Ensemble both models","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if using_ensemble_models:\n    cmdataset = get_validation_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and labels, order matters.\n    images_ds = cmdataset.map(lambda image, label: image)\n    labels_ds = cmdataset.map(lambda image, label: label).unbatch()\n    cm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch\n\n    m1 = model_EB7.predict(images_ds)\n    m2 = model_D201.predict(images_ds)\n\n    scores = []\n    for alpha in np.linspace(0,1,100):\n        cm_probabilities = alpha*m1+(1-alpha)*m2\n        cm_predictions = np.argmax(cm_probabilities, axis=-1)\n        scores.append(f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro'))\n\n    print(\"Correct   labels: \", cm_correct_labels.shape, cm_correct_labels)\n    print(\"Predicted labels: \", cm_predictions.shape, cm_predictions)\n    plt.plot(scores)\n\n    best_alpha = np.argmax(scores)/100\n    cm_probabilities = best_alpha*m1+(1-best_alpha)*m2\n    cm_predictions = np.argmax(cm_probabilities, axis=-1)\n\n    #best_alpha = 0.35","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if using_ensemble_models:\n    print(best_alpha, max(scores))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Run predictions on the test dataset","metadata":{}},{"cell_type":"code","source":"if using_ensemble_models:\n    test_ds = get_test_dataset(ordered=True)\n    #best_alpha = 0.35\n\n    print('Computing predictions...')\n    test_images_ds = test_ds.map(lambda image, idnum: image)\n    probabilities1 = model_EB7.predict(test_images_ds)\n    probabilities2 = model_D201.predict(test_images_ds)\n\n    probabilities = best_alpha * probabilities1 + (1 - best_alpha) * probabilities2\n\n    predictions = np.argmax(probabilities, axis=-1)\n    print(predictions)\n\n    print('Generating submission.csv file...')\n    # Get image ids from test set and convert to unicode\n    test_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\n    test_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n\n    # Write the submission file\n    np.savetxt(\n        'submission.csv',\n        np.rec.fromarrays([test_ids, predictions]),\n        fmt=['%s', '%d'],\n        delimiter=',',\n        header='id,label',\n        comments='',\n    )\n\n    # Look at the first few predictions\n    !head submission.csv","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Step 7: Evaluate Predictions #\n\nBefore making your final predictions on the test set, it's a good idea to evaluate your model's predictions on the validation set. This can help you diagnose problems in training or suggest ways your model could be improved. We'll look at two common ways of validation: plotting the **confusion matrix** and **visual validation**.","metadata":{}},{"cell_type":"code","source":"def display_confusion_matrix(cmat, score, precision, recall):\n    plt.figure(figsize=(25,25))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    \n    if not using_ensemble_models:\n        print('Epoch with min loss and max accuracy:', np.argmin(history.history['val_loss']), np.argmax(history.history['val_sparse_categorical_accuracy']))\n        print('min loss and max accuracy:', round(min(history.history['val_loss']),2), round(max(history.history['val_sparse_categorical_accuracy']),2))\n\n    print(titlestring.replace('\\n', ''))\n    plt.show()\n    \ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Confusion Matrix ##\n\nA [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) shows the actual class of an image tabulated against its predicted class. It is one of the best tools you have for evaluating the performance of a classifier.\n\nThe following cell does some processing on the validation data and then creates the matrix with the `confusion_matrix` function included in [`scikit-learn`](https://scikit-learn.org/stable/index.html).","metadata":{}},{"cell_type":"code","source":"cmdataset = get_validation_dataset(ordered=True)\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\n\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy()\n\nif using_ensemble_models:\n    print('using_ensemble_models')\n    probabilities1 = model_EB7.predict(images_ds)\n    probabilities2 = model_D201.predict(images_ds)\n    cm_probabilities = best_alpha * probabilities1 + (1 - best_alpha) * probabilities2\nelse:\n    cm_probabilities = model.predict(images_ds)\n    \ncm_predictions = np.argmax(cm_probabilities, axis=-1)\n\nlabels = range(len(CLASSES))\ncmat = confusion_matrix(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n)\ncmat = (cmat.T / cmat.sum(axis=1)).T # normalize","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"cmat","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"You might be familiar with metrics like [F1-score](https://en.wikipedia.org/wiki/F1_score) or [precision and recall](https://en.wikipedia.org/wiki/Precision_and_recall). This cell will compute these metrics and display them with a plot of the confusion matrix. (These metrics are defined in the Scikit-learn module `sklearn.metrics`; we've imported them in the helper script for you.)","metadata":{}},{"cell_type":"code","source":"score = f1_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\n\nprecision = precision_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\n\nrecall = recall_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\n\ndisplay_confusion_matrix(cmat, score, precision, recall)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Note 7 😀\n\n## Model comparison\n## [TF version: 2.2.0](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications)\n\n## [VGG16](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/vgg16) \n    12 epochs, no data augmentation\n    Epoch with min loss and max accuracy: 11 11  \n    min loss and max accuracy: 3.47 0.23  \n    f1 = 0.123 precision = 0.146 recall = 0.226   \n\n## [DenseNet201](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/DenseNet201)  \n    12 epochs, no data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 11 10  \n    min loss and max accuracy: 1.31 0.74  \n    f1 = 0.643 precision = 0.761 recall = 0.599  \n\n## [InceptionResNetV2](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/InceptionResNetV2) \n    12 epochs, no data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 11 11\n    min loss and max accuracy: 1.57 0.66\n    f1 = 0.513 precision = 0.640 recall = 0.480      \n    \n\n## [InceptionV3](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/InceptionV3)\n    12 epochs, no data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 11 11\n    min loss and max accuracy: 1.48 0.69\n    f1 = 0.581 precision = 0.728 recall = 0.538  \n    \n## [MobileNet](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/MobileNet)\n    Trains fast compared to other models: 17s 167ms/step \n    12 epochs, no data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 11 10\n    min loss and max accuracy: 1.11 0.76\n    f1 = 0.717 precision = 0.798 recall = 0.679   \n\n## [MobileNetV2](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/MobileNetV2)\n    Trains fast compared to other models: 17s 174ms/step\n    Epoch with min loss and max accuracy: 11 11\n    min loss and max accuracy: 1.26 0.72\n    f1 = 0.650 precision = 0.763 recall = 0.606 \n\n## [NASNetMobile](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/NASNetMobile) \n    input_shape: Optional shape tuple, only to be specified if include_top is False,\n                 otherwise the input shape has to be (224, 224, 3) \n    12 epochs, no data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 11 11\n    min loss and max accuracy: 2.69 0.38\n    f1 = 0.224 precision = 0.401 recall = 0.203  \n    \n## [ResNet50](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/ResNet50) \n    12 epochs, no data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 11 11\n    min loss and max accuracy: 3.85 0.12\n    f1 = 0.017 precision = 0.035 recall = 0.025  \n\n## [ResNet101V2](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/ResNet101V2) \n    12 epochs, no data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 11 9\n    min loss and max accuracy: 0.87 0.83\n    f1 = 0.775 precision = 0.842 recall = 0.741      \n    \n## [VGG19](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/VGG19) \n    12 epochs, no data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 11 11\n    min loss and max accuracy: 3.58 0.21\n    f1 = 0.031 precision = 0.036 recall = 0.048 \n    \n## [Xception](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/Xception)\n    12 epochs, no data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 11 11\n    min loss and max accuracy: 1.43 0.71\n    f1 = 0.575 precision = 0.712 recall = 0.536    \n    \n## [ResNet101V2](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/ResNet101V2) \n    30 epochs, no data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 29 28\n    min loss and max accuracy: 0.83 0.83\n    f1 = 0.788 precision = 0.863 recall = 0.753 \n\n## Tuning Legend\n## tuning1: pooling='avg', exponential_lr()\n## tuning2: optimizer='nadam'\n## tuning3: Dropout(0.3)\n## tuning4: data_augment_v2 with random_blockout\n## tuning5: workers = 3 [Multi-worker training with Keras](https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras)\n## tuning6: additional data\n## tuning7: data_augment_v3\n## tuning8: pretrained_model.trainable=True\n## tuning9: EfficientNetB7\n## tuning10: noisy-student\n## tuning11: class_weight=weight_per_class\n## tuning12: Test Time Augmentation TTA\n\n\n## [ResNet101V2](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/ResNet101V2)  \n## + tuning1 (0.84 val accuracy at 10 epochs)\n## + tuning2 (0.87 val accuracy at 10 epochs, overfit)  \n## + tuning3 (0.84 val accuracy at 10 epochs, no overfit)\n## Overfitting\n## Inspired by Dmitry's notebook [here](https://www.kaggle.com/dmitrynokhrin/densenet201-aug-additional-data)\n    30 epochs, no data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 26 27\n    min loss and max accuracy: 0.52 0.88\n    f1 = 0.864 precision = 0.916 recall = 0.842     \n      \n## [DenseNet201](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/DenseNet201)  \n## + tuning1, tuning2\n    30 epochs, no data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 29 29\n    min loss and max accuracy: 0.92 0.81\n    f1 = 0.767 precision = 0.833 recall = 0.732\n\n## [DenseNet201](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/DenseNet201)  \n## + tuning1, tuning2, tuning4\n    30 epochs, data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 29 27\n    min loss and max accuracy: 0.92 0.82\n    f1 = 0.772 precision = 0.846 recall = 0.734 \n\n## [ResNet101V2](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/ResNet101V2) \n## + tuning1, tuning2, tuning4\n    30 epochs, data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 29 28\n    min loss and max accuracy: 0.66 0.85\n    f1 = 0.829 precision = 0.870 recall = 0.802 \n\n\n## [ResNet101V2](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/ResNet101V2) \n## + tuning1, tuning2, tuning4, tuning5\n    30 epochs, workers=3, data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 29 28\n    min loss and max accuracy: 0.66 0.85\n    f1 = 0.829 precision = 0.870 recall = 0.802   \n    \n## [DenseNet201](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/DenseNet201)  \n## + tuning1, tuning8\n    30 epochs, no data augmentation, model checkpoint, early stopping, training from scratch\n    Epoch with min loss and max accuracy: 26 28\n    min loss and max accuracy: 0.23 0.95\n    f1 = 0.945 precision = 0.950 recall = 0.946     \n    \n## [ResNet101V2](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/ResNet101V2) \n## + tuning1, tuning8\n    30 epochs, no data augmentation, model checkpoint, early stopping, training from scratch\n    Epoch with min loss and max accuracy: 10 16\n    min loss and max accuracy: 0.36 0.92\n    f1 = 0.909 precision = 0.913 recall = 0.911  \n\n## [DenseNet201](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/DenseNet201)  \n## + tuning1, tuning2, tuning8\n    30 epochs, no data augmentation, model checkpoint, early stopping, training from scratch\n    Epoch with min loss and max accuracy: 10 11\n    min loss and max accuracy: 0.21 0.95\n    f1 = 0.953 precision = 0.960 recall = 0.950\n    \n## [EfficientNetB7](https://github.com/qubvel/efficientnet) #tuning9\n## with noisy-student #tuning10, see [Self-training with Noisy Student improves ImageNet classification](https://arxiv.org/abs/1911.04252)\n## tuning1, tuning9, tuning10\n    30 epochs, no data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 29 27\n    min loss and max accuracy: 0.73 0.84\n    f1 = 0.779 precision = 0.839 recall = 0.755\n    \n## [EfficientNetB7](https://github.com/qubvel/efficientnet) \n## with noisy-student, class_weight\n## tuning1, tuning9, tuning10, tuning11\n    30 epochs, no data augmentation, model checkpoint, early stopping\n    Epoch with min loss and max accuracy: 29 28\n    min loss and max accuracy: 1.0 0.81\n    f1 = 0.775 precision = 0.769 recall = 0.821 \n    \n## [EfficientNetB7](https://github.com/qubvel/efficientnet) \n## with noisy-student, with class_weight, training from scratch\n## tuning1, tuning2, tuning8, tuning9, tuning10, tuning11\n    30 epochs, no data augmentation, model checkpoint, early stopping, training from scratch\n    Epoch with min loss and max accuracy: 15 18\n    min loss and max accuracy: 0.25 0.96\n    f1 = 0.955 precision = 0.950 recall = 0.964\n    \n## [DenseNet201 (https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/DenseNet201)  \n## + tuning1, tuning2, tuning8, tuning11\n    30 epochs, no data augmentation, model checkpoint, early stopping, training from scratch\n    Epoch with min loss and max accuracy: 24 23\n    min loss and max accuracy: 0.22 0.95\n    f1 = 0.956 precision = 0.957 recall = 0.958 \n    \n## Ensemble of [EfficientNetB7](https://github.com/qubvel/efficientnet) \n## with noisy-student, with class_weight, training from scratch\n## tuning1, tuning2, tuning8, tuning9, tuning10, tuning11\n## +\n## [DenseNet201](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/applications/DenseNet201)  \n## + tuning1, tuning2, tuning8, tuning11\n    30 epochs, no data augmentation, model checkpoint, early stopping, training from scratch\n    Epoch with min loss and max accuracy: 24 23\n    min loss and max accuracy: 0.22 0.95\n    f1 = 0.962 precision = 0.960 recall = 0.966     ","metadata":{}},{"cell_type":"markdown","source":"## Create model performance report 😀","metadata":{}},{"cell_type":"code","source":"model_performance_report = pd.DataFrame(columns=['model-family', 'model', 'epochs', 'arg min loss', 'arg max accuracy', \n                                                 'min loss', 'max accuracy', 'f1', 'precision', 'recall'])\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family': 'VGG',\n                                                              'model':'VGG16', \n                                                              'epochs':12, \n                                                              'arg min loss':11, \n                                                              'arg max accuracy':11,\n                                                              'min loss':3.47,\n                                                              'max accuracy':0.23,\n                                                              'f1':0.123,\n                                                              'precision':0.146,\n                                                              'recall':0.226}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family': 'DenseNet',\n                                                              'model':'DenseNet201', \n                                                              'epochs':12, \n                                                              'arg min loss':11, \n                                                              'arg max accuracy':10,\n                                                              'min loss':1.31,\n                                                              'max accuracy':0.74,\n                                                              'f1':0.643,\n                                                              'precision':0.761,\n                                                              'recall':0.599}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'InceptionResNet',\n                                                              'model':'InceptionResNetV2', \n                                                              'epochs':12, \n                                                              'arg min loss':11, \n                                                              'arg max accuracy':11,\n                                                              'min loss':1.57,\n                                                              'max accuracy':0.66,\n                                                              'f1':0.513,\n                                                              'precision':0.640,\n                                                              'recall':0.480}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'Inception', \n                                                              'model':'InceptionV3', \n                                                              'epochs':12, \n                                                              'arg min loss':11, \n                                                              'arg max accuracy':11,\n                                                              'min loss':1.48,\n                                                              'max accuracy':0.69,\n                                                              'f1':0.581,\n                                                              'precision':0.728,\n                                                              'recall':0.538}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'MobileNet', \n                                                              'model':'MobileNet', \n                                                              'epochs':12, \n                                                              'arg min loss':11, \n                                                              'arg max accuracy':10,\n                                                              'min loss':1.11,\n                                                              'max accuracy':0.76,\n                                                              'f1':0.717,\n                                                              'precision':0.798,\n                                                              'recall':0.679}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'MobileNet',\n                                                              'model':'MobileNetV2', \n                                                              'epochs':12, \n                                                              'arg min loss':11, \n                                                              'arg max accuracy':11,\n                                                              'min loss':1.26,\n                                                              'max accuracy':0.72,\n                                                              'f1':0.650,\n                                                              'precision':0.763,\n                                                              'recall':0.606}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'NASNetMobile',\n                                                              'model':'NASNetMobile', \n                                                              'epochs':12, \n                                                              'arg min loss':11, \n                                                              'arg max accuracy':11,\n                                                              'min loss':2.69,\n                                                              'max accuracy':0.38,\n                                                              'f1':0.224,\n                                                              'precision':0.401,\n                                                              'recall':0.203}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'ResNet',\n                                                              'model':'ResNet50', \n                                                              'epochs':12, \n                                                              'arg min loss':11, \n                                                              'arg max accuracy':11,\n                                                              'min loss':3.85,\n                                                              'max accuracy':0.12,\n                                                              'f1':0.017,\n                                                              'precision':0.035,\n                                                              'recall':0.025}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'ResNet',\n                                                              'model':'R101V2', \n                                                              'epochs':12, \n                                                              'arg min loss':11, \n                                                              'arg max accuracy':9,\n                                                              'min loss':0.87,\n                                                              'max accuracy':0.83,\n                                                              'f1':0.775,\n                                                              'precision':0.842,\n                                                              'recall':0.741}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'VGG',\n                                                              'model':'VGG19', \n                                                              'epochs':12, \n                                                              'arg min loss':11, \n                                                              'arg max accuracy':11,\n                                                              'min loss':3.58,\n                                                              'max accuracy':0.21,\n                                                              'f1':0.031,\n                                                              'precision':0.036,\n                                                              'recall':0.048}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'Xception',\n                                                              'model':'Xception', \n                                                              'epochs':12, \n                                                              'arg min loss':11, \n                                                              'arg max accuracy':11,\n                                                              'min loss':1.43,\n                                                              'max accuracy':0.71,\n                                                              'f1':0.575,\n                                                              'precision':0.712,\n                                                              'recall':0.536}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'ResNet',\n                                                              'model':'R2 30e', \n                                                              'epochs':30, \n                                                              'arg min loss':29, \n                                                              'arg max accuracy':28,\n                                                              'min loss':0.83,\n                                                              'max accuracy':0.83,\n                                                              'f1':0.788,\n                                                              'precision':0.863,\n                                                              'recall':0.753}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'ResNet',\n                                                              'model':'R101V2 1,2,3+OF', \n                                                              'epochs':30, \n                                                              'arg min loss':26, \n                                                              'arg max accuracy':27,\n                                                              'min loss':0.52,\n                                                              'max accuracy':0.88,\n                                                              'f1':0.864,\n                                                              'precision':0.916,\n                                                              'recall':0.842}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'DenseNet',\n                                                              'model':'D 1,2', \n                                                              'epochs':30, \n                                                              'arg min loss':29, \n                                                              'arg max accuracy':29,\n                                                              'min loss':0.92,\n                                                              'max accuracy':0.81,\n                                                              'f1':0.767,\n                                                              'precision':0.833,\n                                                              'recall':0.732}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'DenseNet',\n                                                              'model':'D201 1,2,4', \n                                                              'epochs':30, \n                                                              'arg min loss':29, \n                                                              'arg max accuracy':27,\n                                                              'min loss':0.92,\n                                                              'max accuracy':0.82,\n                                                              'f1':0.772,\n                                                              'precision':0.846,\n                                                              'recall':0.734}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'ResNet',\n                                                              'model':'R101V2 1,2,4', \n                                                              'epochs':30, \n                                                              'arg min loss':29, \n                                                              'arg max accuracy':28,\n                                                              'min loss':0.66,\n                                                              'max accuracy':0.85,\n                                                              'f1':0.829,\n                                                              'precision':0.870,\n                                                              'recall':0.802}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'ResNet',\n                                                              'model':'R101V2 1,2,4,5', \n                                                              'epochs':30, \n                                                              'arg min loss':29, \n                                                              'arg max accuracy':23,\n                                                              'min loss':0.66,\n                                                              'max accuracy':0.86,\n                                                              'f1':0.829,\n                                                              'precision':0.883,\n                                                              'recall':0.802}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'DenseNet',\n                                                              'model':'D 1,8', \n                                                              'epochs':30, \n                                                              'arg min loss':26, \n                                                              'arg max accuracy':28,\n                                                              'min loss':0.23,\n                                                              'max accuracy':0.95,\n                                                              'f1':0.945,\n                                                              'precision':0.950,\n                                                              'recall':0.946}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'ResNet',\n                                                              'model':'R101V2 1,8', \n                                                              'epochs':30, \n                                                              'arg min loss':10, \n                                                              'arg max accuracy':16,\n                                                              'min loss':0.36,\n                                                              'max accuracy':0.92,\n                                                              'f1':0.909,\n                                                              'precision':0.913,\n                                                              'recall':0.911}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'ResNet',\n                                                              'model':'D 1,2,8', \n                                                              'epochs':30, \n                                                              'arg min loss':10, \n                                                              'arg max accuracy':11,\n                                                              'min loss':0.21,\n                                                              'max accuracy':0.95,\n                                                              'f1':0.953,\n                                                              'precision':0.960,\n                                                              'recall':0.950}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'EfficientNet',\n                                                              'model':'EB7 1,2,9,10', \n                                                              'epochs':30, \n                                                              'arg min loss':29, \n                                                              'arg max accuracy':27,\n                                                              'min loss':0.73,\n                                                              'max accuracy':0.84,\n                                                              'f1':0.779,\n                                                              'precision':0.839,\n                                                              'recall':0.755}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'EfficientNet',\n                                                              'model':'EB7 +11', \n                                                              'epochs':30, \n                                                              'arg min loss':29, \n                                                              'arg max accuracy':28,\n                                                              'min loss':1.0,\n                                                              'max accuracy':0.81,\n                                                              'f1':0.775,\n                                                              'precision':0.769,\n                                                              'recall':0.821}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'EfficientNet',\n                                                              'model':'EB7 1,2,8,9,10,11', \n                                                              'epochs':30, \n                                                              'arg min loss':15, \n                                                              'arg max accuracy':18,\n                                                              'min loss':0.25,\n                                                              'max accuracy':0.96,\n                                                              'f1':0.955,\n                                                              'precision':0.950,\n                                                              'recall':0.964}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'ResNet',\n                                                              'model':'D 1,2,8,11', \n                                                              'epochs':30, \n                                                              'arg min loss':24, \n                                                              'arg max accuracy':23,\n                                                              'min loss':0.22,\n                                                              'max accuracy':0.95,\n                                                              'f1':0.956,\n                                                              'precision':0.957,\n                                                              'recall':0.958}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'Ensemble',\n                                                              'model':'Ensemble EB7+D201', \n                                                              'epochs':30, \n                                                              'arg min loss':24, \n                                                              'arg max accuracy':23,\n                                                              'min loss':0.22,\n                                                              'max accuracy':0.95,\n                                                              'f1':0.962,\n                                                              'precision':0.960,\n                                                              'recall':0.966}\n\nextra_columns = ['total params', 'trainable params', 'non-trainable params','training time per epoch (sec)']\nmodel_performance_report[extra_columns] = pd.DataFrame([[np.nan, np.nan, np.nan, np.nan]], index=model_performance_report.index)\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'DenseNet',\n                                                              'model':'D 1,2,6',\n                                                              'total params':18_521_768,\n                                                              'trainable params':199_784,\n                                                              'non-trainable params':18_321_984,\n                                                              'training time per epoch (sec)':114,\n                                                              'epochs':30, \n                                                              'arg min loss':29, \n                                                              'arg max accuracy':29,\n                                                              'min loss':0.71,\n                                                              'max accuracy':0.85,\n                                                              'f1':0.826,\n                                                              'precision':0.791,\n                                                              'recall':0.890}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'DenseNet',\n                                                              'model':'D 1,2,6,12',\n                                                              'total params':18_521_768,\n                                                              'trainable params':199_784,\n                                                              'non-trainable params':18_321_984,\n                                                              'training time per epoch (sec)':114,\n                                                              'epochs':30, \n                                                              'arg min loss':29, \n                                                              'arg max accuracy':29,\n                                                              'min loss':0.71,\n                                                              'max accuracy':0.85,\n                                                              'f1':0.826,\n                                                              'precision':0.791,\n                                                              'recall':0.890}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'DenseNet',\n                                                              'model':'D 1,2,6,8',\n                                                              'total params':18_521_768,\n                                                              'trainable params':18_292_712,\n                                                              'non-trainable params':229_056,\n                                                              'training time per epoch (sec)':274,\n                                                              'epochs':30, \n                                                              'arg min loss':26, \n                                                              'arg max accuracy':28,\n                                                              'min loss':0.22,\n                                                              'max accuracy':0.96,\n                                                              'f1':0.948,\n                                                              'precision':0.942,\n                                                              'recall':0.957}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'DenseNet',\n                                                              'model':'D 1,2,6,8,12',\n                                                              'total params':18_521_768,\n                                                              'trainable params':18_292_712,\n                                                              'non-trainable params':229_056,\n                                                              'training time per epoch (sec)':274,\n                                                              'epochs':30, \n                                                              'arg min loss':26, \n                                                              'arg max accuracy':28,\n                                                              'min loss':0.22,\n                                                              'max accuracy':0.96,\n                                                              'f1':0.948,\n                                                              'precision':0.942,\n                                                              'recall':0.957}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'EfficientNet',\n                                                              'model':'EB7 1,2,6,8,9,10,11', \n                                                              'total params':64_364_024,\n                                                              'trainable params':64_053_304,\n                                                              'non-trainable params':310_720,\n                                                              'training time per epoch (sec)':511,                                                             \n                                                              'epochs':30, \n                                                              'arg min loss':20, \n                                                              'arg max accuracy':28,\n                                                              'min loss':0.24,\n                                                              'max accuracy':0.96,\n                                                              'f1':0.956,\n                                                              'precision':0.949,\n                                                              'recall':0.967}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'Ensemble',\n                                                              'model':'Ensemble 6,12 EB7+D201', \n                                                              'total params':82_885_792,\n                                                              'trainable params':82_346_016,\n                                                              'non-trainable params':539_776,\n                                                              'training time per epoch (sec)':785,                                                             \n                                                              'epochs':30, \n                                                              'arg min loss':20, \n                                                              'arg max accuracy':28,\n                                                              'min loss':0.24,\n                                                              'max accuracy':0.96,\n                                                              'f1':0.962,\n                                                              'precision':0.956,\n                                                              'recall':0.971}\n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'MobileNet',\n                                                              'model':'MobileNetV2 1,2,6', \n                                                              'total params':2_391_208,\n                                                              'trainable params':133_224,\n                                                              'non-trainable params':2_257_984,\n                                                              'training time per epoch (sec)':79,                                                              \n                                                              'epochs':30, \n                                                              'arg min loss':29, \n                                                              'arg max accuracy':26,\n                                                              'min loss':0.83,\n                                                              'max accuracy':0.8,\n                                                              'f1':0.781,\n                                                              'precision':0.752,\n                                                              'recall':0.850} \n\nmodel_performance_report.loc[len(model_performance_report)]={ 'model-family':'MobileNet',\n                                                              'model':'MobileNetV2 1,2,6,8', \n                                                              'total params':2_391_208,\n                                                              'trainable params':2_357_096,\n                                                              'non-trainable params':34_112,\n                                                              'training time per epoch (sec)':102,                                                              \n                                                              'epochs':30, \n                                                              'arg min loss':24, \n                                                              'arg max accuracy':27,\n                                                              'min loss':0.27,\n                                                              'max accuracy':0.95,\n                                                              'f1':0.936,\n                                                              'precision':0.929,\n                                                              'recall':0.951}","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model_performance_report","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#sns.set_theme(style=\"white\")\n\n# Plot miles per gallon against horsepower with other semantics\nwith sns.axes_style(\"whitegrid\", {'grid.linestyle': '--'}):\n    myplot = sns.relplot(x=\"model\", y=\"max accuracy\", hue=\"model\", size=\"f1\",\n                sizes=(100, 1000), alpha=1, palette=\"pastel\", legend=\"brief\", #, “brief”, “full”, or False\n                height=15, data=model_performance_report)\n\n#myplot.fig.set_size_inches(25,15)\n\n#Slighlty rotate the x-axis labels so model names to not overlap\nmyplot.set_xticklabels(rotation=45)\n\n#Add yaxis gridlines\nmyplot.axes[0][0].set_yticks(np.arange(0,1.05,0.05), minor=False)\n\n#For each model, add model name, val accuracy and f1 score\ndf = model_performance_report.copy()\nfor line in range(0,df.shape[0]):\n    if df['model'][line] in ['D 1,8', 'D 1,2,8', 'D 1,2,6', 'D 1,2,6,12', 'D 1,2,6,8', 'D 1,2,6,8,12', 'D 1,2,8,11', 'R101V2', 'EB7 1,2,6,8,9,10,11']:\n        #print(df['model'][line])\n        mytext = str(df['model'][line][0]) #+' '+str(df['max accuracy'][line])+' '+str(df['f1'][line])\n    else:\n        mytext = str(df['model'][line])+' - acc:'+str(df['max accuracy'][line])+' - f1:'+str(df['f1'][line])\n        \n    myplot.axes[0,0].text(model_performance_report['model'][line], \n                           df['max accuracy'][line], \n                           mytext, \n                           horizontalalignment='left', \n                           size='medium', \n                           color='black', \n                           weight='normal')\n\n#Add title and rename axes        \nmyplot.set(title='Petals to the Metal - Model Performance - y axis:val acc, size:f1 score - Milestones: 12 epochs; 30 epochs; Hyperparameter tuning; End to end training; Ensemble models; 5x data; Ensemble models of 5x models', xlabel='Model', ylabel='Validation Accuracy')\n\n#Add annotation for training from scratch models\nx_location, y_location = 10, 0.97\nmyplot.axes[0][0].annotate('End to end training (tuning8)', xy=(x_location+6, y_location), xytext=(x_location, y_location+0.01),\n             arrowprops=dict(facecolor='#C4F0EF', shrink=0.05, headwidth=20, width=5))\n\n#Add annotation for Transfer Learning with tuning models \nx_location, y_location = 1, 0.85\nmyplot.axes[0][0].annotate('Transfer Learning with tuning for 30 epochs', xy=(x_location+9, y_location), xytext=(x_location, y_location+0.01),\n             arrowprops=dict(facecolor='#F5B78A', shrink=0.05, headwidth=20, width=5))\n\n#Add annotation for transfer learning models\nx_location, y_location = 5, 0.23\nmyplot.axes[0][0].annotate('Transfer Learning for 12 epochs', xy=(x_location-2, y_location-0.03), xytext=(x_location, y_location-0.05),\n             arrowprops=dict(facecolor='lightgrey', shrink=0.05, headwidth=20, width=5))\n\n#Add annotation for transfer learning models\nx_location, y_location = -1.5, 0.8\nmyplot.axes[0][0].annotate('Transfer Learning for 12 epochs', xy=(x_location+6, y_location), xytext=(x_location, y_location+0.01),\n             arrowprops=dict(facecolor='#FFFDAE', shrink=0.05, headwidth=20, width=5))\n\n#Add annotation for Ensemble EB7+D201\nx_location, y_location = 19, 0.97\nmyplot.axes[0][0].annotate('Ensemble EB7+D201', xy=(x_location+5, y_location-0.0005), xytext=(x_location, y_location+0.01),\n             arrowprops=dict(facecolor='#CCBDFA', shrink=0.05, headwidth=20, width=5))\n\n#Add annotation for 70K (5x) additional data\nx_location, y_location = 22, 0.98\nmyplot.axes[0][0].annotate('Additional data 70K (5x)', xy=(x_location+5, y_location-0.01), xytext=(x_location, y_location+0.01),\n             arrowprops=dict(facecolor='lightgrey', shrink=0.05, headwidth=20, width=5))\n\n#Add annotation for Legend and \nx_location, y_location = 15, 0.1\nmyplot.axes[0][0].annotate('Models sorted chronologically, size:f1 score', xy=(x_location+12, y_location), xytext=(x_location, y_location),\n             arrowprops=dict(facecolor='black', shrink=0.05, headwidth=20, width=5))\n\n#Add Tuning Legend\nx_location, y_location, y_delta = 15.1, 0.655, 0.03\nmyplot.axes[0][0].annotate('Author: George Zoto', xy=(x_location, y_location+y_delta), size='x-large')\nmyplot.axes[0][0].annotate('Tuning Legend, models sorted chronologically', xy=(x_location, y_location), size='x-large')\nmyplot.axes[0][0].annotate('tuning1: pooling=avg, exponential_lr()', xy=(x_location, y_location-y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning2: optimizer=nadam', xy=(x_location, y_location-2*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning3: Dropout(0.3)', xy=(x_location, y_location-3*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning4: data_augment_v2 with random_blockout', xy=(x_location, y_location-4*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning5: workers = 3 Multi-worker training with Keras', xy=(x_location, y_location-5*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning6: additional data', xy=(x_location, y_location-6*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning7: data_augment_v3', xy=(x_location, y_location-7*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning8: pretrained_model.trainable=True', xy=(x_location, y_location-8*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning9: EfficientNetB7', xy=(x_location, y_location-9*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning10: noisy-student', xy=(x_location, y_location-10*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning11: weight_per_class', xy=(x_location, y_location-11*y_delta), size='large');\nmyplot.axes[0][0].annotate('tuning12: Test Time Augmentation TTA', xy=(x_location, y_location-12*y_delta), size='large');","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model_performance_report = model_performance_report.sort_values(by='max accuracy')\nmodel_performance_report","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Plot model performance report 😀","metadata":{}},{"cell_type":"code","source":"#sns.set_theme(style=\"white\")\n\n# Plot miles per gallon against horsepower with other semantics\nwith sns.axes_style(\"whitegrid\", {'grid.linestyle': '--'}):\n    myplot = sns.relplot(x=\"model\", y=\"max accuracy\", hue=\"model\", size=\"f1\",\n                sizes=(100, 1000), alpha=1, palette=\"pastel\", legend=\"brief\", #, “brief”, “full”, or False\n                height=15, data=model_performance_report)\n\n#myplot.fig.set_size_inches(25,15)\n\n#Slighlty rotate the x-axis labels so model names to not overlap\nmyplot.set_xticklabels(rotation=70)\n\n#Add yaxis gridlines\nmyplot.axes[0][0].set_yticks(np.arange(0,1.05,0.05), minor=False)\n\n#For each model, add model name, val accuracy and f1 score\ndf = model_performance_report.copy()\nfor line in range(0,df.shape[0]):\n    if df['model'][line] in ['D 1,2', 'D 1,8', 'D 1,2,6', 'D 1,2,6,12', 'D 1,2,6,8', 'D 1,2,6,8,12', 'D 1,2,8', 'D 1,2,8,11', 'R101V2', 'EB7 1,2,8,9,10,11', 'EB7 1,2,6,8,9,10,11', 'Ensemble EB7+D201', 'MobileNetV2 1,2,6,8']:\n        #print(df['model'][line])\n        mytext = str(df['model'][line][0])#+' '+str(df['max accuracy'][line])+' '+str(df['f1'][line])\n    else:\n        mytext = str(df['model'][line])+' - acc:'+str(df['max accuracy'][line])+' - f1:'+str(df['f1'][line])\n        \n    myplot.axes[0,0].text(model_performance_report['model'][line], \n                           df['max accuracy'][line], \n                           mytext, \n                           horizontalalignment='left', \n                           size='medium', \n                           color='black', \n                           weight='normal')\n\n#Add title and rename axes        \nmyplot.set(title='Petals to the Metal - Model Performance - y axis:val acc, size:f1 score - Milestones: 12 epochs; 30 epochs; Hyperparameter tuning; End to end training; Ensemble models; 5x data; Ensemble models of 5x models', xlabel='Model', ylabel='Validation Accuracy')\n\n#Add annotation for training from scratch models\nx_location, y_location = 14, 0.93\nmyplot.axes[0][0].annotate('End to end training (tuning8)', xy=(x_location+7, y_location), xytext=(x_location, y_location+0.01),\n             arrowprops=dict(facecolor='#a0e2a7', shrink=0.05, headwidth=20, width=5))\n\n#Add annotation for Transfer Learning with tuning models \nx_location, y_location = 4, 0.85\nmyplot.axes[0][0].annotate('Transfer Learning with tuning for 30 epochs', xy=(x_location+9, y_location), xytext=(x_location, y_location+0.01),\n             arrowprops=dict(facecolor='#CCBDFA', shrink=0.05, headwidth=20, width=5))\n\n#Add annotation for transfer learning models\nx_location, y_location = 5, 0.23\nmyplot.axes[0][0].annotate('Transfer Learning for 12 epochs', xy=(x_location-2, y_location-0.03), xytext=(x_location, y_location-0.05),\n             arrowprops=dict(facecolor='lightgrey', shrink=0.05, headwidth=20, width=5))\n\n#Add annotation for transfer learning models\nx_location, y_location = 0, 0.75\nmyplot.axes[0][0].annotate('Transfer Learning for 12 epochs', xy=(x_location+7, y_location), xytext=(x_location, y_location+0.01),\n             arrowprops=dict(facecolor='lightgrey', shrink=0.05, headwidth=20, width=5))\n\n#Add annotation for Ensemble EB7+D201\nx_location, y_location = 19, 0.96\nmyplot.axes[0][0].annotate('Ensemble EB7+D201', xy=(x_location+5, y_location-0.001), xytext=(x_location, y_location+0.01),\n             arrowprops=dict(facecolor='#ccbdfa', shrink=0.05, headwidth=20, width=5))\n\n#Add annotation for 70K (5x) additional data\nx_location, y_location = 21.5, 0.975\nmyplot.axes[0][0].annotate('Additional data 70K (5x)', xy=(x_location+5.5, y_location-0.005), xytext=(x_location, y_location+0.01),\n             arrowprops=dict(facecolor='lightgrey', shrink=0.05, headwidth=20, width=5))\n\n#Add annotation for Ensemble 6,12 EB7+D201\nx_location, y_location = 26.5, 0.985\nmyplot.axes[0][0].annotate('Ensemble 6,12 EB7+D201', xy=(x_location+5.5, y_location-0.01), xytext=(x_location, y_location+0.01),\n             arrowprops=dict(facecolor='#a0e2a7', shrink=0.05, headwidth=20, width=5))\n\n#Add annotation for Legend and \nx_location, y_location = 15, 0.1\nmyplot.axes[0][0].annotate('Models sorted by val accuracy, size:f1 score', xy=(x_location+12, y_location), xytext=(x_location, y_location),\n             arrowprops=dict(facecolor='black', shrink=0.05, headwidth=20, width=5))\n\n#Add Tuning Legend\nx_location, y_location, y_delta = 15.1, 0.655, 0.03\nmyplot.axes[0][0].annotate('Author: George Zoto', xy=(x_location, y_location+y_delta), size='x-large')\nmyplot.axes[0][0].annotate('Tuning Legend, models sorted by performance', xy=(x_location, y_location), size='x-large')\nmyplot.axes[0][0].annotate('tuning1: pooling=avg, exponential_lr()', xy=(x_location, y_location-y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning2: optimizer=nadam', xy=(x_location, y_location-2*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning3: Dropout(0.3)', xy=(x_location, y_location-3*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning4: data_augment_v2 with random_blockout', xy=(x_location, y_location-4*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning5: workers = 3 Multi-worker training with Keras', xy=(x_location, y_location-5*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning6: additional data', xy=(x_location, y_location-6*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning7: data_augment_v3', xy=(x_location, y_location-7*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning8: pretrained_model.trainable=True', xy=(x_location, y_location-8*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning9: EfficientNetB7', xy=(x_location, y_location-9*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning10: noisy-student', xy=(x_location, y_location-10*y_delta), size='large')\nmyplot.axes[0][0].annotate('tuning11: weight_per_class', xy=(x_location, y_location-11*y_delta), size='large');\nmyplot.axes[0][0].annotate('tuning12: Test Time Augmentation TTA', xy=(x_location, y_location-12*y_delta), size='large');","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Plot model performance report in 3D 😎","metadata":{}},{"cell_type":"code","source":"model_performance_report.head(3)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig = px.scatter_3d(model_performance_report, \n                    title='How early (arg min/max) did a model perform best', symbol='model-family', color='model', \n                    x='epochs', y='arg min loss', z='arg max accuracy',\n                    size_max=12, opacity=0.7,\n                    width=1200, height=700,\n                   )\n\nfig.update_layout(margin=dict(l=0, r=0, b=0, t=30))\n\nfig.show()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model_performance_report.query('epochs == 12')","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Filter only for 12 epoch models\nmodel_performance_report_filtered = model_performance_report.query('epochs == 12').copy()\n\nfig = px.scatter_3d(model_performance_report_filtered, \n                    title='12 epoch model performance - loss and accuracy - by model-family', symbol='model-family', color='model', \n                    x='model-family', y='min loss', z='max accuracy', text='model',\n                    size_max=12, opacity=0.7,\n                    width=1200, height=700,\n                   )\n\nfig.update_layout(margin=dict(l=0, r=0, b=0, t=30))\n\nfig.show()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model_performance_report.query('epochs == 30')","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Filter only for 30 epoch models\nmodel_performance_report_filtered = model_performance_report.query('epochs == 30').copy()\n\nfig = px.scatter_3d(model_performance_report_filtered, \n                    title='30 epoch model performance - loss and accuracy - by model-family', symbol='model-family', color='model', \n                    x='model-family', y='min loss', z='max accuracy', text='model',\n                    size_max=12, opacity=0.7,\n                    width=1200, height=700,\n                   )\n\nfig.update_layout(margin=dict(l=0, r=0, b=0, t=30))\n\nfig.show()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Filter only for 30 epoch models\nmodel_performance_report_filtered = model_performance_report.query('epochs == 30').copy()\n\nfig = px.scatter_3d(model_performance_report_filtered, \n                    title='30 epoch model performance - f1, precision, recall (color) - by model-family', symbol='model-family', color='recall', \n                    x='model-family', y='f1', z='precision', text='model',\n                    size_max=12, opacity=0.7,\n                    width=1200, height=700,\n                   )\n\nfig.update_layout(margin=dict(l=0, r=0, b=0, t=30))\nfig.show()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Filter only for 30 epoch models\nmodel_performance_report_filtered = model_performance_report.query('epochs == 30').copy()\n\nfig = px.scatter_3d(model_performance_report_filtered, \n                    title='30 epoch model performance - f1, precision, recall - by model-family', symbol='model-family', color='model', \n                    x='f1', y='precision', z='recall', text='model',\n                    size_max=12, opacity=0.7,\n                    width=1200, height=700,\n                   )\n\nfig.update_layout(margin=dict(l=0, r=0, b=0, t=30))\nfig.show()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Visual Validation ##\n\nIt can also be helpful to look at some examples from the validation set and see what class your model predicted. This can help reveal patterns in the kinds of images your model has trouble with.\n\nThis cell will set up the validation set to display 20 images at a time -- you can change this to display more or fewer, if you like.","metadata":{}},{"cell_type":"code","source":"dataset = get_validation_dataset()\ndataset = dataset.unbatch().batch(20)\nbatch = iter(dataset)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"And here is a set of flowers with their predicted species. Run the cell again to see another set.","metadata":{}},{"cell_type":"code","source":"images, labels = next(batch)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if using_ensemble_models:\n    probabilities1 = model_EB7.predict(images)\n    probabilities2 = model_D201.predict(images)\n    probabilities = best_alpha * probabilities1 + (1 - best_alpha) * probabilities2\nelse:\n    probabilities = model.predict(images)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"predictions = np.argmax(probabilities, axis=-1)\ndisplay_batch_of_images((images, labels), predictions)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Mismatches on validation data\n## Inspired by Rosa's notebook [here](https://www.kaggle.com/wrrosa/tpu-enet-b7-densenet)","metadata":{}},{"cell_type":"code","source":"mismatches = sum(cm_predictions!=cm_correct_labels)\nprint('Number of mismatches on validation data: {} out of {} or ({:.2%})'.format(mismatches, NUM_VALIDATION_IMAGES, mismatches/NUM_VALIDATION_IMAGES))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"cmdataset = get_validation_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and labels, order matters.\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch\n\nmismatches_images, mismatches_predictions, mismatches_labels = [], [], []\nmismatches_dataset = tf.data.Dataset.from_tensors([])\nval_batch = iter(cmdataset.unbatch().batch(1))\n\nfor image_index in range(NUM_VALIDATION_IMAGES):\n    batch = next(val_batch)\n    if cm_predictions[image_index] != cm_correct_labels[image_index]:\n        print('Predicted vs Correct labels: {}, {}'.format(cm_predictions[image_index], cm_correct_labels[image_index]))\n        #display_batch_of_images(batch, np.array([cm_predictions[image_index]]))\n        #mismatches_dataset = tf.data.Dataset.from_tensors(batch)\n        #mismatches_images.append(tf.data.Dataset.from_tensors(batch))\n        #mismatches_predictions.append(cm_predictions[image_index])\n        #mismatches_labels.append(cm_correct_labels[image_index])","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"dataset = get_validation_dataset()\ndataset = dataset.unbatch().batch(20)\nbatch = iter(dataset)\nimages, labels = next(batch)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for i in range(3):\n    display_batch_of_images((images, labels), predictions, display_mismatches_only=True)\n    images, labels = next(batch)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#mismatches_predictions[0]","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let's take a look again at some training images","metadata":{}},{"cell_type":"code","source":"one_batch = next(ds_iter)\ndisplay_batch_of_images(one_batch)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Step 8: Make Test Predictions #\n\nOnce you're satisfied with everything, you're ready to make predictions on the test set.","metadata":{}},{"cell_type":"markdown","source":"## tuning12: Test Time Augmentation TTA inspired by Araik's notebook [here](https://www.kaggle.com/atamazian/fc-ensemble-external-data-effnet-densenet), Andrew's notebook [here](https://www.kaggle.com/andrewkh/test-time-augmentation-tta-worth-it) and Nathan's article [here](https://towardsdatascience.com/test-time-augmentation-tta-and-how-to-perform-it-with-keras-4ac19b67fb4d)\n## [Learning Loss for Test-Time Augmentation Paper](https://arxiv.org/pdf/2010.11422.pdf)","metadata":{}},{"cell_type":"code","source":"using_tta = False #tuning12\ntta_iterations = 3","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if using_tta:\n    def get_test_dataset(ordered=False):\n        dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n        dataset = dataset.map(data_augment, num_parallel_calls=AUTO) #tuning4\n        #dataset = dataset.map(data_augment_v2, num_parallel_calls=AUTO) #tuning4 #error in shapes\n        #dataset = dataset.map(data_augment_v3, num_parallel_calls=AUTO) #tuning4 0.44 performance :(\n        dataset = dataset.batch(BATCH_SIZE)\n        dataset = dataset.prefetch(AUTO)\n        return dataset","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def predict_tta(model, tta_iterations):\n    probs  = []\n    for i in range(tta_iterations):\n        print('TTA iteration ', i)\n        test_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n        test_images_ds = test_ds.map(lambda image, idnum: image)\n        \n        if using_ensemble_models:\n            print('using_ensemble_models')\n            probabilities1 = model_EB7.predict(test_images_ds)\n            probabilities2 = model_D201.predict(test_images_ds)\n            probabilities = best_alpha * probabilities1 + (1 - best_alpha) * probabilities2\n            probs.append(probabilities)\n        else:\n            probs.append(model.predict(test_images_ds,verbose=0))\n        \n    return probs","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True)\ntest_images_ds = test_ds.map(lambda image, idnum: image)\n\nif using_tta:\n    print('Computing predictions using TTA...')\n    probabilities = np.mean(predict_tta(model, tta_iterations), axis=0)\nelse:\n    print('Computing predictions...')\n    probabilities = model.predict(test_images_ds)\npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We'll generate a file `submission.csv`. This file is what you'll submit to get your score on the leaderboard.","metadata":{}},{"cell_type":"code","source":"print('using_ensemble_models:', using_ensemble_models)\nprint('Generating submission.csv file...')\n\n# Get image ids from test set and convert to unicode\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n\n# Write the submission file\nnp.savetxt(\n    'submission.csv',\n    np.rec.fromarrays([test_ids, predictions]),\n    fmt=['%s', '%d'],\n    delimiter=',',\n    header='id,label',\n    comments='',\n)\n\n# Look at the first few predictions\n!head submission.csv","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Step 9: Make a submission #\n\nIf you haven't already, create your own editable copy of this notebook by clicking on the **Copy and Edit** button in the top right corner. Then, submit to the competition by following these steps:\n\n1. Begin by clicking on the blue **Save Version** button in the top right corner of the window.  This will generate a pop-up window.  \n2. Ensure that the **Save and Run All** option is selected, and then click on the blue **Save** button.\n3. This generates a window in the bottom left corner of the notebook.  After it has finished running, click on the number to the right of the **Save Version** button.  This pulls up a list of versions on the right of the screen.  Click on the ellipsis **(...)** to the right of the most recent version, and select **Open in Viewer**.  This brings you into view mode of the same page. You will need to scroll down to get back to these instructions.\n4. Click on the **Output** tab on the right of the screen.  Then, click on the file you would like to submit, and click on the blue **Submit** button to submit your results to the leaderboard.\n\nYou have now successfully submitted to the competition!\n\nIf you want to keep working to improve your performance, select the blue **Edit** button in the top right of the screen. Then you can change your code and repeat the process. There's a lot of room to improve, and you will climb up the leaderboard as you work.\n","metadata":{}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161321) to chat with other Learners.*","metadata":{}}]}