{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n# Project Introduction #\n\nWelcome to the [**Petals to the Metal**](https://www.kaggle.com/c/tpu-getting-started) Kaggle competition! \n\nIn this competition, we’re challenged to build a machine learning model to classify 104 types of flowers based on their images.\n\nThis notebook builds upon [Ryan's notebook](https://www.kaggle.com/ryanholbrook/create-your-first-submission) on how to build an image classifen in Keras and train it on [Tensor Processing Unit (TPU)](https://www.kaggle.com/docs/tpu). We then expand from there to train with other models and technique with ideas of our own.\n\nPlease enjoy!","metadata":{}},{"cell_type":"markdown","source":"# Step 1: Imports #\n\nWe begin by importing several Python packages.\n\nWe used TensorFlow, which is an open-source Python library for deep learning, to improve performance and ability to learn from more models.","metadata":{}},{"cell_type":"code","source":"# data analysis tool\nimport math, re, os\nimport numpy as np\nimport pandas as pd\nimport random\n\n# efficientNet\n!pip install -q efficientnet\nimport efficientnet.tfkeras as efn\n\n# tensorflow for keras\nimport tensorflow as tf\nprint(\"Tensorflow version \" + tf.__version__)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T02:02:49.606968Z","iopub.execute_input":"2022-03-09T02:02:49.607564Z","iopub.status.idle":"2022-03-09T02:03:05.752509Z","shell.execute_reply.started":"2022-03-09T02:02:49.607474Z","shell.execute_reply":"2022-03-09T02:03:05.751740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 2: Distribution Strategy #\n\nA TPU has eight different *cores* and each of these cores acts as its own accelerator. A TPU is sort of like having eight GPUs in one machine. We used TPUs for this challenge because these are equipped with 128 GB high-speed memory, so even with 512x512 pixels input images (provided in the dataset as well), TPUs can handle these properlly. \n\nWe tell TensorFlow how to make use of all these cores at once through a **distribution strategy**. Run the following cell to create the distribution strategy that we'll later apply to our model.\n\nRead more about [TPU](https://www.kaggle.com/docs/tpu).","metadata":{}},{"cell_type":"code","source":"# Detect TPU, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T02:03:05.754644Z","iopub.execute_input":"2022-03-09T02:03:05.756101Z","iopub.status.idle":"2022-03-09T02:03:11.817194Z","shell.execute_reply.started":"2022-03-09T02:03:05.756040Z","shell.execute_reply":"2022-03-09T02:03:11.816531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll use the distribution strategy when we create our neural network model. Then, TensorFlow will distribute the training among the eight TPU cores by creating eight different *replicas* of the model, one for each core.\n\n# Step 3: Loading the Competition Data #\n\n## Get GCS Path ##\n\nWhen used with TPUs, datasets need to be stored in a [Google Cloud Storage bucket](https://cloud.google.com/storage/). ","metadata":{}},{"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\n\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('tpu-getting-started')\nprint(GCS_DS_PATH) # what do gcs paths look like?","metadata":{"execution":{"iopub.status.busy":"2022-03-09T02:03:11.818612Z","iopub.execute_input":"2022-03-09T02:03:11.818878Z","iopub.status.idle":"2022-03-09T02:03:12.273568Z","shell.execute_reply.started":"2022-03-09T02:03:11.818826Z","shell.execute_reply":"2022-03-09T02:03:12.272438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can use data from any public dataset here on Kaggle in just the same way. If you'd like to use data from one of your private datasets, see [here](https://www.kaggle.com/docs/tpu#tpu3pt5).\n\n## Load Data ##\n\nWhen used with TPUs, datasets are often serialized into [TFRecords](https://www.kaggle.com/ryanholbrook/tfrecords-basics). This is a format convenient for distributing data to each of the TPUs cores. We've hidden the cell that reads the TFRecords for our dataset since the process is a bit long. You could come back to it later for some guidance on using your own datasets with TPUs.","metadata":{}},{"cell_type":"code","source":"\nIMAGE_SIZE = [512, 512]\nGCS_PATH = GCS_DS_PATH + '/tfrecords-jpeg-512x512'\nAUTO = tf.data.experimental.AUTOTUNE # read from multiple files if possible\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') \n\nCLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 102\n\n\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-09T02:03:12.275047Z","iopub.execute_input":"2022-03-09T02:03:12.275294Z","iopub.status.idle":"2022-03-09T02:03:12.561454Z","shell.execute_reply.started":"2022-03-09T02:03:12.275263Z","shell.execute_reply":"2022-03-09T02:03:12.560159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 💽 Modification 7: Incorporate external data ###\n\nIn one of the top-rated notebook [FC Ensemble External Data (EffNet+DenseNet)](https://www.kaggle.com/atamazian/fc-ensemble-external-data-effnet-densenet), it mentions how using external data can increase model’s accuracy. This makes a lot of sense theoretically since with more training data, we can refine the classifier. So let’s add in more [data](https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec)!\n\nIn Kaggle notebook, that means we need to open the right hand column and add data to it. Search for the ```tf-flower-photo-tfrec``` dataset and add it!\n\nNow run the next cell, you shall see both dataset presented. \n\n```python\n# list datasets\nls /kaggle/input\n\n# load in external data\nGCS_DS_PATH_EXT = KaggleDatasets().get_gcs_path('tf-flower-photo-tfrec')\nGCS_PATH_EXT = '/tfrecords-jpeg-512x512' # choose 512*512 again thanks to TPUs\n\n# five different folders are provided in the dataset, load separatly\nIMAGENET_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/imagenet' + GCS_PATH_EXT + '/*.tfrec')\nINATURALIST_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/inaturalist' + GCS_PATH_EXT + '/*.tfrec')\nOPENIMAGE_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/openimage' + GCS_PATH_EXT + '/*.tfrec')\nOXFORD_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/oxford_102' + GCS_PATH_EXT + '/*.tfrec')\nTFFLOWERS_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/tf_flowers' + GCS_PATH_EXT + '/*.tfrec')\n\nEXTRA_TRAINING_FILENAMES = IMAGENET_FILENAMES + INATURALIST_FILENAMES + OPENIMAGE_FILENAMES + OXFORD_FILENAMES + TFFLOWERS_FILENAMES\n\n# add extra training to the existing training dataset\nTRAINING_FILENAMES = TRAINING_FILENAMES + EXTRA_TRAINING_FILENAMES\n```\n\nOriginal Dataset: 12753 training images, 3712 validation images, 7382 unlabeled test images\n\nWith Extra Training Dataset: 68094 training images, 3712 validation images, 7382 unlabeled test images\n\nThat is a lot of new data added! We go from 12k to 68k now. The time for training for sure is going to go up. Unfortunatly, we are running out of our TPU quotas, so we cannot test it out at this point.","metadata":{}},{"cell_type":"code","source":"# load in external data\nGCS_DS_PATH_EXT = KaggleDatasets().get_gcs_path('tf-flower-photo-tfrec')\nGCS_PATH_EXT = '/tfrecords-jpeg-512x512' # choose 512*512 again thanks to TPUs\n\n# five different folders are provided in the dataset, load separatly\nIMAGENET_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/imagenet' + GCS_PATH_EXT + '/*.tfrec')\nINATURALIST_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/inaturalist' + GCS_PATH_EXT + '/*.tfrec')\nOPENIMAGE_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/openimage' + GCS_PATH_EXT + '/*.tfrec')\nOXFORD_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/oxford_102' + GCS_PATH_EXT + '/*.tfrec')\nTFFLOWERS_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH_EXT + '/tf_flowers' + GCS_PATH_EXT + '/*.tfrec')\n\nEXTRA_TRAINING_FILENAMES = IMAGENET_FILENAMES + INATURALIST_FILENAMES + OPENIMAGE_FILENAMES + OXFORD_FILENAMES + TFFLOWERS_FILENAMES\n\n# add extra training to the existing training dataset\nTRAINING_FILENAMES = TRAINING_FILENAMES + EXTRA_TRAINING_FILENAMES","metadata":{"execution":{"iopub.status.busy":"2022-03-09T02:03:12.563601Z","iopub.execute_input":"2022-03-09T02:03:12.563921Z","iopub.status.idle":"2022-03-09T02:03:13.355931Z","shell.execute_reply.started":"2022-03-09T02:03:12.563884Z","shell.execute_reply":"2022-03-09T02:03:13.354997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create Data Pipelines ##\n\nIn this final step we'll use the `tf.data` API to define an efficient data pipeline for each of the training, validation, and test splits.\n\n### 🚧 Modification 5: Data Augmenting ###\n\nIn the provided starter code, only one simple data augmentation is done, which is flip the image horizontally. \n\n```python\ndef data_augment(image, label):\n    # Thanks to the dataset.prefetch(AUTO)\n    # statement in the next function (below), this happens essentially\n    # for free on TPU. Data pipeline code is executed on the \"CPU\"\n    # part of the TPU while the TPU itself is computing gradients.\n    image = tf.image.random_flip_left_right(image)\n    #image = tf.image.random_saturation(image, 0, 2)\n    return image, label   \n```\n\nWe encounter two interesting image augmentations, which both involves some randomness. Not exactly sure the theory behind it, but it can be effective. Let's test it out.\n\n#### Trial 5.1 Random Blockout ####\nAdapted from Dmitry's [notebook](https://www.kaggle.com/dmitrynokhrin/densenet201-aug-additional-data):","metadata":{}},{"cell_type":"code","source":"SEED = 2022\n\ndef random_blockout(img, sl=0.1, sh=0.2, rl=0.4):\n    p=random.random()\n    if p>=0.25:\n        w, h, c = IMAGE_SIZE[0], IMAGE_SIZE[1], 3\n        origin_area = tf.cast(h*w, tf.float32)\n\n        # set up the erase block\n        e_size_l = tf.cast(tf.round(tf.sqrt(origin_area * sl * rl)), tf.int32)\n        e_size_h = tf.cast(tf.round(tf.sqrt(origin_area * sh / rl)), tf.int32)\n\n        e_height_h = tf.minimum(e_size_h, h)\n        e_width_h = tf.minimum(e_size_h, w)\n\n        erase_height = tf.random.uniform(shape=[], minval=e_size_l, maxval=e_height_h, dtype=tf.int32)\n        erase_width = tf.random.uniform(shape=[], minval=e_size_l, maxval=e_width_h, dtype=tf.int32)\n\n        erase_area = tf.zeros(shape=[erase_height, erase_width, c])\n        erase_area = tf.cast(erase_area, tf.uint8)\n\n        pad_h = h - erase_height\n        pad_top = tf.random.uniform(shape=[], minval=0, maxval=pad_h, dtype=tf.int32)\n        pad_bottom = pad_h - pad_top\n\n        pad_w = w - erase_width\n        pad_left = tf.random.uniform(shape=[], minval=0, maxval=pad_w, dtype=tf.int32)\n        pad_right = pad_w - pad_left\n\n        erase_mask = tf.pad([erase_area], [[0,0],[pad_top, pad_bottom], [pad_left, pad_right], [0,0]], constant_values=1)\n        erase_mask = tf.squeeze(erase_mask, axis=0)\n        erased_img = tf.multiply(tf.cast(img,tf.float32), tf.cast(erase_mask, tf.float32))\n\n        return tf.cast(erased_img, img.dtype)\n    else:\n        return tf.cast(img, img.dtype)\n\n    \ndef data_augment_v2(image, label):\n    flag = random.randint(1,3)\n    coef_1 = random.randint(70, 90) * 0.01\n    coef_2 = random.randint(70, 90) * 0.01\n    \n    if flag == 1:\n        image = tf.image.random_flip_left_right(image, seed=SEED)\n    elif flag == 2:\n        image = tf.image.random_flip_up_down(image, seed=SEED)\n    else:\n        image = tf.image.random_crop(image, [int(IMAGE_SIZE[0]*coef_1), int(IMAGE_SIZE[0]*coef_2), 3],seed=SEED)\n        \n    image = random_blockout(image)\n    \n    return image, label ","metadata":{"execution":{"iopub.status.busy":"2022-03-09T02:03:13.357292Z","iopub.execute_input":"2022-03-09T02:03:13.357575Z","iopub.status.idle":"2022-03-09T02:03:13.377522Z","shell.execute_reply.started":"2022-03-09T02:03:13.357542Z","shell.execute_reply":"2022-03-09T02:03:13.376268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Trial 5.2 Random Modifications to Images ####\nAdapted from Xuanzhi Huang and Rahul Paul's [notebook](https://www.kaggle.com/xuanzhihuang/flower-classification-densenet-201):\n\n```python\nimport tensorflow_addons as tfa\n\n# Randomly make some changes to the images and return the new images and labels\ndef data_augment_v3(image, label):\n        \n    # Set seed for data augmentation\n    seed = 100\n    \n    # Randomly resize and then crop images\n    image = tf.image.resize(image, [720, 720])\n    image = tf.image.random_crop(image, [512, 512, 3], seed = seed)\n\n    # Randomly reset brightness of images\n    image = tf.image.random_brightness(image, 0.6, seed = seed)\n    \n    # Randomly reset saturation of images\n    image = tf.image.random_saturation(image, 3, 5, seed = seed)\n        \n    # Randomly reset contrast of images\n    image = tf.image.random_contrast(image, 0.3, 0.5, seed = seed)\n\n    # Randomly reset hue of images, but this will make the colors really weird, which we think will not happen\n    # in common photography\n    # image = tf.image.random_hue(image, 0.5, seed = seed)\n    \n    # Blur images\n    image = tfa.image.mean_filter2d(image, filter_shape = 10)\n    \n    # Randomly flip images\n    image = tf.image.random_flip_left_right(image, seed = seed)\n    image = tf.image.random_flip_up_down(image, seed = seed)\n    \n    return image, label\n```\n\n### 🚧 Modification 5: Data Augmenting - Comparisons ###\nHere are the statistics on validation set:\n\n* random blocking: f1 = 0.960, precision = 0.961, recall = 0.963\n* random changing: f1 = 0.791, precision = 0.844, recall = 0.781\n\nSo by randomly changing the pictures decreases the accuracy significantly, but blocking some parts of the training images does help improve the accuracy a little bit. We will keep that since it does gives us better results when submitted to the competition -- **0.95462**!","metadata":{}},{"cell_type":"code","source":"def get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    # dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.map(data_augment_v2, num_parallel_calls=AUTO)\n    # dataset = dataset.map(data_augment_v3, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec\n    # files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","metadata":{"execution":{"iopub.status.busy":"2022-03-09T02:03:13.379000Z","iopub.execute_input":"2022-03-09T02:03:13.379922Z","iopub.status.idle":"2022-03-09T02:03:13.398379Z","shell.execute_reply.started":"2022-03-09T02:03:13.379828Z","shell.execute_reply":"2022-03-09T02:03:13.397212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This next cell will create the datasets that we'll use with Keras during training and inference. Notice how we scale the size of the batches to the number of TPU cores.","metadata":{}},{"cell_type":"code","source":"# Define the batch size. This will be 16 with TPU off and 128 (=16*8) with TPU on\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n\nds_train = get_training_dataset()\nds_valid = get_validation_dataset()\nds_test = get_test_dataset()\n\nprint(\"Training:\", ds_train)\nprint (\"Validation:\", ds_valid)\nprint(\"Test:\", ds_test)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T02:03:13.400001Z","iopub.execute_input":"2022-03-09T02:03:13.400280Z","iopub.status.idle":"2022-03-09T02:03:14.895026Z","shell.execute_reply.started":"2022-03-09T02:03:13.400247Z","shell.execute_reply":"2022-03-09T02:03:14.893685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These datasets are `tf.data.Dataset` objects. You can think about a dataset in TensorFlow as a *stream* of data records. The training and validation sets are streams of `(image, label)` pairs.","metadata":{}},{"cell_type":"code","source":"np.set_printoptions(threshold=15, linewidth=80)\n\nprint(\"Training data shapes:\")\nfor image, label in ds_train.take(3):\n    print(image.numpy().shape, label.numpy().shape)\nprint(\"Training data label examples:\", label.numpy())","metadata":{"execution":{"iopub.status.busy":"2022-03-09T02:03:14.896251Z","iopub.execute_input":"2022-03-09T02:03:14.896463Z","iopub.status.idle":"2022-03-09T02:03:21.783677Z","shell.execute_reply.started":"2022-03-09T02:03:14.896438Z","shell.execute_reply":"2022-03-09T02:03:21.782066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The test set is a stream of `(image, idnum)` pairs; `idnum` here is the unique identifier given to the image that we'll use later when we make our submission as a `csv` file.","metadata":{}},{"cell_type":"code","source":"print(\"Test data shapes:\")\nfor image, idnum in ds_test.take(3):\n    print(image.numpy().shape, idnum.numpy().shape)\nprint(\"Test data IDs:\", idnum.numpy().astype('U')) # U=unicode string","metadata":{"execution":{"iopub.status.busy":"2022-03-09T02:04:12.476925Z","iopub.execute_input":"2022-03-09T02:04:12.477730Z","iopub.status.idle":"2022-03-09T02:04:16.090252Z","shell.execute_reply.started":"2022-03-09T02:04:12.477673Z","shell.execute_reply":"2022-03-09T02:04:16.089540Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 4: Explore Data #\n\nLet's take a moment to look at some of the images in the dataset.","metadata":{}},{"cell_type":"code","source":"\nfrom matplotlib import pyplot as plt\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case,\n                                     # these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is\n    # the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square\n    # or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n\n\ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can display a single batch of images from a dataset with another of our helper functions. The next cell will turn the dataset into an iterator of batches of 20 images.","metadata":{}},{"cell_type":"code","source":"ds_iter = iter(ds_train.unbatch().batch(20))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Use the Python `next` function to pop out the next batch in the stream and display it with the helper function.","metadata":{}},{"cell_type":"code","source":"one_batch = next(ds_iter)\ndisplay_batch_of_images(one_batch)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By defining `ds_iter` and `one_batch` in separate cells, you only need to rerun the cell above to see a new batch of images.","metadata":{}},{"cell_type":"markdown","source":"# Step 5: Define Model #\n\nNow we're ready to create a neural network for classifying images! We'll use what's known as **transfer learning**. With transfer learning, you reuse part of a pretrained model to get a head-start on a new dataset.\n\nFor the starter code, it uses a model called **VGG16** pretrained on [ImageNet](http://image-net.org/)).\n\nThe distribution strategy we created earlier contains a [context manager](https://docs.python.org/3/reference/compound_stmts.html#with), `strategy.scope`. This context manager tells TensorFlow how to divide the work of training among the eight TPU cores. When using TensorFlow with a TPU, it's important to define your model in a `strategy.scope()` context.\n\n#### Starter Code ##\n\n```python\nwith strategy.scope():\n    pretrained_model = tf.keras.applications.VGG16(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    pretrained_model.trainable = False\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        pretrained_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n```\n\nResults from last three epochs are shown: \n\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 24s 238ms/step - loss: 3.4754 - sparse_categorical_accuracy: 0.2208 - val_loss: 3.4896 - val_sparse_categorical_accuracy: 0.2231\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 23s 236ms/step - loss: 3.4629 - sparse_categorical_accuracy: 0.2204 - val_loss: 3.4852 - val_sparse_categorical_accuracy: 0.2239\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 24s 238ms/step - loss: 3.4694 - sparse_categorical_accuracy: 0.2204 - val_loss: 3.4813 - val_sparse_categorical_accuracy: 0.2249\n\nAs we can see, the accuracy is about 22%. Although it does converges -- from 5.6% to 20+%, it is far below what we want. Let us try other models. ","metadata":{}},{"cell_type":"markdown","source":"### 🔔 Modification 1: Train with Other Models ###\nWe want to experiment with [other models](https://www.tensorflow.org/api_docs/python/tf/keras/applications) included with Keras. ","metadata":{}},{"cell_type":"code","source":"# get all the models provided by tf.keras.applications\n', '.join(tf.keras.applications.__dir__())","metadata":{"execution":{"iopub.status.busy":"2022-03-08T20:04:14.384668Z","iopub.execute_input":"2022-03-08T20:04:14.384927Z","iopub.status.idle":"2022-03-08T20:04:14.392328Z","shell.execute_reply.started":"2022-03-08T20:04:14.384899Z","shell.execute_reply":"2022-03-08T20:04:14.391654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below are the code used for training different models. Most are commented out because we only test out one model each time: ","metadata":{}},{"cell_type":"markdown","source":"#### Trial 1.1: VGG16() with Trainable = True ####\nWe will start simple. We stick with VGG16() for now, but we would like to change the ```trainable``` attribute, so that it unfreezes the layers and keep changing the values. Read more [here](https://www.tensorflow.org/guide/keras/transfer_learning).\n\nInitially we also want to increase EPOCHS to 20, but since we have a limitation on how long we can use TPUs for, let us first figure about which model works best before further modifications. \n\n```python\nwith strategy.scope():\n    # VGG16(), with trainable = True\n    vgg16_model = tf.keras.applications.VGG16(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n\n    vgg16_model.trainable = True\n   \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        vgg16_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n```\n\nResults from last three epochs are shown: \n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 41s 411ms/step - loss: 2.9319 - sparse_categorical_accuracy: 0.2498 - val_loss: 2.9557 - val_sparse_categorical_accuracy: 0.2602\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 41s 412ms/step - loss: 2.9168 - sparse_categorical_accuracy: 0.2585 - val_loss: 2.9345 - val_sparse_categorical_accuracy: 0.2683\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 44s 447ms/step - loss: 2.8843 - sparse_categorical_accuracy: 0.2630 - val_loss: 2.8902 - val_sparse_categorical_accuracy: 0.2734\n\nSlight improvements. We are going from 22% to 26%, and the accuracy does increase over epochs. We decided to move on with **```Trainable=True```** for now with other models. ","metadata":{}},{"cell_type":"markdown","source":"#### Trial 1.2: VGG19() ####\n```python\nwith strategy.scope():\n    # VGG19(), with trainable = True\n    vgg19_model = tf.keras.applications.VGG19(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    vgg19_model.trainable = True\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        vgg19_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n```\nResults from last three epochs are shown: \n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 45s 452ms/step - loss: 3.6064 - sparse_categorical_accuracy: 0.1462 - val_loss: 3.5898 - val_sparse_categorical_accuracy: 0.1482\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 45s 453ms/step - loss: 3.5920 - sparse_categorical_accuracy: 0.1512 - val_loss: 3.5825 - val_sparse_categorical_accuracy: 0.1509\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 45s 453ms/step - loss: 3.5894 - sparse_categorical_accuracy: 0.1496 - val_loss: 3.5865 - val_sparse_categorical_accuracy: 0.1517\n\nOh no, the accuracy is worse than before. Let's remove VGG19() out of our list. Next!","metadata":{"execution":{"iopub.status.busy":"2022-03-07T04:10:10.540247Z","iopub.execute_input":"2022-03-07T04:10:10.540524Z","iopub.status.idle":"2022-03-07T04:10:10.552839Z","shell.execute_reply.started":"2022-03-07T04:10:10.540488Z","shell.execute_reply":"2022-03-07T04:10:10.551758Z"}}},{"cell_type":"markdown","source":"#### Trial 1.3: Xception() ####\n```python\nwith strategy.scope():\n    # Xception(), with trainable = True\n    xception_model = tf.keras.applications.xception.Xception(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    xception_model.trainable = True\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        xception_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n```\nResults from last three epochs are shown: \n\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 35s 350ms/step - loss: 0.0113 - sparse_categorical_accuracy: 0.9989 - val_loss: 0.2524 - val_sparse_categorical_accuracy: 0.9402\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 35s 359ms/step - loss: 0.0110 - sparse_categorical_accuracy: 0.9987 - val_loss: 0.2522 - val_sparse_categorical_accuracy: 0.9402\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 35s 351ms/step - loss: 0.0087 - sparse_categorical_accuracy: 0.9992 - val_loss: 0.2525 - val_sparse_categorical_accuracy: 0.9399\n \n\nWow, 94% accuracy is quite impressive impressive! Because of the accuracy is so high, we moved one step forward, which is evaluating the model with the validation set. Here is the result: \n\nWe will compare this with other models. ","metadata":{"execution":{"iopub.status.busy":"2022-03-07T04:10:10.554231Z","iopub.execute_input":"2022-03-07T04:10:10.555029Z","iopub.status.idle":"2022-03-07T04:10:10.567419Z","shell.execute_reply.started":"2022-03-07T04:10:10.554986Z","shell.execute_reply":"2022-03-07T04:10:10.566312Z"}}},{"cell_type":"markdown","source":"\n#### Trial 1.4 DenseNet201() ###\n\nThere are three different Densenet architectures provided. We decided to use DenseNet201() because it has a larger number attached to it, and that usually means it is developed more recently, so could give us better results. \n\n```python\nwith strategy.scope():\n    # DenseNet201(), with trainable = True\n    densenet_model = tf.keras.applications.densenet.DenseNet201(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        densenet_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n ```\n Results from last three epochs are shown: \n\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 54s 544ms/step - loss: 0.0245 - sparse_categorical_accuracy: 0.9983 - val_loss: 0.2274 - val_sparse_categorical_accuracy: 0.9445\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 54s 547ms/step - loss: 0.0229 - sparse_categorical_accuracy: 0.9988 - val_loss: 0.2247 - val_sparse_categorical_accuracy: 0.9461\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 54s 546ms/step - loss: 0.0240 - sparse_categorical_accuracy: 0.9985 - val_loss: 0.2258 - val_sparse_categorical_accuracy: 0.9448\n\nSome good training is happening, even compared to Xception()! ","metadata":{}},{"cell_type":"markdown","source":"#### Trial 1.5 EfficientNetB7() ####\n\nWe would like to play with EfficientNetB7, which seems to perform pretty well according to this [blog](https://www.cnblogs.com/DreamingFishZIHao/p/12982945.html). The good news is, with the upgraded TensorFlow library, EfficientNetB7() is included in keras now.\n\n```python\nwith strategy.scope():\n    # EfficientNetB7()\n    efficientnet_model = tf.keras.applications.efficientnet.EfficientNetB7(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        efficientnet_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n```\nResults from last three epochs are shown: \n\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 95s 957ms/step - loss: 0.0179 - sparse_categorical_accuracy: 0.9977 - val_loss: 0.3992 - val_sparse_categorical_accuracy: 0.9124\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 94s 952ms/step - loss: 0.0152 - sparse_categorical_accuracy: 0.9976 - val_loss: 0.4293 - val_sparse_categorical_accuracy: 0.9052\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 94s 952ms/step - loss: 0.0161 - sparse_categorical_accuracy: 0.9980 - val_loss: 0.4146 - val_sparse_categorical_accuracy: 0.9089\n\nPretty good accuracy, but not as good as DenseNet(). Also, the time it needs to train the data is longer compared to DenseNet() or Xception(). \n\nThen, when experimenting with our ⚖️ modification 2, we came across a [blog](https://towardsdatascience.com/efficient-nets-with-noisy-student-training-5ac6e239ff14), which basically says how the performance can be improved if we changed the value assigned to the ```weights``` attribute. Why not give it a try?\n\nInterestingly, the keras library does not allow us to modify the weights to be 'noisy-student'. So we need to import ```efficientnet``` and go from there. \n\n```python\nwith strategy.scope():\n    # EfficientNetB7()\n    efficientnet_model = efn.EfficientNetB7(\n        weights='noisy-student',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        efficientnet_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n```\nResults from last three epochs are shown: \n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 94s 948ms/step - loss: 0.0684 - sparse_categorical_accuracy: 0.9823 - val_loss: 0.2297 - val_sparse_categorical_accuracy: 0.9507\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 93s 944ms/step - loss: 0.0649 - sparse_categorical_accuracy: 0.9822 - val_loss: 0.2295 - val_sparse_categorical_accuracy: 0.9504\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 93s 942ms/step - loss: 0.0610 - sparse_categorical_accuracy: 0.9826 - val_loss: 0.2311 - val_sparse_categorical_accuracy: 0.9515\n\nImproved a lot (from 90% to 95%)! It seems like ```noisy-student``` is a better weight to be used for EfficientNetB7(). We will use it from now on. ","metadata":{}},{"cell_type":"markdown","source":"#### Trial 1.6: InceptionV3() ####\n\nAnother cool architecture with InceptionV3(). \n\n```python\nwith strategy.scope():\n    # InceptionV3()\n    inception_model = tf.keras.applications.inception_v3.InceptionV3(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        inception_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n```\nResults from last three epochs are shown: \n\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 32s 326ms/step - loss: 0.0301 - sparse_categorical_accuracy: 0.9963 - val_loss: 0.2770 - val_sparse_categorical_accuracy: 0.9340\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 33s 331ms/step - loss: 0.0295 - sparse_categorical_accuracy: 0.9959 - val_loss: 0.2778 - val_sparse_categorical_accuracy: 0.9348\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 33s 330ms/step - loss: 0.0234 - sparse_categorical_accuracy: 0.9983 - val_loss: 0.2788 - val_sparse_categorical_accuracy: 0.9343","metadata":{}},{"cell_type":"markdown","source":"#### Trial 1.7: MobileNetV3Small() ####\nAgain, there are several options provided by MobileNet: MobileNet(), MobileNetV2(), MobileNetV3Large(), MobileNetV3Small(). \n\nAccording to this [research paper](https://arxiv.org/pdf/1905.02244.pdf) from Cornell University, \"MobileNetV3-Small is 4.6\\% more accurate while reducing latency by 5\\% compared to MobileNetV2. MobileNetV3-Large detection is 25\\% faster at roughly the same accuracy as MobileNetV2 on COCO detection. MobileNetV3-Large LR-ASPP is 30\\% faster than MobileNetV2 R-ASPP at similar accuracy for Cityscapes segmentation.\" Therefore, we use MobileNetV3Small() for the model. \n\n```python\nwith strategy.scope():\n    # MobileNetV3Small()\n    mobilenet_model = tf.keras.applications.MobileNetV3Small(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        mobilenet_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n```\n\nResults from last three epochs are shown: \n\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 18s 184ms/step - loss: 0.1393 - sparse_categorical_accuracy: 0.9620 - val_loss: 4.8778 - val_sparse_categorical_accuracy: 0.0442\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 18s 184ms/step - loss: 0.1361 - sparse_categorical_accuracy: 0.9641 - val_loss: 4.8564 - val_sparse_categorical_accuracy: 0.0442\n- Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 18s 183ms/step - loss: 0.1393 - sparse_categorical_accuracy: 0.9657 - val_loss: 4.9026 - val_sparse_categorical_accuracy: 0.0442\n\nNot sure what exactly is happening, but the model accuracy for validation set is quite low (only 4%). ","metadata":{}},{"cell_type":"markdown","source":"#### Trial 1.8: NASNetMobile() ####\n\nNot a lot of papers are found when comparing NASNetMobile() with NasNetLarge(), so we just choose one to do the modeling. Interestingly, we cannot use imagenet as weights now because from the [documentations](https://www.tensorflow.org/api_docs/python/tf/keras/applications/nasnet/NASNetMobile), if it is, then the input_shape must be (224, 224, 3). Since our images are (512, 512, 3) instead, change the weights to be None (random initialization). \n```python\nwith strategy.scope():\n    # NASNetMobile()\n    nasnet_model = tf.keras.applications.nasnet.NASNetMobile(\n        weights=None, # cannot use imagenet weighte because input_shape is not (224, 224, 3)\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        nasnet_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n```\nResults from last three epochs are shown: \n\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 59s 597ms/step - loss: 1.4590 - sparse_categorical_accuracy: 0.6165 - val_loss: 4.4366 - val_sparse_categorical_accuracy: 0.0447\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 59s 598ms/step - loss: 1.4326 - sparse_categorical_accuracy: 0.6142 - val_loss: 4.4375 - val_sparse_categorical_accuracy: 0.0442\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 59s 595ms/step - loss: 1.3871 - sparse_categorical_accuracy: 0.6330 - val_loss: 4.4542 - val_sparse_categorical_accuracy: 0.0439\n\nAgain, though the accuracy on training data is around 63%, the performance on validation dataset is not good.","metadata":{"execution":{"iopub.status.busy":"2022-03-07T19:53:03.56732Z","iopub.execute_input":"2022-03-07T19:53:03.567646Z","iopub.status.idle":"2022-03-07T19:53:03.58324Z","shell.execute_reply.started":"2022-03-07T19:53:03.567604Z","shell.execute_reply":"2022-03-07T19:53:03.582333Z"}}},{"cell_type":"markdown","source":"#### Trial 1.9: ResNet152V2() ####\n\nResNet is a powerful deep neural network tool, where the numbers after \"ResNet\" just represent differen number of layers.\n\nAccording to another paper on [ResNet](https://arxiv.org/pdf/1512.03385.pdf), seems like ResNet-152 has the least top-1 and top-5 error rates compared to other shallower ResNet models (such as ResNet50 or ResNet101) for classification on Imagenet validation. Hence, we use ResNet152V2() here. \n\n```python\nwith strategy.scope():\n    # ResNet152V2()\n    resnet_model = tf.keras.applications.resnet_v2.ResNet152V2(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        resnet_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n```\nResults from last three epochs are shown:\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 46s 464ms/step - loss: 0.0558 - sparse_categorical_accuracy: 0.9922 - val_loss: 0.3634 - val_sparse_categorical_accuracy: 0.9119\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 46s 463ms/step - loss: 0.0491 - sparse_categorical_accuracy: 0.9935 - val_loss: 0.3672 - val_sparse_categorical_accuracy: 0.9116\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 47s 477ms/step - loss: 0.0437 - sparse_categorical_accuracy: 0.9949 - val_loss: 0.3666 - val_sparse_categorical_accuracy: 0.9111\n\n91%! Pretty good accuracy on the validation set.","metadata":{}},{"cell_type":"markdown","source":"### 🔔 Modification 1: Train with Other Models - Comparisons ###\n\nLet's do a quick comparisons for models that perform with an accuracy higher than 85% on validation set:\n\n* Xception(): f1 = 0.936, precision = 0.944, recall = 0.932\n* DenseNet201(): f1 = 0.942, precision = 0.944, recall = 0.944\n* EfficientNetB7() with weights = 'imagenet': f1 = 0.895, precision = 0.901, recall = 0.895\n* InceptionV3(): f1 = 0.925, precision = 0.930, recall = 0.926\n* RasNet152V2(): f1 = 0.898, precision = 0.912, recall = 0.891\n* (in second modification found) EfficientNetB7() with weights = 'noisy-student': f1 = 0.949, 0.948, 0.955\n\nFrom the numbers, DenseNet201() provides a good model for training. It also has a relatively fast training speed. \n\nIt is possible to use a mixture of models for predictions, and that will be an improvement to be explored in future modifications. \n","metadata":{}},{"cell_type":"markdown","source":"The `'sparse_categorical'` versions of the loss and metrics are appropriate for a classification task with more than two labels, like this one.","metadata":{}},{"cell_type":"markdown","source":"### 🚀 Modification 3: Optimizers ###\n\nHow about optimizers? What are some optimizers that are available and maybe have a better performance?\n\n#### Starter Code ####\n\n```python\nmodel.compile(\n    optimizer='adam',\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy'],\n)\n```\n\nBecause we do have time limitation on running with TPUs, we cannot train on every model that we found in the first modifications. We will start with ```Xception()``` for now because of its faster runtime among all models with relatively high accuracy. \n\nBefore we get started exploring, there is some notes on choosing the optimizers. Even though we have decided to not continue with training with class weights calculated from training dataset, there is still some facts we found out about our data. Namely, our dataset is imbalanced. From this [example on examining imabalanced data](https://www.tensorflow.org/tutorials/structured_data/imbalanced_data#examine_the_class_label_imbalance), it is mentioned that: \n> Note: Using class_weights changes the range of the loss. This may affect the stability of the training depending on the optimizer. Optimizers whose step size is dependent on the magnitude of the gradient, like tf.keras.optimizers.SGD, may fail. The optimizer used here, tf.keras.optimizers.Adam, is unaffected by the scaling change. Also note that because of the weighting, the total losses are not comparable between the two models.\n<!-- -->\n\nSo it is possible that some optimizers will perform worse than Adam, which is okay because we can always go back to it. But let's start experimenting first:\n\n```python\n# get all the models provided by tf.keras.applications\n', '.join(tf.keras.optimizers.__dir__())\n\nmodel.compile(\n    optimizer='adam',\n    # other optimizers are not use due to insignificant improvements\n#     optimizer = 'Adadelta',\n#     optimizer = 'Adagrad',\n#     optimizer = 'Adamax',\n#     optimizer = 'Ftrl',\n#     optimizer = 'Nadam',\n#     optimizer = 'RMSprop',\n#     optimizer = 'SGD',\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy'],\n)\n```\n\n#### Trial 3.0 Adam - Xception() ####\nThis is the last three epochs from 🔔 modification 1:\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 35s 350ms/step - loss: 0.0113 - sparse_categorical_accuracy: 0.9989 - val_loss: 0.2524 - val_sparse_categorical_accuracy: 0.9402\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 35s 359ms/step - loss: 0.0110 - sparse_categorical_accuracy: 0.9987 - val_loss: 0.2522 - val_sparse_categorical_accuracy: 0.9402\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 35s 351ms/step - loss: 0.0087 - sparse_categorical_accuracy: 0.9992 - val_loss: 0.2525 - val_sparse_categorical_accuracy: 0.9399\n\n#### Trial 3.1 Adadelta - Xception() ####\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 35s 352ms/step - loss: 4.5748 - sparse_categorical_accuracy: 0.0152 - val_loss: 4.5683 - val_sparse_categorical_accuracy: 0.0180\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 35s 352ms/step - loss: 4.5681 - sparse_categorical_accuracy: 0.0162 - val_loss: 4.5681 - val_sparse_categorical_accuracy: 0.0183\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 39s 394ms/step - loss: 4.5749 - sparse_categorical_accuracy: 0.0153 - val_loss: 4.5677 - val_sparse_categorical_accuracy: 0.0189\n\nNo luck here. All the evaluations are pretty low (1%, seriously?). Next!\n\n#### Trial 3.2 Adagrad - Xception() ####\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 34s 348ms/step - loss: 4.1973 - sparse_categorical_accuracy: 0.1020 - val_loss: 4.1869 - val_sparse_categorical_accuracy: 0.1048\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 36s 359ms/step - loss: 4.1956 - sparse_categorical_accuracy: 0.1079 - val_loss: 4.1858 - val_sparse_categorical_accuracy: 0.1064\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 35s 358ms/step - loss: 4.1915 - sparse_categorical_accuracy: 0.1073 - val_loss: 4.1826 - val_sparse_categorical_accuracy: 0.1048\n\nBetter, but not actually. 10% is still pretty low. \n\n#### Trial 3.3 Adamax - Xception() ####\n*  Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 35s 352ms/step - loss: 0.0413 - sparse_categorical_accuracy: 0.9965 - val_loss: 0.2505 - val_sparse_categorical_accuracy: 0.9348\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 35s 354ms/step - loss: 0.0395 - sparse_categorical_accuracy: 0.9966 - val_loss: 0.2509 - val_sparse_categorical_accuracy: 0.9356\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 36s 366ms/step - loss: 0.0397 - sparse_categorical_accuracy: 0.9974 - val_loss: 0.2505 - val_sparse_categorical_accuracy: 0.9353\n\nMuch better results compared to the previous two. Therefore, we continue the evaluation process to get its f1, precision, and recall.\n\n#### Trial 3.4 Ftrl - Xception() ####\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 35s 352ms/step - loss: 4.6439 - sparse_categorical_accuracy: 0.0651 - val_loss: 4.6438 - val_sparse_categorical_accuracy: 0.0614\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 35s 351ms/step - loss: 4.6439 - sparse_categorical_accuracy: 0.0613 - val_loss: 4.6439 - val_sparse_categorical_accuracy: 0.0614\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 35s 352ms/step - loss: 4.6439 - sparse_categorical_accuracy: 0.0591 - val_loss: 4.6439 - val_sparse_categorical_accuracy: 0.0614\n\nToo bad, guess we have to throw this one away as well. \n\n#### Trial 3.5 Nadam - Xception() ####\n* Epoch 00009: LearningRateScheduler reducing learning rate to 3.0480000000000006e-05. 99/99 [==============================] - 36s 365ms/step - loss: 0.0097 - sparse_categorical_accuracy: 0.9992 - val_loss: 0.2293 - val_sparse_categorical_accuracy: 0.9448\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 35s 354ms/step - loss: 0.0089 - sparse_categorical_accuracy: 0.9990 - val_loss: 0.2297 - val_sparse_categorical_accuracy: 0.9445\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 35s 350ms/step - loss: 0.0083 - sparse_categorical_accuracy: 0.9993 - val_loss: 0.2318 - val_sparse_categorical_accuracy: 0.9448\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 35s 358ms/step - loss: 0.0076 - sparse_categorical_accuracy: 0.9998 - val_loss: 0.2319 - val_sparse_categorical_accuracy: 0.9459\n\nSomething nice is happening!\n\n#### Trial 3.6 RMSprop - Xception() ####\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 34s 347ms/step - loss: 0.0046 - sparse_categorical_accuracy: 0.9998 - val_loss: 0.2784 - val_sparse_categorical_accuracy: 0.9407\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 35s 354ms/step - loss: 0.0031 - sparse_categorical_accuracy: 0.9998 - val_loss: 0.2741 - val_sparse_categorical_accuracy: 0.9423\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 35s 352ms/step - loss: 0.0022 - sparse_categorical_accuracy: 0.9998 - val_loss: 0.2750 - val_sparse_categorical_accuracy: 0.9421\n\nAlso great job. Proceed with computing f1, precision, and recall. \n\n#### Trial 3.7 SGD - Xception() ####\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 34s 343ms/step - loss: 3.6998 - sparse_categorical_accuracy: 0.2574 - val_loss: 3.6428 - val_sparse_categorical_accuracy: 0.2740\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 34s 343ms/step - loss: 3.6812 - sparse_categorical_accuracy: 0.2606 - val_loss: 3.6416 - val_sparse_categorical_accuracy: 0.2729\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 34s 346ms/step - loss: 3.6767 - sparse_categorical_accuracy: 0.2592 - val_loss: 3.6417 - val_sparse_categorical_accuracy: 0.2737\n\nNot good. Let's put it away.\n\n\n### 🚀 Modification 3: Optimizers - Comparisons ###\n\nAgain, let's do a comparison of the optimizers with relatively high accuracy on validation set:\n\nRecall that we only train these different optimizers on one model Xception(), which can give different results when trained with DenseNet201() or EfficientNetB7(). That can be a topic to be explored later on. \n\n* Adam: f1 = 0.936, precision = 0.944, recall = 0.932\n* Adamax: f1 = 0.921, precision = 0.933, recall = 0.918\n* Nadam: f1 = 0.937, precision = 0.942, recall = 0.936\n* RMSprop: f1 = 0.937, precision = 0.941, recall = 0.937\n\nThe improvements by changing the optimizers is not significant either. Since we cannot predict how the optimizers will perform on DenseNet201() and EfficientNetB7(), we will stick with ```Adam```, which we know for sure that have a good performance on the other two models. ","metadata":{}},{"cell_type":"markdown","source":"# Step 6: Training #\n\n## Learning Rate Schedule ##\n\nWe'll train this network with a special learning rate schedule.","metadata":{}},{"cell_type":"markdown","source":"###  📈 Modification 6: Increase EPOCHS ###\n\nIn the starter code, the number of EPOCHS is set to 12We stick with 12 for most iterations so far because if we try more EPOCHS, the time needed to train will increase. \n\nSince Kaggle has a limitation on 20 hours of TPUs/week, we do not want to spend too much time on \"picking the models\" round. Now we are kinda happy with what we get so far, let's try increase that EPOCHS number to 20. ","metadata":{}},{"cell_type":"code","source":"EPOCHS = 12\n# EPOCHS = 20 # Modification 6\n\n# Learning Rate Schedule for Fine Tuning #\ndef exponential_lr(epoch,\n                   start_lr = 0.00001, min_lr = 0.00001, max_lr = 0.00005,\n                   rampup_epochs = 5, sustain_epochs = 0,\n                   exp_decay = 0.8):\n\n    def lr(epoch, start_lr, min_lr, max_lr, rampup_epochs, sustain_epochs, exp_decay):\n        # linear increase from start to rampup_epochs\n        if epoch < rampup_epochs:\n            lr = ((max_lr - start_lr) /\n                  rampup_epochs * epoch + start_lr)\n        # constant max_lr during sustain_epochs\n        elif epoch < rampup_epochs + sustain_epochs:\n            lr = max_lr\n        # exponential decay towards min_lr\n        else:\n            lr = ((max_lr - min_lr) *\n                  exp_decay**(epoch - rampup_epochs - sustain_epochs) +\n                  min_lr)\n        return lr\n    return lr(epoch,\n              start_lr,\n              min_lr,\n              max_lr,\n              rampup_epochs,\n              sustain_epochs,\n              exp_decay)\n\nlr_callback = tf.keras.callbacks.LearningRateScheduler(exponential_lr, verbose=True)\n\nrng = [i for i in range(EPOCHS)]\ny = [exponential_lr(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-08T20:04:14.3936Z","iopub.execute_input":"2022-03-08T20:04:14.393863Z","iopub.status.idle":"2022-03-08T20:04:14.657626Z","shell.execute_reply.started":"2022-03-08T20:04:14.393834Z","shell.execute_reply":"2022-03-08T20:04:14.656857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"###  📈 Modification 6: Increase EPOCHS - Comparisons ###\n\nLet's do a quick comparisons on the validation set evaluations (same combo are used Xception() + EfficientNetB7()):\n* 12 EPOCHS: f1 = 0.961, precision = 0.962, recall = 0.962\n* 20 EPOCHS: f1 = 0.959, precision = 0.960, recall = 0.961\n\nThe score submitted to the competition does not change a lot either; it even goes down a bit from 0.95462 to 0.95417. Hence, we will switch back to 12 EPOCHS. \n\n## Fit Model ##\n\nAnd now we're ready to train the model. After defining a few parameters, we're good to go!","metadata":{}},{"cell_type":"code","source":"# Define training epochs\nEPOCHS = 12\n# EPOCHS = 20 # Modification 6\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE","metadata":{"execution":{"iopub.status.busy":"2022-03-08T20:04:14.658632Z","iopub.execute_input":"2022-03-08T20:04:14.659245Z","iopub.status.idle":"2022-03-08T20:04:14.663887Z","shell.execute_reply.started":"2022-03-08T20:04:14.659208Z","shell.execute_reply":"2022-03-08T20:04:14.662641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ⚖️ Modification 2: Calculate Weights ###\n\nInside ```model.fit()```, there is an attribute called ```class_weight```. According to Keras [doc](https://keras.io/models/sequential/): this attribute is an \"optional dictionary mapping class indices (integers) to a weight (float) value, used for weighting the loss function (during training only)\". It is set up for [classification on imbalanced data](https://www.tensorflow.org/tutorials/structured_data/imbalanced_data). \n\nThe general idea is that since we do not have many samples for those less represented classes, we should have the classifer weight heavily on those few examples that are available. In other words, we want the models to \"pay more attention\" to example from an under-represented class. \n\nMaybe we can customize this attribute? Let us start by looking at the dataset imbalance:\n\n```python\nfrom collections import Counter\nimport gc\ngc.enable()\n\ntraining_data_copy = load_dataset(TRAINING_FILENAMES, labeled = True, ordered = False)\n\nlabel_counter = Counter()\nfor images, labels in training_data_copy:\n    label_counter.update([labels.numpy()])\n\ndel training_data_copy\n\nTARGET_NUM_PER_CLASS = NUM_TRAINING_IMAGES / 104 # if balanced, every class should have about 122 images\n\ndef get_weight_for_class(class_id):\n    counting = label_counter[class_id]\n    weight = TARGET_NUM_PER_CLASS / counting\n    return weight\n\ntraining_class_weights = {class_id: get_weight_for_class(class_id) for class_id in range(104)}\n```\n\nA quick graph to show different weights (code from this [notebook](https://www.kaggle.com/georgezoto/computer-vision-petals-to-the-metal/notebook#Step-5:-Define-Model)):\n```python\nimport seaborn as sns\nfrom matplotlib import cm\n\ndata = pd.DataFrame.from_dict(training_class_weights, orient='index', columns=['class_weight'])\nplt.figure(figsize=(30, 9))\n\n#barplot color based on value\nbplot = sns.barplot(x=data.index, y='class_weight', data=data, palette= cm.Blues(data['class_weight']*0.15));\nfor p in bplot.patches:\n    bplot.annotate(format(p.get_height(), '.1f'), \n                   (p.get_x() + p.get_width() / 2., p.get_height()), \n                   ha = 'center', va = 'center', \n                   xytext = (0, 9), \n                   textcoords = 'offset points')\nplt.xlabel(\"Class\", size=14)\nplt.ylabel(\"Class weight (inverse of %)\", size=14)\n```\n\nLet's train the model with class weights and evaluate it now. See if it affects our predictions.\n\nWe pick the models that perform pretty well in the last round: Xception(), DenseNet201(), EfficientNetB7(), Inception2(), and RasNet152V2().\n\n```python\nhistory = model.fit(\n    ds_train,\n    validation_data=ds_valid,\n    epochs=EPOCHS,\n    steps_per_epoch=STEPS_PER_EPOCH,\n    callbacks=[lr_callback],\n    # modification 2: class weights (abandoned because no improvements)\n    # class_weight = training_class_weights\n)\n```\n\nThe last three epochs for each models are shown below:\n\n#### Trial 2.1 Xception() ####\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 35s 357ms/step - loss: 0.0211 - sparse_categorical_accuracy: 0.9918 - val_loss: 0.2537 - val_sparse_categorical_accuracy: 0.9386\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 36s 361ms/step - loss: 0.0201 - sparse_categorical_accuracy: 0.9919 - val_loss: 0.2517 - val_sparse_categorical_accuracy: 0.9378\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 35s 351ms/step - loss: 0.0217 - sparse_categorical_accuracy: 0.9905 - val_loss: 0.2528 - val_sparse_categorical_accuracy: 0.9386\n\n#### Trial 2.2 DenseNet201() ####\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 66s 668ms/step - loss: 0.0612 - sparse_categorical_accuracy: 0.9767 - val_loss: 0.2711 - val_sparse_categorical_accuracy: 0.9337\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 54s 544ms/step - loss: 0.0562 - sparse_categorical_accuracy: 0.9817 - val_loss: 0.2689 - val_sparse_categorical_accuracy: 0.9337\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 54s 549ms/step - loss: 0.0520 - sparse_categorical_accuracy: 0.9824 - val_loss: 0.2669 - val_sparse_categorical_accuracy: 0.9356\n\n\n#### Trial 2.3 EfficientNetB7() with weights = 'imagenet' ####\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 95s 956ms/step - loss: 0.0461 - sparse_categorical_accuracy: 0.9766 - val_loss: 0.4328 - val_sparse_categorical_accuracy: 0.8955\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 94s 948ms/step - loss: 0.0440 - sparse_categorical_accuracy: 0.9759 - val_loss: 0.4292 - val_sparse_categorical_accuracy: 0.8963\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 94s 951ms/step - loss: 0.0380 - sparse_categorical_accuracy: 0.9803 - val_loss: 0.4328 - val_sparse_categorical_accuracy: 0.8947\n\n#### Trial 2.4 InceptionV2() ####\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 32s 325ms/step - loss: 0.0986 - sparse_categorical_accuracy: 0.9568 - val_loss: 0.3714 - val_sparse_categorical_accuracy: 0.9068\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 32s 323ms/step - loss: 0.0846 - sparse_categorical_accuracy: 0.9621 - val_loss: 0.3675 - val_sparse_categorical_accuracy: 0.9079\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 32s 328ms/step - loss: 0.0790 - sparse_categorical_accuracy: 0.9633 - val_loss: 0.3650 - val_sparse_categorical_accuracy: 0.9062\n\n\n#### Trial 2.5 RasNet152V2() ####\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 46s 467ms/step - loss: 0.3368 - sparse_categorical_accuracy: 0.8629 - val_loss: 0.7441 - val_sparse_categorical_accuracy: 0.8079\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 46s 464ms/step - loss: 0.3156 - sparse_categorical_accuracy: 0.8659 - val_loss: 0.7330 - val_sparse_categorical_accuracy: 0.8093\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 46s 462ms/step - loss: 0.2884 - sparse_categorical_accuracy: 0.8798 - val_loss: 0.7148 - val_sparse_categorical_accuracy: 0.8155\n\n#### Trial 2.6 EfficientNetB7() with weights = 'noisy-student' ####\n* Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05. 99/99 [==============================] - 94s 947ms/step - loss: 0.1077 - sparse_categorical_accuracy: 0.9444 - val_loss: 0.2795 - val_sparse_categorical_accuracy: 0.9405\n* Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05. 99/99 [==============================] - 94s 948ms/step - loss: 0.1126 - sparse_categorical_accuracy: 0.9487 - val_loss: 0.2788 - val_sparse_categorical_accuracy: 0.9405\n* Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05. 99/99 [==============================] - 94s 947ms/step - loss: 0.0948 - sparse_categorical_accuracy: 0.9512 - val_loss: 0.2755 - val_sparse_categorical_accuracy: 0.9418\n\n\n### ⚖️ Modification 2: Calculate Weights - Comparisons ###\n\nAnother quick comparison on f1, precision, and recall (before -> after applying class weights): \n* Xception(): f1 = 0.936 -> 0.941, precision = 0.944 -> 0.940, recall = 0.932 -> 0.946\n* DenseNet201(): f1 = 0.942 -> 0.935, precision = 0.944 -> 0.931, recall = 0.944 -> 0.944\n* EfficientNetB7() with weights = 'imagenet': f1 = 0.895 -> 0.901, precision = 0.901 -> 0.894, recall = 0.895 -> 0.914\n* InceptionV3(): f1 = 0.925 -> 0.907, precision = 0.930 -> 0.899, recall = 0.926 -> 0.922\n* RasNet152V2(): f1 = 0.898 -> 0.812, precision = 0.912 -> 0.801, recall = 0.891 -> 0.893\n* EfficientNetB7() with weights = 'noisy-student': f1 = 0.949 -> 0.941, 0.948 -> 0.932, 0.955 -> 0.955\n\nFrom the results, we can see that not everything is improved with customized weights. Even when there is an improvement, the change is quite small. Therefore, we are not going to use training_class_weights in future iterations. \n\nWe also decide to remove EfficientNetB7() with weights = 'imagenet', InceptionV3() and RasNet152V2() from the list of usable models because their accuracy is the relatively low among all in both rounds now. \n\nEven though this modification is not applicable, we are able to find a better model when changing the weights for EfficientNetB7(). Worth it!","metadata":{}},{"cell_type":"markdown","source":"### Loss and Metrics ###\n\nThis next cell shows how the loss and metrics progressed during training.\n\n```python\ndisplay_training_curves(\n    history.history['loss'],\n    history.history['val_loss'],\n    'loss',\n    211,\n)\ndisplay_training_curves(\n    history.history['sparse_categorical_accuracy'],\n    history.history['val_sparse_categorical_accuracy'],\n    'accuracy',\n    212,\n)\n```","metadata":{}},{"cell_type":"markdown","source":"### 🎨 Modification 4: Mixing Models ###\n\nInspired by this [notebook](https://www.kaggle.com/wrrosa/tpu-enet-b7-densenet#Finding-best-alpha), we want to try out mix and match models together. We have reduced our models to 3 now (Xception(), DenseNet201(), and EfficientNetB7()), so there will be 2! = 3 combinations of models. Let's examine each pair, and compare their performance.\n\n**Combo 1: Xception() + DenseNet201()**\n\n```python\nwith strategy.scope():\n    # Model 1: Xception()\n    # Xception(), with trainable = True\n    xception_model = tf.keras.applications.xception.Xception(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    xception_model.trainable = True\n    \n    model1 = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        xception_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n    \n    # Model 2: DenseNet201()\n    # DenseNet201(), with trainable = True\n    densenet_model = tf.keras.applications.densenet.DenseNet201(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    \n    model2 = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        densenet_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n```","metadata":{}},{"cell_type":"markdown","source":"**Combo 2: Xception() + EfficientNetB7()**\n\n```python\nwith strategy.scope():\n    # Model 1: Xception()\n    # Xception(), with trainable = True\n    xception_model = tf.keras.applications.xception.Xception(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    xception_model.trainable = True\n    \n    model1 = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        xception_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n    \n    # Model 2: EfficientNetB7()\n    efficientnet_model = efn.EfficientNetB7(\n        weights='noisy-student',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    \n    model2 = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        efficientnet_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n```","metadata":{}},{"cell_type":"markdown","source":"**Combo 3: DenseNet201() + EfficientNetB7()**\n\n```python\nwith strategy.scope():\n    # Model 1: DenseNet201()\n    # DenseNet201(), with trainable = True\n    densenet_model = tf.keras.applications.densenet.DenseNet201(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    \n    model1 = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        densenet_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n    \n    # Model 2: EfficientNetB7()\n    efficientnet_model = efn.EfficientNetB7(\n        weights='noisy-student',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    \n    model2 = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        efficientnet_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n```","metadata":{}},{"cell_type":"code","source":"\nwith strategy.scope():\n    # Model 1: Xception()\n    # Xception(), with trainable = True\n    xception_model = tf.keras.applications.xception.Xception(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    xception_model.trainable = True\n    \n    model1 = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        xception_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n    \n    # Model 2: EfficientNetB7()\n    efficientnet_model = efn.EfficientNetB7(\n        weights='noisy-student',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    \n    model2 = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        efficientnet_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])","metadata":{"execution":{"iopub.status.busy":"2022-03-08T20:04:14.666853Z","iopub.execute_input":"2022-03-08T20:04:14.667129Z","iopub.status.idle":"2022-03-08T20:05:09.919687Z","shell.execute_reply.started":"2022-03-08T20:04:14.667088Z","shell.execute_reply":"2022-03-08T20:05:09.918705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that we have defined the two models we want to mix, we need to train the models separately. ","metadata":{}},{"cell_type":"code","source":"# train model 1\nmodel1.compile(\n    optimizer='adam',\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy'],\n)\n\nhistory = model1.fit(\n    ds_train,\n    validation_data=ds_valid,\n    epochs=EPOCHS,\n    steps_per_epoch=STEPS_PER_EPOCH,\n    callbacks=[lr_callback],\n)\n\n# display curves for model 1\ndisplay_training_curves(\n    history.history['loss'],\n    history.history['val_loss'],\n    'loss',\n    211,\n)\ndisplay_training_curves(\n    history.history['sparse_categorical_accuracy'],\n    history.history['val_sparse_categorical_accuracy'],\n    'accuracy',\n    212,\n)\n\n# train model 2\nmodel2.compile(\n    optimizer='adam',\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy'],\n)\n\nhistory2 = model2.fit(\n    ds_train,\n    validation_data=ds_valid,\n    epochs=EPOCHS,\n    steps_per_epoch=STEPS_PER_EPOCH,\n    callbacks=[lr_callback],\n)\n\n# display curves for model 2\ndisplay_training_curves(\n    history2.history['loss'],\n    history2.history['val_loss'],\n    'loss',\n    211,\n)\ndisplay_training_curves(\n    history2.history['sparse_categorical_accuracy'],\n    history2.history['val_sparse_categorical_accuracy'],\n    'accuracy',\n    212,\n)","metadata":{"execution":{"iopub.status.busy":"2022-03-08T20:05:09.92103Z","iopub.execute_input":"2022-03-08T20:05:09.921288Z","iopub.status.idle":"2022-03-08T20:13:19.943229Z","shell.execute_reply.started":"2022-03-08T20:05:09.921256Z","shell.execute_reply":"2022-03-08T20:13:19.941857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, we need to find the best_alpha to set the weight between the two models. We adapted the code from this [notebook](https://www.kaggle.com/wrrosa/tpu-enet-b7-densenet#Finding-best-alpha) on how to calculate the alpha that works the best. ","metadata":{}},{"cell_type":"code","source":"# Code adpated from: https://www.kaggle.com/wrrosa/tpu-enet-b7-densenet#Finding-best-alpha\nfrom sklearn.metrics import f1_score\ndef find_best_alpha(model_lst):\n    # get validation dataset; because we are splitting the dataset and iterating separately on images and labels, order matters.\n    valid_data = get_validation_dataset(ordered=True)\n    images_data = valid_data.map(lambda image, label: image)\n    labels_data = valid_data.map(lambda image, label: label).unbatch()\n    valid_true_labels = next(iter(labels_data.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch\n    \n    # make predictions based on given models\n    pred = []\n    for model in model_lst:\n        pred.append(model.predict(images_data))\n\n    scores = []\n    for alpha in np.linspace(0,1,100):\n        preds = np.argmax(alpha * pred[0]+ (1-alpha) * pred[1], axis=-1)\n        scores.append(f1_score(valid_true_labels, preds, labels=range(len(CLASSES)), average='macro'))\n\n    best_alpha = np.argmax(scores)/100\n    return best_alpha\n\nmodels = []\nmodels.append(model1)\nmodels.append(model2)\nbest_alpha = find_best_alpha(models)\nprint(best_alpha)","metadata":{"execution":{"iopub.status.busy":"2022-03-08T20:13:19.94432Z","iopub.status.idle":"2022-03-08T20:13:19.944685Z","shell.execute_reply.started":"2022-03-08T20:13:19.944486Z","shell.execute_reply":"2022-03-08T20:13:19.944503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 🎨 Modification 4: Mixing Models - Comparisons ###\nLet's do another round of comparisons on these three combinations:\n* Xception() + DenseNet201(): f1 = 0.95, precision = 0.952, recall = 0.952\n* Xception() + EfficientNetB7(): f1 = 0.961, precision = 0.962, recall = 0.962\n* DenseNet201() + EfficientNetB7(): f1 = 0.956, precision = 0.954, recall = 0.960\n\nSeems like the Xception() + EfficientNetB7() performs the best. Let's submit that! Indeed, the combination of Xception() and EfficientNetB7() gives us the best result we have so far -- a score of **0.95303**!\n\nAs for reference, the score by submitting DenseNet201() + EfficientNetB7() is 0.94967, which is also pretty good, but not as big as an improvement compared to the combination above: Xception() + EfficientNetB7(). We will stick with this combo for now.","metadata":{}},{"cell_type":"markdown","source":"# Step 7: Evaluate Predictions #\n\nBefore making your final predictions on the test set, it's a good idea to evaluate your model's predictions on the validation set. This can help you diagnose problems in training or suggest ways your model could be improved. We'll look at two common ways of validation: plotting the **confusion matrix** and **visual validation**.","metadata":{}},{"cell_type":"code","source":"\nimport matplotlib.pyplot as plt\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\n\ndef display_confusion_matrix(cmat, score, precision, recall):\n    plt.figure(figsize=(15,15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    plt.show()\n    \ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-08T20:13:19.946079Z","iopub.status.idle":"2022-03-08T20:13:19.946405Z","shell.execute_reply.started":"2022-03-08T20:13:19.946244Z","shell.execute_reply":"2022-03-08T20:13:19.946259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Confusion Matrix ##\n\nA [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) shows the actual class of an image tabulated against its predicted class. It is one of the best tools you have for evaluating the performance of a classifier.\n\nThe following cell does some processing on the validation data and then creates the matrix with the `confusion_matrix` function included in [`scikit-learn`](https://scikit-learn.org/stable/index.html).","metadata":{}},{"cell_type":"code","source":"cmdataset = get_validation_dataset(ordered=True)\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\n\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy()\n# Modification 4: mix two models\ncm_prob1 = model1.predict(images_ds)\ncm_prob2 = model2.predict(images_ds)\ncm_probabilities = best_alpha * cm_prob1 + (1 - best_alpha) * cm_prob2\n# cm_probabilities = model.predict(images_ds) # starter code\ncm_predictions = np.argmax(cm_probabilities, axis=-1)\n\nlabels = range(len(CLASSES))\ncmat = confusion_matrix(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n)\ncmat = (cmat.T / cmat.sum(axis=1)).T # normalize","metadata":{"execution":{"iopub.status.busy":"2022-03-08T20:13:19.949289Z","iopub.status.idle":"2022-03-08T20:13:19.949798Z","shell.execute_reply.started":"2022-03-08T20:13:19.94951Z","shell.execute_reply":"2022-03-08T20:13:19.949552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You might be familiar with metrics like [F1-score](https://en.wikipedia.org/wiki/F1_score) or [precision and recall](https://en.wikipedia.org/wiki/Precision_and_recall). This cell will compute these metrics and display them with a plot of the confusion matrix. (These metrics are defined in the Scikit-learn module `sklearn.metrics`; we've imported them in the helper script for you.)","metadata":{}},{"cell_type":"code","source":"score = f1_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\nprecision = precision_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\nrecall = recall_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\ndisplay_confusion_matrix(cmat, score, precision, recall)","metadata":{"execution":{"iopub.status.busy":"2022-03-08T20:13:19.950963Z","iopub.status.idle":"2022-03-08T20:13:19.951444Z","shell.execute_reply.started":"2022-03-08T20:13:19.95119Z","shell.execute_reply":"2022-03-08T20:13:19.951215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visual Validation ##\n\nIt can also be helpful to look at some examples from the validation set and see what class your model predicted. This can help reveal patterns in the kinds of images your model has trouble with.\n\nThis cell will set up the validation set to display 20 images at a time -- you can change this to display more or fewer, if you like.","metadata":{}},{"cell_type":"code","source":"dataset = get_validation_dataset()\ndataset = dataset.unbatch().batch(20)\nbatch = iter(dataset)","metadata":{"execution":{"iopub.status.busy":"2022-03-08T20:13:19.952936Z","iopub.status.idle":"2022-03-08T20:13:19.953401Z","shell.execute_reply.started":"2022-03-08T20:13:19.953149Z","shell.execute_reply":"2022-03-08T20:13:19.953173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And here is a set of flowers with their predicted species. Run the cell again to see another set.","metadata":{}},{"cell_type":"code","source":"images, labels = next(batch)\n# Modification 4\nprob1 = model1.predict(images)\nprob2 = model2.predict(images)\n# probabilities = model.predict(images)\nprobabilities = best_alpha * prob1 + (1 - best_alpha) * prob2\npredictions = np.argmax(probabilities, axis=-1)\ndisplay_batch_of_images((images, labels), predictions)","metadata":{"execution":{"iopub.status.busy":"2022-03-08T20:13:19.954985Z","iopub.status.idle":"2022-03-08T20:13:19.955477Z","shell.execute_reply.started":"2022-03-08T20:13:19.955201Z","shell.execute_reply":"2022-03-08T20:13:19.955226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 8: Make Test Predictions #\n\nOnce you're satisfied with everything, you're ready to make predictions on the test set.","metadata":{}},{"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True)\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\n# Modification 4: mix the two models\nprob1 = model1.predict(test_images_ds)\nprob2 = model2.predict(test_images_ds)\nprobabilities = best_alpha * prob1 + (1 - best_alpha) * prob2\n# probabilities = model.predict(test_images_ds)\npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)","metadata":{"execution":{"iopub.status.busy":"2022-03-08T20:13:19.956644Z","iopub.status.idle":"2022-03-08T20:13:19.957113Z","shell.execute_reply.started":"2022-03-08T20:13:19.956858Z","shell.execute_reply":"2022-03-08T20:13:19.956883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll generate a file `submission.csv`. This file is what you'll submit to get your score on the leaderboard.","metadata":{}},{"cell_type":"code","source":"print('Generating submission.csv file...')\n\n# Get image ids from test set and convert to unicode\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n\n# Write the submission file\nnp.savetxt(\n    'submission.csv',\n    np.rec.fromarrays([test_ids, predictions]),\n    fmt=['%s', '%d'],\n    delimiter=',',\n    header='id,label',\n    comments='',\n)\n\n# Look at the first few predictions\n!head submission.csv","metadata":{"execution":{"iopub.status.busy":"2022-03-08T20:13:19.958095Z","iopub.status.idle":"2022-03-08T20:13:19.95847Z","shell.execute_reply.started":"2022-03-08T20:13:19.958283Z","shell.execute_reply":"2022-03-08T20:13:19.958307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 9: Submissions #\n\nAt this point, we have made 5 submissions to the competitions:\n* Version 1: Starter code, score 0.04083\n* Version 3: DenseNet201(), score 0.94145\n* Version 9: mix DenseNet201() + EfficientNetB7('noisy-student'), score 0.94967\n* Version 11: mix Xception() + DenseNet201(), score 0.95303\n* Version 12: random blocking + mix Xception() + DenseNet201(), score 0.95462\n","metadata":{}},{"cell_type":"markdown","source":"# Step 10 Potential Modifications #\n\nWe listed some modifications that can be made for better results:\n\n### 🥇 Potential Modification 8: Find Best Optimizer for Each Model ###\n\nIn 🚀 modification 3, we experiment with different optimizers on Xception() model, and use the result as a general rule for other models as well. This may not be true, though. If time allows, we will also figure out the best optimizer for each of Xception(), DenseNet201() and EfficientNetB7(). ","metadata":{}}]}