{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction","metadata":{}},{"cell_type":"markdown","source":"The challenge of [**Petals to the Metal - Flower Classification on TPU**](https://www.kaggle.com/c/tpu-getting-started/) is to build a machine learning model that classifies 104 type of flowers in a test dataset on Kaggle.com.  I have learned a lot when exploring different techniques to build, train and hypertune two convolutional neural networks.   I wrote a report covering the purpose, scope, methodology, results, conclusions of this study, and lessons learned from participating in this code competition.  Please take a look at my [**report**](https://saukha.weebly.com/uploads/1/1/8/0/118025626/flower_classification_cnn_-_sau_kha_fall_2020_rev.3.pdf), which is included in [**my porfolio**](https://saukha.weebly.com/portfolio.html) and leave me some comments below.\n\n#### First Notebook  \nAfter briefly exploring different pretrained models and differnt data dimensions, I built my first model with pretrained DenseNet201 on 512x512 data.  I used external data, data augmentation and L2 kernel regularizer to overcome overfitting. I also learned that adding a dropout layer and where to place it could make a difference.  (See [my first notebook](https://www.kaggle.com/saukha/petals-to-the-metals-flower-classification) on Kaggle.)   \n\n#### Second Notebook  \nIn this notebook, I built on EfficientNetB7 as the pretrained base model, tried different amount of external data and other techniques.  I used various random data augmentation techniques and compared results to having them turned off, repositioned the dropout layer and compared results using dropout rate ranging from 0 to 0.5, added a BatchNorm layer, tried different factors for L2 regularizer for the kernel of the dense layer, and lastly, tried using \"noisy-student\" versus \"imagenet\" for initialized weights.  So far, I got the best score of 0.95403 with just a little bit external data (no augmentation) with the following hypertuning:  \n* initialize with \"imagenet\"  \n* 512x512 image size  \n* no augmentation  \n* not a lot of external data: just the 941 images from tf-flower-photo-tfrec/oxford_102_no_test/tfrecords-jpeg-512x512/ \n* added a dropout layer on top of the EfficientNetB7 base, with 0.4 drop rate  \n* followed by Global_Average_Pooling2D layer and a batch normalization layer   \n* and then the final dense layer, with a factor 0.035 for the L2 regularizer and \"softmax\" for activation  \n\nI think EfficientNet rightfully earned its name \"Efficient\".  Even though there are seven blocks of a total of 806 layers at the base, plus a few layers I added on top, my model finished training 35/35 epochs in 5379.3 seconds.  The val_loss metrics are still in a gradually decreasing trend.  I could have let it train for more epochs to get a better score for a longer run-time but within 180 minutes.  Or, I could use the time quota to crank up just a little bit on the regularizer or dropout rate.  Anyway, after exploring and comparing with EfficientNetB7 with DenseNet201, I will choose EfficientNet over DenseNet201.  There are more flexibility as far as how many blocks of layers to use for your model: EfficientNet family has models lined up from B0 through B7. So, you have an option of how many layers to use as your base model.  See [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://arxiv.org/abs/1905.11946), by Mingxing Tan, Quoc V. Le, arxiv.org, last \nupdate 11 September 2020, v5. \n\n#### Model Improvement   \nI highly recommend to check out the following to take this code challenge to the next level.  There is a nice tutorial on TensorFlow.org.\n* [tfa.image.random_cutout](https://www.tensorflow.org/addons/api_docs/python/tfa/image/random_cutout?hl=en),  \n* [tf.image.crop_to_bounding_box](https://www.tensorflow.org/api_docs/python/tf/image/crop_to_bounding_box?version=nightly), and  \n* [tf.image.sample_distorted_bounding_box](https://www.tensorflow.org/api_docs/python/tf/image/sample_distorted_bounding_box?version=nightly), [tf.image.draw_bounding_boxes](https://www.tensorflow.org/api_docs/python/tf/image/draw_bounding_boxes), and [tf.slice](https://www.tensorflow.org/api_docs/python/tf/slice).    \n\n**Data Sources:**  \n[Petals to the Metal - Flower Classification on TPU](https://www.kaggle.com/c/tpu-getting-started/data) - for this challenge  \n[tf_flower_photo_tfrec](https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec) - external data shared by [Kirill Blinov](https://www.kaggle.com/kirillblinov)    \n\n**Refernce:**  \n[A Simple Petals TF 2.2 notebook](https://www.kaggle.com/philculliton/a-simple-petals-tf-2-2-notebook) by [Phil Culliton](https://www.kaggle.com/philculliton)  \n[Create Your First Submission](https://www.kaggle.com/ryanholbrook/create-your-first-submission) by [Ryan Holbrook](https://www.kaggle.com/ryanholbrook). ","metadata":{}},{"cell_type":"markdown","source":"# Step 1a: Install new libraries","metadata":{}},{"cell_type":"code","source":"# upgrade pip if needed\n#!pip install --upgrade pip  # get the latest version\n# install tensorflow-addons\n!pip install -q -U tensorflow-addons\n# install EfficientNet\n!pip install -q efficientnet","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 1b: Import Libraries","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed on Kaggle\n# It is defined by the [kaggle/python Docker image](https://github.com/kaggle/docker-python)\nimport re, os\nimport numpy as np \nimport seaborn as sns\nimport tensorflow as tf\nimport matplotlib.pyplot as plt\nimport tensorflow_addons as tfa\nimport efficientnet.tfkeras as efn\nfrom tensorflow.keras import regularizers      # mitigate overfitting \nfrom kaggle_datasets  import KaggleDatasets    # import kaggle data files\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\nprint(\"Tensorflow version \" + tf.__version__)  # verify tensorflow versionis 2.x","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 2: Detect Hardware & Distribution Strategy","metadata":{}},{"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment  \n    # variable is set.  On Kaggle this is always the case.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # default distribution strategy in Tensorflow. \n    #  Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 3: Set Input Paths","metadata":{}},{"cell_type":"markdown","source":"## Data Directories","metadata":{}},{"cell_type":"code","source":"# Input Direcotry\n# Data files are available in the read-only \"kaggle/input/\" directory\n# image files are in TFRecords format, each of which contains a sequence\n# of records and can only be read sequentially.\n\nIMG_DIM = '512x512'\nfor dirpath, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        if IMG_DIM in dirpath: # \n            print(os.path.join(dirpath, filename))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Working Directory\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved \n# as output when you create a version using \"Save & Run All\" \nprint('list of entries contained in /kaggle/working/:',tf.io.gfile.listdir('/kaggle/working'))   \n\n# Temp Directory  \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of \n# the current session\n!gsutil cp /kaggle/input/tpu-getting-started/sample_submission.csv /kaggle/temp/test.csv\nprint('list of entries contained in /kaggle/temp/:',tf.io.gfile.listdir('/kaggle/temp'))    ","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Google Cloud Storage Data Paths\nGCS_DS_PATH_0 = KaggleDatasets().get_gcs_path(\"tpu-getting-started\")  # Google Cloud Storage\nprint(GCS_DS_PATH_0)\nprint('Entries in the bucket:')\n!gsutil ls $GCS_DS_PATH_0 # list items in the bucket ","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"GCS_DS_PATH_2 = KaggleDatasets().get_gcs_path(\"tf-flower-photo-tfrec\")  # Google Cloud Storage\nprint(GCS_DS_PATH_2)\nprint('Entries in the bucket:')\n!gsutil ls $GCS_DS_PATH_2  # list items in the bucket ","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Image Files","metadata":{}},{"cell_type":"code","source":"# current competition data: \"tpu-getting-started\" \nVAL_FILENAMES_0 = tf.io.gfile.glob(\n    GCS_DS_PATH_0 + '/tfrecords-jpeg-512x512/val/*.tfrec') \nTEST_FILENAMES_0 = tf.io.gfile.glob(\n    GCS_DS_PATH_0 + '/tfrecords-jpeg-512x512/test/*.tfrec') \nTRAIN_FILENAMES_0 = tf.io.gfile.glob(\n    GCS_DS_PATH_0 + '/tfrecords-jpeg-512x512/train/*.tfrec') \n\n# external data: \"tf-flower-photo-tfrec\" - duplicates removed\nTRAIN_FILENAMES_2A = tf.io.gfile.glob(\n    GCS_DS_PATH_2 + '/imagenet_no_test/tfrecords-jpeg-512x512/*.tfrec')  \nTRAIN_FILENAMES_2B = tf.io.gfile.glob(\n    GCS_DS_PATH_2 + '/inaturalist_no_test/tfrecords-jpeg-512x512/*.tfrec')\nTRAIN_FILENAMES_2C = tf.io.gfile.glob(\n    GCS_DS_PATH_2 + '/openimage_no_test/tfrecords-jpeg-512x512/*.tfrec')\nTRAIN_FILENAMES_2D = tf.io.gfile.glob(\n    GCS_DS_PATH_2 + '/tf_flowers_no_test/tfrecords-jpeg-512x512/*.tfrec')\nTRAIN_FILENAMES_2E = tf.io.gfile.glob(\n    GCS_DS_PATH_2 + '/oxford_102_no_test/tfrecords-jpeg-512x512/*.tfrec') ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" # Step 4: Set Parameters","metadata":{}},{"cell_type":"code","source":"# parameters \nIMAGE_SIZE         = [512, 512] \nHEIGHT             = IMAGE_SIZE[0]\nWIDTH              = IMAGE_SIZE[1]\nEPOCHS             = 35\nBATCH_SIZE         = 16 * strategy.num_replicas_in_sync\nDROP_RATE          = 0.4\nREG_FACTOR         = 0.035\nAUGMENTATION       = False\nWEIGHTS            = 'imagenet'  #  or 'noisy-student'\nMODEL_NAME         = 'EffNetB7_imagenet_NoAug_XD_DO_BN_v34.h5'\n\n# competition and external data for training\nTRAIN_FILENAMES    = TRAIN_FILENAMES_0 + TRAIN_FILENAMES_2E\nVAL_FILENAMES      = VAL_FILENAMES_0\nTEST_FILENAMES     = TEST_FILENAMES_0\n\n# number of images\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, \n    # i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)\nNUM_TRAIN_IMAGES = count_data_items(TRAIN_FILENAMES)\nNUM_VAL_IMAGES   = count_data_items(VAL_FILENAMES)\nNUM_TEST_IMAGES  = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH  = NUM_TRAIN_IMAGES // BATCH_SIZE\nAUTO             = tf.data.experimental.AUTOTUNE\n\n# print total number of images\nprint('Training Images:  ', NUM_TRAIN_IMAGES)\nprint('Validation Images:', NUM_VAL_IMAGES)\nprint('Test Images:      ', NUM_TEST_IMAGES)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# class names of flowers in the order of label idnum\nCLASSES = [\n    'pink primrose', 'hard-leaved pocket orchid', 'canterbury bells',   \n    'sweet pea', 'wild geranium', 'tiger lily', 'moon orchid',             \n    'bird of paradise', 'monkshood', 'globe thistle', 'snapdragon',                    \n    \"colt's foot\", 'king protea', 'spear thistle', 'yellow iris',   \n    'globe-flower', 'purple coneflower', 'peruvian lily', 'balloon flower', \n    'giant white arum lily', 'fire lily', 'pincushion flower',  \n    'fritillary', 'red ginger', 'grape hyacinth', 'corn poppy',  \n    'prince of wales feathers', 'stemless gentian', 'artichoke',\n    'sweet william', 'carnation', 'garden phlox', 'love in the mist',  \n    'cosmos', 'alpine sea holly', 'ruby-lipped cattleya', 'cape flower', \n    'great masterwort', 'siam tulip', 'lenten rose', 'barberton daisy',  \n    'daffodil', 'sword lily', 'poinsettia', 'bolero deep blue',   \n    'wallflower', 'marigold', 'buttercup', 'daisy', 'common dandelion',\n    'petunia', 'wild pansy', 'primula',  'sunflower', 'lilac hibiscus', \n    'bishop of llandaff', 'gaura', 'geranium', 'orange dahlia', \n    'pink-yellow dahlia', 'cautleya spicata', 'japanese anemone', \n    'black-eyed susan', 'silverbush',  'californian poppy', 'osteospermum',      \n    'spring crocus', 'iris', 'windflower', 'tree poppy', 'gazania', \n    'azalea', 'water lily', 'rose', 'thorn apple', 'morning glory',    \n    'passion flower', 'lotus', 'toad lily', 'anthurium', 'frangipani',  \n    'clematis', 'hibiscus', 'columbine', 'desert-rose', 'tree mallow', \n    'magnolia', 'cyclamen ', 'watercress', 'canna lily', 'hippeastrum ', \n    'bee balm', 'pink quill', 'foxglove', 'bougainvillea', 'camellia',  \n    'mallow', 'mexican petunia', 'bromelia', 'blanket flower', \n    'trumpet creeper', 'blackberry lily', 'common tulip',  'wild rose'               \n]    \nprint('Number of classes:', len(CLASSES))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 5: Utility Functions to handle data","metadata":{}},{"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    # convert image to floats in [0, 1] range\n    image = tf.cast(image, tf.float32) / 255.0  \n    # explicit size needed for TPU\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) \n    return image\n    \ndef read_labeled_tfrecord(example):\n    \"\"\"returns a dataset of (image, label) pairs\"\"\"\n    LABELED_TFREC_FORMAT = {\n        # shape [] means single element; tf.string means bytestring\n        \"image\": tf.io.FixedLenFeature([], tf.string), \n        \"class\": tf.io.FixedLenFeature([], tf.int64), # from 0 to 103\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label \n\ndef read_unlabeled_tfrecord(example):\n    \"\"\"returns a dataset of (image, idnum) pairs\"\"\"\n    UNLABELED_TFREC_FORMAT = {\n        # shape [] means single element; tf.string means bytestring \n        \"image\": tf.io.FixedLenFeature([], tf.string), \n        \"id\": tf.io.FixedLenFeature([], tf.string),    \n        # class is missing, to be predicted flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum \n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    \"\"\"\n    Read from TFRecords. For optimal performance, reading from multiple files \n    at once and disregarding data order. Order does not matter since we will\n    be shuffling the data anyway.\n    \"\"\"\n    ignore_order = tf.data.Options()\n    if not ordered:\n        # disable order, increase speed\n        ignore_order.experimental_deterministic = False \n    \n    # automatically interleaves reads from multiple file\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) \n    \n    # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.with_options(ignore_order) \n    \n    # returns a dataset of (image, label) pairs if labeled=True \n    #   or (image, id) pairs if labeled=False\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, \n                          num_parallel_calls=AUTO)   \n    return dataset\n\ndef get_validation_dataset(ordered=False):\n    dataset = load_dataset(VAL_FILENAMES,labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    # prefetch next batch while training (autotune prefetch buffer size)\n    dataset = dataset.prefetch(AUTO) \n    return dataset\n\ndef get_test_dataset(ordered=True):  # order matters to submit predictions to Kaggle\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    # prefetch next batch while training (autotune prefetch buffer size)\n    dataset = dataset.prefetch(AUTO) \n    return dataset\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Augmentation Modules","metadata":{}},{"cell_type":"markdown","source":"Data augmentation is a way to reduce overfitting by randomly altering the training images as they are fit to the model for training.  Thus the images vary different at each epoch as though they are a different dataset, thus increasing the data size.  Here is more info at [TensorFlow](https://www.tensorflow.org/tutorials/images/data_augmentation#overview).\n\n\n## 1. Use tf.image or tfa.image  \n\nAs shown in the data_augment(image, label) function below, tf.image is used to apply random augmentation on one image at a time:\n\n    image = tf.image.random_flip_left_right(image) \n    image = tf.image.random_contrast(image, lower=0.8, upper=1.2)\n    image = tf.image.random_brightness(image,max_delta=0.1)\n    image = tf.image.random_saturation(image, lower=0.7, upper=1.0)\n    image = tfa.image.mean_filter2d(image, filter_shape = 3)\n    etc\n\nThis function is then used to map to a batch of images at the time images are fetched prior to model training:\n\n    if augmentation:\n        # map the data_augment function to each image in dataset prefetched during training\n        dataset = dataset.map(data_augment, num_parallel_calls=AUTO)   \n        \n\n## 2. Use Keras preprocessing layers  \n\nI explored a few image preprocessing layers for augmentation:\n\n    data_aug_layers = tf.keras.Sequential([\n        tf.keras.layers.experimental.preprocessing.RandomRotation(\n            0.125, fill_mode='constant'),\n        tf.keras.layers.experimental.preprocessing.RandomZoom(\n            (-0.17, -0.01), fill_mode='constant'),\n        tf.keras.layers.experimental.preprocessing.RandomFlip(mode='horizontal'),\n        tf.keras.layers.experimental.preprocessing.RandomTranslation(\n            (-.15,.15), (-.15,.15), fill_mode='constant')\n    ])\n\n* **Option 1**: Apply the preprocessing layers to the dataset at the time images are fetched prior to model training:\n\n        if augmentation:  \n            # apply data augmentation preprocessing layers in batch of images\n            dataset = dataset.map(lambda image, y: (data_aug_layers(image, training=True), y))  \n\n* **Option 2**: Make the preprocessing layers part of the model  \n\n        with strategy.scope():    \n        pretrained_model = efn.EfficientNetB7(\n            weights='imagenet', \n            include_top=False ,\n            input_shape=[*IMAGE_SIZE, 3]\n        )\n        pretrained_model.trainable = True # transfer learning\n        model = tf.keras.Sequential([\n            pretrained_model, \n            data_aug_layers,   # preprocessing layers as part of the model                  \n            tf.keras.layers.GlobalAveragePooling2D(),\n            tf.keras.layers.Dense(len(CLASSES), kernel_regularizer=regularizers.L2(0.005), \n                activation='softmax')\n        ])\n\n\n## 3. ImageDataGenerator  \n \nI am adding this third option to randomly tranform my training dataset.  This is great because one ImageDataGenerator with one random_transform method can do a lot of difference random augmentation on an image in one pass.  More info at [TensorFlow](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator).\n\n    tf.keras.preprocessing.image.ImageDataGenerator(\n        featurewise_center=False, samplewise_center=False,\n        featurewise_std_normalization=False, samplewise_std_normalization=False,\n        zca_whitening=False, zca_epsilon=1e-06, rotation_range=0, width_shift_range=0.0,\n        height_shift_range=0.0, brightness_range=None, shear_range=0.0, zoom_range=0.0,\n        channel_shift_range=0.0, fill_mode='nearest', cval=0.0,\n        horizontal_flip=False, vertical_flip=False, rescale=None,\n        preprocessing_function=None, data_format=None, validation_split=0.0, dtype=None\n    )\n     \nMy understanding is to first create the image.ImageDataGenerator with whatever data argumentation arguments you choose. Here is an example from TensorFlow:\n \n    img_gen = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255, rotation_range=20)\n\nThen, use the generator with the [random_transform method](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator#random_transform) to apply a random transformation to a single image according to the pre-set arguments/ranges of the generator.  Below, input x is a 3D tensor, single image. The output is a randomly transformed version of x of the same shape.\n\n    random_transform(\n        x, seed=None\n    )","metadata":{}},{"cell_type":"markdown","source":"## Random Data Augmentation Functions","metadata":{}},{"cell_type":"code","source":"################### tf.image; tfa.image ################\n\n# define data augmentation function, one image at a time                  \ndef data_augment(image,  label):\n    \n    # using tf.image \n    image = tf.image.random_flip_left_right(image) \n    image = tf.image.random_contrast(image, lower=0.8, upper=1.2)\n    image = tf.image.random_brightness(image, max_delta=0.1) \n    image = tf.image.random_saturation(image, lower=0.85, upper=1.0)\n    # these commented out:\n    # Pad the image with a black, 90-pixel border\n    #image = tf.image.resize_with_crop_or_pad(\n    #            image, HEIGHT + 180, WIDTH + 180\n    #)\n    # Randomly crop to original size from the padded image\n    #image = tf.image.random_crop(image, size=[*IMAGE_SIZE,3])\n\n    # using tfa.image \n    #rdn = tf.random.normal([1], mean=0, stddev=1, dtype=tf.float32) \n    #if rdn > 2.0:  # blur 2.5% of the images (1 tail, 2 stddev above mean)\n    #    image = tfa.image.mean_filter2d(image, filter_shape = 3,\n    #                                   padding='constant')\n        \n    return image, label","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"################# Keras preprocessing layers #################\n\n# create image augmentation layers\n# 0.125 rotation = 360*0.125 = 45 deg\ndata_aug_layers = tf.keras.Sequential([\n    tf.keras.layers.experimental.preprocessing.RandomRotation(\n        0.125, fill_mode='constant'),\n    tf.keras.layers.experimental.preprocessing.RandomZoom(\n        height_factor=(-0.5, 0.25), width_factor=(-0.5, 0.25), \n        fill_mode='constant')\n])\n# these layers removed:\n#    tf.keras.layers.experimental.preprocessing.RandomTranslation(\n#        (-.15,.15),(-.15,.15), fill_mode='constant'),\n#    tf.keras.layers.experimental.preprocessing.RandomZoom(\n#        (-0.17, -0.01), fill_mode='constant'),  \n#    tf.keras.layers.experimental.preprocessing.RandomFlip(\n#        mode='horizontal'), ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"################### ImageDataGenerator  ###################\n\n# create an ImageDataGenerator \n# update this based on image augmenation exploration results\nimg_gen = tf.keras.preprocessing.image.ImageDataGenerator(\n    rotation_range=45, width_shift_range=0.25, height_shift_range=0.25,\n    brightness_range=None, zoom_range=[0.5, 1.25], shear_range=0.2, fill_mode='constant', \n    horizontal_flip=True, preprocessing_function=True\n)\n\n# define data augmentation function with random_transform method \n# for dataset.map( ... )\ndef img_gen_random_transform(image, label):\n    # apply random_transform method to single image\n    image = img_gen.random_transform(image)\n    return image, label\n\ndef img_gen_random_transform_image(image):\n    # apply random_transform method to single image\n    image = img_gen.random_transform(image)\n    return image","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Get Training Dataset Function\n `augmentation=True` or `augmentation=False`","metadata":{}},{"cell_type":"code","source":"# get training datatset with augmentation option\n\"\"\"\nBut img_gen.random_transform raises errors in user code when call in get_training_dataset:\n/opt/conda/lib/python3.7/site-packages/keras_preprocessing/image/image_data_generator.py\n\"\"\"\ndef get_training_dataset(augmentation=False):\n    dataset = load_dataset(TRAIN_FILENAMES, labeled=True, ordered=False)\n    if augmentation:\n        # map the data_augment function ones mages at a time\n        dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n        # map the img_gen_random_transform function across images of the dataset\n        #dataset = dataset.map(img_gen_random_transform, num_parallel_calls=AUTO)  # didn't work  \n               \n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(buffer_size=1920)\n    dataset = dataset.batch(BATCH_SIZE)\n    \n    if augmentation:\n        # apply data augmentation preprocessing layers in batch of images\n        dataset = dataset.map(lambda image, y: (data_aug_layers(image, training=True), y))\n        # map the img_gen_random_transform_image function in batch of images\n        #dataset = dataset.map(lambda image, y: (img_gen_random_transform_image(image), y)) #didn't work\n        \n    dataset = dataset.prefetch(AUTO)  # prefetch next batch while training\n    return dataset","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Image Visualization Functions (in batches)","metadata":{}},{"cell_type":"code","source":"np.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(databatch):\n    images, labels = databatch\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef numpylabels_to_classlabels(numpy_labels):\n    class_labels = []  # initialize list\n    if numpy_labels[0] == None:\n        class_labels = numpy_labels  # no label for test images\n    else:\n        for num in enumerate(numpy_labels):\n            class_labels.append(CLASSES[num[1]])    \n    return class_labels\n\ndef show_images(databatch, row=6, col=8):  # row, col of subplots\n    numpy_images, numpy_labels = batch_to_numpy_images_and_labels(databatch)\n    labels = numpylabels_to_classlabels(numpy_labels)   \n\n    FIGSIZE = (col*3, row*3)  # 3X3 inch per image\n    plt.figure(figsize=FIGSIZE)      \n    for j in range(row*col):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.title(labels[j])\n        plt.imshow(numpy_images[j])\n    plt.show()\n","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 6: Image Augmentation Exploration","metadata":{}},{"cell_type":"code","source":"# get training dataset with augmentation=False\nno_aug_train_set = get_training_dataset(augmentation=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Re-run these codes to get the next batch of no aug training images\nno_aug_train_batch = (next(iter(no_aug_train_set.unbatch().batch(16)))) \nimages, _ = batch_to_numpy_images_and_labels(no_aug_train_batch)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Explore image data augmentation: tf.image & tfa.image","metadata":{}},{"cell_type":"code","source":"# function to show image with random data augmentation\ndef show_data_aug(image):\n    ROW=len(images)\n    COL=7  # 1 no-aug plus 6 aug images\n    plt.figure(figsize=(COL*2,ROW*2))\n    i=0\n    for image in images:\n        plt.subplot(ROW,COL,i*COL+1)\n        plt.title('rdm flip L/R')\n        plt.axis('off')  \n        # augmented with random flip\n        plt.imshow(tf.image.random_flip_left_right(image))       \n\n        plt.subplot(ROW,COL,i*COL+2)\n        plt.title('resize & rdm crop')\n        plt.axis('off')    \n        # Pad the image with a black, 90-pixel border\n        image1 = tf.image.resize_with_crop_or_pad(\n            image, HEIGHT + 180, WIDTH + 180\n        )\n        # Randomly crop to original size from the padded image\n        image1 = tf.image.random_crop(image1, size=[*IMAGE_SIZE,3])\n        plt.imshow(image1)\n\n        plt.subplot(ROW,COL,i*COL+3)\n        plt.title('rdm contrast')\n        plt.axis('off')\n        # augmented with contrast\n        plt.imshow(tf.image.random_contrast(image, lower=0.8, upper=1.2))  \n\n        plt.subplot(ROW,COL,i*COL+4)\n        plt.title('rdm brightness')\n        plt.axis('off')\n        # augmented with brightness\n        plt.imshow(tf.image.random_brightness(image, max_delta=0.1))       \n\n        plt.subplot(ROW,COL,i*COL+5)\n        plt.title('no aug')\n        plt.axis('off')\n        plt.imshow(image)\n\n        plt.subplot(ROW,COL,i*COL+6)\n        plt.title('rdm saturation')\n        plt.axis('off')\n        # augmented with saturation\n        plt.imshow(tf.image.random_saturation(image, lower=0.7, upper=1.3))  \n\n        plt.subplot(ROW,COL,i*COL+7)\n        plt.title('rdm blur')\n        plt.axis('off')        \n        # ouput a rdm value from a normal distribtion \n        rdn = tf.random.normal([1], mean=0, stddev=1, dtype=tf.float32)              \n        if rdn > 2.0:  # 2 stddev above mean  \n            # blur 2.5% of the images\n            # using tfa.image mean filter\n            plt.imshow(\n                tfa.image.mean_filter2d(image, filter_shape = 3,\n                padding='constant')\n            )  \n        else:\n            plt.imshow(image)\n            \n        i+=1\n        \n    plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Re-run this after adjusting image augmentation settings \n#   of the show_data_aug() function\n# compare no aug training images with random data augmentation\nprint('Training Dataset')\nprint('Image Augmentation with tf.image and tfa.image')\nshow_data_aug(images)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Explore image data augmentation: for keras.layers","metadata":{}},{"cell_type":"code","source":"# Run these to visualize effects before implementing in\n#     tf.keras.layers.experimental.preprocessing.Random___()\nprint('Training Dataset')\nprint('Image Augmentation with tf.keras.preprocesing.image.random ...')\nROW=len(images)\nCOL=4  # 1 no-aug plus 3 aug images\nplt.figure(figsize=(COL*3,ROW*3))\ni=0\nfor image in images:\n    plt.subplot(ROW,COL,i*4+1)\n    plt.title('no aug')\n    plt.axis('off')\n    plt.imshow(image)\n    \n    plt.subplot(ROW,COL,i*4+2)\n    plt.title('rdm shift')\n    plt.axis('off')\n    # random shift on one numpy image tensor \n    # compared to tf.keras.layers.experimental.preprocessing.RandomTranslation(...)\n    image2 = tf.keras.preprocessing.image.random_shift(\n        image, wrg=0.15, hrg=0.15, row_axis=1, col_axis=2, channel_axis=2,\n        fill_mode='constant'\n    )    \n    plt.imshow(image2) \n\n    plt.subplot(ROW,COL,i*4+3)\n    plt.title('rdm 45-deg rotation')\n    plt.axis('off')\n    # random rotation on one numpy image tensor\n    # compared to tf.keras.layers.experimental.preprocessing.RandomRotation(...)\n    image3 = tf.keras.preprocessing.image.random_rotation(\n        image, rg=45, row_axis=1, col_axis=2, channel_axis=2, fill_mode='constant'\n    )\n    plt.imshow(image3)\n\n    plt.subplot(ROW,COL,i*4+4)\n    plt.title('rdm zoom')\n    plt.axis('off')\n    # random zoom on one numpy image tensor\n    # comapred to tf.keras.layers.experimental.preprocessing.RandomZoom(...)\n    image4 = tf.keras.preprocessing.image.random_zoom(\n        #image, (.75, 1.0), row_axis=1, col_axis=2, channel_axis=2, fill_mode='constant'\n        image, (.5, 1.25), row_axis=0, col_axis=1, channel_axis=2, fill_mode='constant'\n    )\n    plt.imshow(image4)\n    i+=1\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Explore image data augmentation: ImageDataGenerator","metadata":{}},{"cell_type":"markdown","source":"### random_transform method","metadata":{}},{"cell_type":"code","source":"# create an ImageDataGenerator for random transformation\nexplore_img_gen = tf.keras.preprocessing.image.ImageDataGenerator(\n    rotation_range=45, width_shift_range=0.25, height_shift_range=0.25,\n    brightness_range=None, zoom_range=[0.5, 1.25], shear_range=0.2, fill_mode='constant', \n    horizontal_flip=True, preprocessing_function=True\n)\n\nprint('Training Dataset')\nprint('Image Augmentation with random_transform method from ImageDataGenerator')\ni = 0\nROW=8  # rows of subplots\nCOL=4  # cols of subplots\nplt.figure(figsize=(COL*3.4,ROW*3))\nfor im in images:\n    plt.subplot(ROW,COL,i*2+1)\n    plt.title('no augmentation')\n    plt.axis('off')\n    plt.imshow(im)\n    plt.subplot(ROW,COL,i*2+2)\n    plt.title('rdm transorm from img_gen')\n    plt.axis('off')\n    plt.imshow(explore_img_gen.random_transform(im))\n    i+=1\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Using functions for dataset.map()","metadata":{}},{"cell_type":"code","source":"print('Training Dataset')\nprint('Image Augmentation with ImageDataGenerator')\ni = 0\nROW=8  # rows of subplots\nCOL=4  # cols of subplots\nplt.figure(figsize=(COL*3.4,ROW*3))\n\nfor im in images:\n    plt.subplot(ROW,COL,i*2+1)\n    plt.title('no augmentation')\n    plt.axis('off')\n    plt.imshow(im)\n    # use one of the functions, both work here \n    #   but not with get_training_dataset()\n    #im = img_gen_random_transform_image(im) # augmented\n    im, _ = img_gen_random_transform(im, _) # augmented\n    plt.subplot(ROW,COL,i*2+2)\n    plt.title('rdm transorm from img_gen')\n    plt.axis('off')\n    plt.imshow(im)\n    i+=1\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Decide on final settings for image data augmentation\nAfter exploring data augmentation with tf.image, tfa.image, tf.kersa.preprocessing.image and the random_transform method of ImageDataGenerator, remember to go back to finalize the data augmentation function.  Note: for demonstration purpose, some augmentations were actually implemented with keras.layers, where the final settings for augmentation were adjusted, too.  \n\nTo train with data augmentation, set `augmentation=True` before getting traning images from `training_dataset = get_training_dataset(augmentation=True)`   \n\nTo train only with autoaug feature that comes with EfficientNetB7, get data with `augmentation=False`.","metadata":{}},{"cell_type":"markdown","source":"## Visualization: Sample Images of All Datasets\n\n### observations  \n\nThe following allows me to get an idea about the three datasets.  I was surprised to see a picture that looks like a bridge (no flowers), a little girl (holding a very tiny flower behind her back) and a tatoo of flowers on someone's leg in the training dataset.  There is a blank (white) picture (label=70) in the validation dataset.  There is a picture of a fountain (didn't see flowers) in the test dataset.  There are pictures with people, hands, pets and insects in the pictures in all datasets.  Pictures taken from top view and side views, a single flower or flowers in bundles, close-up, from far in a meadow or along a riverside, zoomed-in and cropped, etc, are common in all datasets.","metadata":{}},{"cell_type":"code","source":"# Get datasets for visualization\ntraining_dataset   = get_training_dataset(augmentation=True) # with data augmentation\nvalidation_dataset = get_validation_dataset(ordered=False)\ntest_dataset       = get_test_dataset(ordered=False)  # not for prediction","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# you may run these lines multiple times to view different samples from the image sets\nR = 7     # rows of subplots/images\nC = 6     # cols of subplots/images\nB = R*C   # number of images in a batch\nprint('Training Images - optional random data augmentation')\nshow_images(next(iter(training_dataset.unbatch().batch(B))), row=R, col=C)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# you may run these lines multiple times to view different samples from the image sets\nprint('Validation Images')\nshow_images(next(iter(validation_dataset.unbatch().batch(B))), row=R, col=C)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# you may run these lines multiple times to view different samples from the image sets\nprint('Test Images - not ordered, shuffled') # randomly shuffles the test dataset for visualization\nshow_images(next(iter(test_dataset.unbatch().shuffle(buffer_size=BATCH_SIZE).batch(B))), \n            row=R, col=C)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 7: Callbacks for Model Training ","metadata":{}},{"cell_type":"markdown","source":"## Learning Rate Scheduler callback  \nThe [Learning Rate Scheduler callback](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/LearningRateScheduler) gets the updated learning rate value from schedule function at the beginning of each epoch.  The schedule function takes an epoch index (integer, indexed from 0) and current learning rate (float) as inputs and returns a new learning rate as output (float).  \n\n    tf.keras.callbacks.LearningRateScheduler(\n        schedule, verbose=0\n    )","metadata":{}},{"cell_type":"code","source":"# define a fine-tuned schedule for the Learning Rate Scheduler \ndef exponential_lr(epoch,\n                  start_lr=0.00001,min_lr=0.00001,max_lr=0.00005,\n                  rampup_epochs = 5, sustain_epochs = 0,\n                  exp_decay = 0.8):\n    def lr(epoch, start_lr, min_lr,max_lr,rampup_epochs,sustain_epochs,\n          exp_decay):\n        # linear increase from start to rampup_epochs\n        if epoch < rampup_epochs:\n            lr= ((max_lr-start_lr)/\n                rampup_epochs * epoch + start_lr)\n        elif epoch < rampup_epochs + sustain_epochs:\n            lr = max_lr \n        else:\n            lr = ((max_lr - min_lr)* exp_decay ** (epoch-rampup_epochs-sustain_epochs)\n                  + min_lr)\n            \n        return lr\n    return lr(epoch,start_lr,min_lr,max_lr,rampup_epochs,sustain_epochs,exp_decay)\n\n# set learning rate scheduler for callback\nlr_callback = tf.keras.callbacks.LearningRateScheduler(schedule=exponential_lr,verbose=True)\n\n# learning rate chart\nepoch_rng = [i for i in range(EPOCHS)] \ny = [exponential_lr(x) for x in epoch_rng]\nplt.plot(epoch_rng,y)\nplt.xlim(-1, EPOCHS)\n\nprint(\"Learning rate schedule: start = {:.3g}; peak = {:.3g}; end = {:.3g}\".format(y[0], max(y), y[-1]))","metadata":{"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## EarlyStopping callback\nReference: https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping    \nThe metric \"val_loss' will be monitored and allows training to be stopped early when the metric stops improving.  I set patience = 2, so training will be stopped after 2 epochs with no improvement.\n\n    tf.keras.callbacks.EarlyStopping(\n        monitor='val_loss', min_delta=0, patience=0, verbose=0,\n        mode='auto', baseline=None, restore_best_weights=False\n    )","metadata":{}},{"cell_type":"code","source":"# set earlystopping for callback\nes_callback = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Checkpoint callback","metadata":{}},{"cell_type":"markdown","source":"I referenced tutorials on TensorFlow on [Save and load models](https://www.tensorflow.org/tutorials/keras/save_and_load) and on [Save checkpoints during training](\nhttps://www.tensorflow.org/tutorials/keras/save_and_load#save_checkpoints_during_training)  \n\n    tf.keras.callbacks.ModelCheckpoint(\n        filepath, monitor='val_loss', verbose=0, save_best_only=False,\n        save_weights_only=False, mode='auto', save_freq='epoch',\n        options=None, **kwargs\n    )\n\nWith `tf.keraas.callbacks.ModelCheckpoint(..., save_best_only=True, save_weights_only=False, ...)`, I saved my best model after each epoch if it is considered the best at the time during training.  With the saved models, I could do the following:  \n1. After training, I may load my best model prior to computing predictions. (Trained model at the last epoch may not be the best.)  \n2. I may download my best model as pretrained base model to use in other platform.  When creating a notebook version using \"Save & Run All\", the model will be saved locally and preserved as output in my Kaggle working directory: `/kaggle/working/`.  I may download and use the model at another computing platform, like Google Colab. I may use the saved model as a base model to build on it by adding more layers, or make an adversarial model with it, etc., and see whether that makes a difference in the classification accuracy.  \n3. I may use models that are output to my kaggle/working directory in other notebooks, by simply clicking \"Add Data\" and selecting \"Notebook Output Files\".  I may continue to train the model in anther session with more external data or more epochs, etc.  Remember to save your model with a different filename each time so you don't write over what you have.  \n\n### Writing checkpoints locally from a TPU model\nI spent a lot of time looking up for a solution for the UnimplementedError, so I want to capture what I found here. The following [example from Kaggle](https://www.kaggle.com/docs/tpu#tpu5a) shows an important argument for tf.keras.callbacks.ModelCheckpoint: `options=save_locally`.  And save_locally is configured to save model on `experimental_io_device='/job:localhost'`\n```\nsave_locally   = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')\ncheckpoints_cb = tf.keras.callbacks.ModelCheckpoint('./checkpoints', options=save_locally)\nmodel.fit( . . . , callbacks=[checkpoints_cb])\n```\nTensorflow has [API documentation](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint) that explains how to set related arguments to save weights only as opposed to save model locally. It look simple Tensorflow's argument table for the argument \"options\": \n\n```\nOptional tf.train.CheckpointOptions object if save_weights_only is true, or, \noptional tf.saved_model.SaveOptions object if save_weights_only is false.\n```\nThat means the following:  \n\n#### **A)** If you **save weights only**, i.e. set `save_weights_only=True`:\n\n```\ntf.keras.callbacks.ModelCheckpoint(..., \n                                     save_weights_only=True, \n                                     options=save_locally,\n                                     ...)\n                                     \n```\nthen, `options= ...` must be a **tf.train.CheckpointOptions( ... ) object**, i.e. have `save_locally` pre-defined as below:  \n`save_locally = tf.train.CheckpointOptions(experimental_io_device='/job:localhost')`  \n\nNote: if not saving weights locally, set `options=None`   \n\n#### **B)** If you **save the model**, i.e. set `save_weights_only=False`:\n```\ntf.keras.callbacks.ModelCheckpoint(...,\n                                   save_weights_only=False`,\n                                   options=save_locally,\n                                   ...)\n```                                   \nthen, `options= ...` must be a **tf.saved_model.SaveOptions( ... ) object**, i.e. have `save_locally` pre-defined as below:  \n`save_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')`\n\n\nIf options to save locally is not set up properly, the following error will occur when checkpoint files are to be saved during training:  \n```\nUnimplementedError - File system scheme '[local]' not implemented \n(file: '/kaggle/working/checkpoint_temp/variables/variables_temp/part-00000-of-00001') \nEncountered when executing an operation using EagerExecutor. \nThis error cancels all future operations and poisons their output tensors.\n```","metadata":{"_kg_hide-input":true}},{"cell_type":"markdown","source":"Expand the above for more details.  The following codes are to set up to save the best model locally.","metadata":{}},{"cell_type":"code","source":"# set file path to save model locally to your /kaggle/working/\nBEST_MODEL_PATH = \"/kaggle/working/\" + MODEL_NAME   \nFILE_DIR = os.path.dirname(BEST_MODEL_PATH)                  \n\n# Create a checkpoint callback that saves the best trained model locally \n#   save_best_only=True, \nsave_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')\ncp_callback  = tf.keras.callbacks.ModelCheckpoint(filepath=BEST_MODEL_PATH,      \n                   options=save_locally, monitor='val_loss', verbose=1,\n                   save_best_only=True, save_weights_only=False, mode='min')\n\n# show current entries saved in Kaggle output directory\nprint('list of entries contained in', FILE_DIR, tf.io.gfile.listdir(FILE_DIR))\n# the following show current entries saved in Kaggle output directory too\n#!ls {FILE_DIR}  # same as \n!ls \"/kaggle/working/\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 8: Build the model","metadata":{}},{"cell_type":"code","source":"# With pretrained model \nwith strategy.scope():    \n    pretrained_model = efn.EfficientNetB7(\n        weights=WEIGHTS, \n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    pretrained_model.trainable = True # transfer learning\n    model = tf.keras.Sequential([\n        pretrained_model, \n        tf.keras.layers.Dropout(DROP_RATE),\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.BatchNormalization(),\n        tf.keras.layers.Dense(len(CLASSES), kernel_regularizer=regularizers.L2(REG_FACTOR), \n            activation='softmax')\n    ])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# display model summary, including all layers, the output shape and the number of parameters for each layer,  \n#  the number of trainable parameters and the number of non-trainable parameters. \npretrained_model.summary()","metadata":{"_kg_hide-output":true,"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('---------------- Pretrained Model: EfficientNetB7 ----------------')\nprint('Total number of layers in pretained base model =', len(pretrained_model.layers))\nprint('')\n\nprint('---------------------------- My Model ----------------------------')\nmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Compile the model ","metadata":{}},{"cell_type":"markdown","source":"`optimizer='adam'` implements the Adam algorithm with some default values set for some arguments, e.g. learning_rate. Adam optimization is a stochastic gradient descent method.\n\n`loss = 'sparse_categorical_crossentropy'` specifies that crossentropy metric is computed between the labels and predictions. This metric is used when there are two or more label classes. Labels are expected to be provided as integers. In this floser classification challenge, there are 104 different classes of flowers.\n\n`metrics=['sparse_categorical_accuracy']`  \nInteger labels are used in the training, validation and test datasets. Thus metric is set to use sparse categorical accuracy, which calculates how often predictions matches the integer labels.","metadata":{}},{"cell_type":"code","source":"model.compile(\n    optimizer='adam',\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy']\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 9: Train the model - with callbacks list","metadata":{}},{"cell_type":"markdown","source":"Now the model is created, configured and compiled with losses and metrics with `model.compile(...)`, it is time to train the model with `model.fit()`, which outputs a History object.  \n\n    historical = model.fit( ...,                  \n                           callbacks=[..., ...]\n                           )                   \n                           \nAs the codes indicated, while the model is being trained, events are recorded into the History object named \"historical\".   This object's attribute, `historical.history` is then used to create plots to show accuracy and loss metrics in the next section.  \n\n### Callbacks\nThe callbacks argument `callbacks=[lr_callback, es_callback, cp_callback]` allows the listed callbacks to appy during training. Here is the list of [keras.callbacks.Callback](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks) instances.  \n\nThe cp_callback works well and saves the best model in the .h5 format.  This allows the best model to be loaded prior to doing predictions on test dataset.  (The trained model at the last epoch may not be necessarily the best model.)  ","metadata":{}},{"cell_type":"code","source":"# get datasets for model training and validation\n# AUGMENTATION = True or False \ntraining_dataset   = get_training_dataset(augmentation=AUGMENTATION) \nvalidation_dataset = get_validation_dataset(ordered=False)\nprint('training dataset:       ', training_dataset)\nprint('validation dataset:     ', validation_dataset)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fit/train the model \n# save the History object to the variable \"historical\"\n# save checkpoints during training\nhistorical = model.fit(\n    training_dataset, \n    steps_per_epoch=STEPS_PER_EPOCH, \n    epochs=EPOCHS, \n    validation_data=validation_dataset,\n    callbacks=[lr_callback, es_callback, cp_callback]\n) ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# load best model saved by cp_callback during training\nmodel = tf.keras.models.load_model(BEST_MODEL_PATH)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 10: Evaluate the Model","metadata":{}},{"cell_type":"markdown","source":"## Confusion Matrix","metadata":{}},{"cell_type":"code","source":"# Construct confusion matrix\ndef display_confusion_matrix(cmat, score, precision, recall):\n    plt.figure(figsize=(15, 15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment': 'right', 'verticalalignment': 'top', 'color': '#804040'})\n    plt.show()\n\n\ncmdataset = get_validation_dataset(ordered=True)\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\n\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VAL_IMAGES))).numpy()\ncm_probabilities = model.predict(images_ds)\ncm_predictions = np.argmax(cm_probabilities, axis=-1)\n\nlabels = range(len(CLASSES))\ncmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=labels)\ncmat = (cmat.T / cmat.sum(axis=1)).T  # normalize\n\nscore = f1_score(cm_correct_labels, cm_predictions, labels=labels, average='macro')\nprecision = precision_score(cm_correct_labels, cm_predictions, labels=labels, average='macro')\nrecall = recall_score(cm_correct_labels, cm_predictions, labels=labels, average='macro')\ndisplay_confusion_matrix(cmat, score, precision, recall)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plots: accuracy and loss metrics\n","metadata":{}},{"cell_type":"code","source":"# Create plots of loss and accuracy on the training and validation datasets.\n\nacc = historical.history['sparse_categorical_accuracy']\nval_acc = historical.history['val_sparse_categorical_accuracy']\n\nloss = historical.history['loss']\nval_loss = historical.history['val_loss']\n\nepochs_range = range(1, len(historical.history['loss'])+1)\n\nplt.figure(figsize=(14, 14))\nplt.subplot(2, 1, 1)\nplt.plot(epochs_range, acc, label='Training Accuracy')\nplt.plot(epochs_range, val_acc, label='Validation Accuracy')\nplt.legend(loc='lower right')\nplt.title('Training and Validation Accuracy')\nplt.xlabel('Epoch')\n\nplt.subplot(2, 1, 2)\nplt.plot(epochs_range, loss, label='Training Loss')\nplt.plot(epochs_range, val_loss, label='Validation Loss')\nplt.legend(loc='upper right')\nplt.title('Training and Validation Loss')\nplt.xlabel('Epoch')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 11: Compute predictions \nIf you are happy with training results above, it is time to use the best trained model to make predictions with `model.predict(...)`.  \n\n`np.savetxt(...)` will create a file that can be submitted to the competition.\n","metadata":{}},{"cell_type":"code","source":"# get test dataet for prediction\n# ordered for prediction for submission to Kaggle\ntest_dataset = get_test_dataset(ordered=True) \nprint('test dataset:', test_dataset)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# show files in local working directories \nprint('list of entries contained in', FILE_DIR, tf.io.gfile.listdir(FILE_DIR)) \n\n# predict probabilities and match to the most probable integer label for each image\nprint('Computing predictions...')\ntest_images_ds = test_dataset.map(lambda image, idnum: image)\nprobabilities = model.predict(test_images_ds)  \npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)\n\n# create file to submit to the competition\nprint('Generating submission.csv file...')\ntest_ids_ds = test_dataset.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\nnp.savetxt('submission.csv', np.rec.fromarrays([test_ids, predictions]), \n           fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}