{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Updates\n\n### **Comments and upvotes are very much appreciated!**\n\n**02-18-2021:**   \n* added data augmentation using tfa.image  \n* added checkpoint to callback list: saving the checkpoint files to kaggle's local directory (/kaggle/working) is tricky.  But I did it!\n* explored additional techniques of for random image augmentation using ImageDataGenerator :  \n  But I haven't been successful to make it work in the data pipeline yet.  I have commented out a few lines of codes that caused errors, noting \"didn't work\".   If you've got some tips to share, please leave me some comments.  :-)  \n  \n**02-24-2021:**  \n* adjusted some data augmentation, dropout and regularization parameters.\n* added checkpoint callback to the callback list.\n* load the saved best model saved during training prior to computing predictions.\n\n**02-27-20201:**\n* updated and formatted quite a lot the markdown cells\n* adjusted some arguments settings of data augmentation  \n* explored regularization with a dropout layer and L1L2 kernel regularizer  \n* explored wrapping the model with [adversarial regularization](https://www.tensorflow.org/neural_structured_learning/tutorials/adversarial_keras_cnn_mnist):  \n`adv_config = nsl.configs.make_adv_reg_config( ... )`\n`adv_model - nsl.keras.AdversarialRegularization(model, adv_config=adv_config)`\n* complied the model: `adv_model.compile( ...)`\n* worked on fitting the adv_model, but I kept getting errors.  So after the the efforts of wrapping my model with adversarial regularization and compiling it, I changed the cell into a markdow cell and reverted back to fitting my orginally model. If anyone can land me a hand or point me to some info, please leave me some comments.  Thanks.\n\n**03-22-2021:**  \n* added external data [Oxford Flowers TFRecords (png).](https://www.kaggle.com/cdeotte/oxford-flowers-tfrecords) Thank to Chris Deotte for converting and sharing!  \n* built and trained model with DenseNet201.  \n* adjusted regularization:\n`tf.keras.layers.Dropout(0.3),  `\n`tf.keras.layers.Dense(104, kernel_regularizer=regularizers.L2(0.0003), activation='softmax\")`\n* forgot to revise the total number of training images, only which resulted in swapping out some training images of this challenge with external data\n* Best score = 0.94639\n\n**03-22-2021:**  \n* set up to add other external data [tf_flower_photo_tfrec (jpeg)](https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec) created from @hengck23 data. Most duplicates removed and images converted to tfrecords for convenience. Thanks to @hengck23 for creating and to Kirill Blinov for converting and sharing!   \n\n**03-25-2021:**  \n* corrected count for total number of training images  \n\n**03-27-2021:**  \n* limited external data to only [tf_flower_photo_tfrec (jpeg)](https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec) (duplicates removed): oxford_102 \n\n**04-03-2021:**  \n* removed Dropout layer  \n* adjusted parameter for kernel L2 regularizar in the dense layer  \n\n\n**04-04-2021:**  \n* added tf.keras.layers.Dropout(0.1) **before** tf.keras.layers.GlobalAveragePooling2D()  "},{"metadata":{},"cell_type":"markdown","source":"# Introduction"},{"metadata":{},"cell_type":"markdown","source":"The challenge of **Petals to the Metal - Flower Classification on TPU** is to build a machine learning model that classifies 104 type of flowers in a test dataset on Kaggle.com.  Datasets and details af the challenge are on https://www.kaggle.com/c/tpu-getting-started/  After exploring different pretrained models and datasets of different image sizes, I decided to focus my efforts in building my model on DenseNet201 for the 512x512 datasets.  I tried different data augmentation techniques and other techniques to overcome overfitting.  The following lists what I have explored:  \n\n* **Input image size**:  \n  192x192  \n  224x224  \n  512x512  \n* **Epochs**:  \n  50  \n  100  \n  150  \n  15 with EarlyStopping \n  30 with EarlyStopping\n* **Pretrained model**:   \n  VGG16   \n  InceptionV3   \n  DenseNet201   \n  EfficientNetB7 (with autoaug)  \n* **Number of trainable params**:  \n  14,768,040 (VGG16 + dropout layer)  \n  21,981,448 (InceptionV3 + dropout layer)   \n  18,292,712 (DenseNet201 + dropout layer)  \n  64,053,304 (EfficientNet7)    \n* **Data Source**:  \n  This Challenge: [Petals to the Metal - Flower Classification on TPU](https://www.kaggle.com/c/tpu-getting-started/data)  \n  External: [tf_flower_photo_tfrec](https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec)  \n* **Data augmentation**:   \n\n  tf.image and tfa.image in data_augment function:\n  ```\n  image = tf.image.random_flip_left_right(image)\n  image = tf.image.random_contrast(image, 0.9, 0.99)\n  image = tf.image.random_brightness(image, 0.05)\n  image = tf.image.random_saturation(image, 0.9, 0.99)\n  image = tf.image.resize_with_crop_or_pad(image, HEIGHT + 180, WIDTH + 180)\n  image = tf.image.random_crop(image, size=[*IMAGE_SIZE,3])\n  image = tfa.image.mean_filter2d(image, filter_shape = 3) \n  ```\n    tf.keras image augmentation layers:\n  ```    \n  data_aug_layers = tf.keras.Sequential([\n      tf.keras.layers.experimental.preprocessing.RandomRotation(\n          0.15, fill_mode='constant'),\n      tf.keras.layers.experimental.preprocessing.RandomZoom(\n          (-0.17, -0.01), fill_mode='constant'),\n      tf.keras.layers.experimental.preprocessing.RandomFlip(mode='horizontal'),\n      tf.keras.layers.experimental.preprocessing.RandomTranslation(\n          (-.15,.15), (-.15,.15), fill_mode='constant')\n  ])\n  ```\n  ImageDataGenerator: random_transform method\n  ```       \n  img_gen = tf.keras.preprocessing.image.ImageDataGenerator(\n      rotation_range=54, width_shift_range=0.15, height_shift_range=0.15,\n      brightness_range=None, zoom_range=[1.0, 1.25], fill_mode='constant', \n      horizontal_flip=True, preprocessing_function=None) \n\n  image = img_gen.random_transform(image)\n  ```\n* **Dropout layer**:  \n  ```\n  tf.keras.layers.Dropout(0.3) \n  ```     \n* **L2 kernel regularizer**: \n```\n  tf.keras.layers.Dense(\n      104, \n      kernel_regularizer=regularizers.l2(0.001), \n      activation='softmax'\n  )\n  ```                          \n* **Callback list**: \n\n      callbacks=[lr_callback,es_callback, cp_callback] \n      \n  Fine-tuned learning rate scheduler: exponential_lr  (credit to another Kaggler!)\n  ```\n  lr_callback = tf.keras.callbacks.LearningRateScheduler(\n                    exponential_lr,verbose=True\n  )  \n  ```    \n  EarlyStopping: \n  ```\n  es_callback = tf.keras.callbacks.EarlyStopping(\n                    monitor='val_loss', patience=2\n  )\n  ```    \n  Checkpoint: \n  ```\n  cp_callback = tf.keras.callbacks.ModelCheckpoint(\n                    filepath=checkpoint_path,\n                    save_weights_only=True, verbose=1\n  )\n  ```"},{"metadata":{},"cell_type":"markdown","source":"# Install new libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"# upgrade pip if needed\n!pip install --upgrade pip  # get the latest version\n# install tensorflow-addons\n!pip install -q -U tensorflow-addons\n# install the Neural Structured Learning package\n!pip install --quiet neural-structured-learning","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Import Libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed on Kaggle\n# It is defined by the [kaggle/python Docker image](https://github.com/kaggle/docker-python)\nimport re, os\nimport numpy as np \nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nimport tensorflow_addons as tfa\nimport neural_structured_learning as nsl\nfrom tensorflow.keras import regularizers      # mitigate overfitting \nfrom kaggle_datasets  import KaggleDatasets    # import kaggle data files\n# Stop training when a monitored metric has stopped improving\n# from tensorflow.keras.callbacks import EarlyStopping   \nprint(\"Tensorflow version \" + tf.__version__)  # verify tensorflow versionis 2.x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Detect Hardware"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is set. \n    # On Kaggle this is always the case.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Directories"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Input data files are available in the read-only \"kaggle/input/\" directory\n#   image files are in TFRecords format, each of which contains a sequeence\n#   of records and can only be read sequentially.\n\nTFRec_selected = '512x512'\nfor dirpath, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        if TFRec_selected in dirpath: # \n            print(os.path.join(dirpath, filename))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved \n#   as output when you create a version using \"Save & Run All\" \n!gsutil cp /kaggle/input/tpu-getting-started/sample_submission.csv /kaggle/working/test.csv\nfor dirpath, _, filenames in os.walk('/kaggle/working'):\n    for filename in filenames:\n        print(os.path.join(dirpath, filename))     \nprint('list of entries contained in /kaggle/working/:',tf.io.gfile.listdir('/kaggle/working'))   \n        \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of \n# the current session\n!gsutil cp /kaggle/input/tpu-getting-started/sample_submission.csv /kaggle/temp/sample_submission.csv\nfor dirpath, _, filenames in os.walk('/kaggle/temp'):\n    for filename in filenames:\n        print(os.path.join(dirpath, filename))   \n\nprint('list of entries contained in /kaggle/temp/:',tf.io.gfile.listdir('/kaggle/temp'))    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Set up data path"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"GCS_DS_PATH_0 = KaggleDatasets().get_gcs_path(\"tpu-getting-started\")  # Google Cloud Storage\nprint(GCS_DS_PATH_0)\nprint('Entries in the bucket 0:')\n!gsutil ls $GCS_DS_PATH_0  # list items in the bucket \n\n# external data from \"oxford-flowers-tfrecords\"\n#GCS_DS_PATH_1 = KaggleDatasets().get_gcs_path(\"oxford-flowers-tfrecords\")  # Google Cloud Storage\n#print(GCS_DS_PATH_1)\n#print('Entries in the bucket 1:')\n#!gsutil ls $GCS_DS_PATH_1  # list items in the bucket \n\n# external data from \"tf-flower-photo-tfrec\"\nGCS_DS_PATH_2 = KaggleDatasets().get_gcs_path(\"tf-flower-photo-tfrec\")  # Google Cloud Storage\nprint(GCS_DS_PATH_2)\nprint('Entries in the bucket 2:')\n!gsutil ls $GCS_DS_PATH_2  # list items in the bucket \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Set up parameters"},{"metadata":{"trusted":true},"cell_type":"code","source":"# parameters set for tfrecords-jpeg-512x512 TFRecord files\nIMAGE_SIZE         = [512, 512] \nHEIGHT             = IMAGE_SIZE[0]\nWIDTH              = IMAGE_SIZE[1]\nEPOCHS             = 15\nBATCH_SIZE         = 16 * strategy.num_replicas_in_sync\n\n# gather filepaths\n# tpu-getting-started data\nTRAIN_FILENAMES_0  = tf.io.gfile.glob(GCS_DS_PATH_0 + '/tfrecords-jpeg-512x512/train/*.tfrec') \nVAL_FILENAMES      = tf.io.gfile.glob(GCS_DS_PATH_0 + '/tfrecords-jpeg-512x512/val/*.tfrec') \nTEST_FILENAMES     = tf.io.gfile.glob(GCS_DS_PATH_0 + '/tfrecords-jpeg-512x512/test/*.tfrec') \n# oxford-flowers-tfrecords data\n#TRAIN_FILENAMES_1  = tf.io.gfile.glob(GCS_DS_PATH_1 + '/tfrecords-png-512x512/*.tfrec') \n# tf-flower-photo-tfrec data\n#TRAIN_FILENAMES_2a = tf.io.gfile.glob(GCS_DS_PATH_2 + '/imagenet/tfrecords-jpeg-512x512/*.tfrec')  # pretrained data\n#TRAIN_FILENAMES_2b = tf.io.gfile.glob(GCS_DS_PATH_2 + '/inaturalist_1/tfrecords-jpeg-512x512/*.tfrec')\n#TRAIN_FILENAMES_2c = tf.io.gfile.glob(GCS_DS_PATH_2 + '/openimage/tfrecords-jpeg-512x512/*.tfrec')\n#TRAIN_FILENAMES_2d = tf.io.gfile.glob(GCS_DS_PATH_2 + '/tf_flowers/tfrecords-jpeg-512x512/*.tfrec')\nTRAIN_FILENAMES_2e = tf.io.gfile.glob(GCS_DS_PATH_2 + '/oxford_102/tfrecords-jpeg-512x512/*.tfrec')  \n#TRAIN_FILENAMES_2  = TRAIN_FILENAMES_2b + TRAIN_FILENAMES_2c + TRAIN_FILENAMES_2d + TRAIN_FILENAMES_2e\n\n# this challenge + external training images\nTRAIN_FILENAMES    = TRAIN_FILENAMES_0 + TRAIN_FILENAMES_2e\n\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nNUM_TRAIN_IMAGES = count_data_items(TRAIN_FILENAMES)\nNUM_VAL_IMAGES   = count_data_items(VAL_FILENAMES)\nNUM_TEST_IMAGES  = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH  = NUM_TRAIN_IMAGES // BATCH_SIZE\nAUTO             = tf.data.experimental.AUTOTUNE\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# class names of flowers in the order of label idnum\nCLASSES = [\n    'pink primrose',        'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',      'wild geranium',         # 00-04\n    'tiger lily',           'moon orchid',               'bird of paradise', 'monkshood',      'globe thistle',         # 05-09\n    'snapdragon',           \"colt's foot\",               'king protea',      'spear thistle',  'yellow iris',           # 10-14\n    'globe-flower',         'purple coneflower',         'peruvian lily',    'balloon flower', 'giant white arum lily', # 15-19\n    'fire lily',            'pincushion flower',         'fritillary',       'red ginger',     'grape hyacinth',        # 20-24\n    'corn poppy',           'prince of wales feathers',  'stemless gentian', 'artichoke',      'sweet william',         # 25-29\n    'carnation',            'garden phlox',              'love in the mist', 'cosmos',         'alpine sea holly',      # 30-34\n    'ruby-lipped cattleya', 'cape flower',               'great masterwort', 'siam tulip',     'lenten rose',           # 35-39\n    'barberton daisy',      'daffodil',                  'sword lily',       'poinsettia',     'bolero deep blue',      # 40-44\n    'wallflower',           'marigold',                  'buttercup',        'daisy',          'common dandelion',      # 45-49\n    'petunia',              'wild pansy',                'primula',          'sunflower',      'lilac hibiscus',        # 50-54\n    'bishop of llandaff',   'gaura',                     'geranium',         'orange dahlia',  'pink-yellow dahlia',    # 55-59\n    'cautleya spicata',     'japanese anemone',          'black-eyed susan', 'silverbush',     'californian poppy',     # 60-64\n    'osteospermum',         'spring crocus',             'iris',             'windflower',     'tree poppy',            # 65-69\n    'gazania',              'azalea',                    'water lily',       'rose',           'thorn apple',           # 70-74\n    'morning glory',        'passion flower',            'lotus',            'toad lily',      'anthurium',             # 75-79\n    'frangipani',           'clematis',                  'hibiscus',         'columbine',      'desert-rose',           # 80-84\n    'tree mallow',          'magnolia',                  'cyclamen ',        'watercress',     'canna lily',            # 85-89\n    'hippeastrum ',         'bee balm',                  'pink quill',       'foxglove',       'bougainvillea',         # 90-94\n    'camellia',             'mallow',                    'mexican petunia',  'bromelia',       'blanket flower',        # 95-99\n    'trumpet creeper',      'blackberry lily',           'common tulip',     'wild rose'                                #100-103\n]    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Functions to handle data"},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),    # shape [] means single element\n        # class is missing, to be predicted flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of (image, idnum) pairs\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n    \n    # automatically interleaves reads from multiple file\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) \n    \n    # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.with_options(ignore_order) \n    \n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    dataset = dataset.map(read_labeled_tfrecord if labeled \n                          else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    return dataset\n\ndef get_validation_dataset(filenames):\n    dataset = load_dataset(filenames,labeled=True, ordered=False)\n    dataset = dataset.cache()\n    dataset = dataset.shuffle(buffer_size=1920)\n    dataset = dataset.batch(BATCH_SIZE)\n    # prefetch next batch while training (autotune prefetch buffer size)\n    dataset = dataset.prefetch(AUTO) \n    return dataset\n\ndef get_test_dataset(filenames, ordered=True):  # order matters to submit predictions to Kaggle\n    dataset = load_dataset(filenames, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    # prefetch next batch while training (autotune prefetch buffer size)\n    dataset = dataset.prefetch(AUTO) \n    return dataset\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data augmentation options  \n\nData augmentation is a way to reduce overfitting by randomly altering the training images as they are fit to the model for training.  Thus the images vary different at each epoch as though they are a different dataset, thus increasing the data size.  Here is more info at [TensorFlow](https://www.tensorflow.org/tutorials/images/data_augmentation#overview).\n\n\n## 1. Use tf.image or tfa.image (on single image)\n\nAs shown in the data_augment(image, label) function below, tf.image is used to apply random augmentation on one image at a time:\n\n    image = tf.image.random_flip_left_right(image) \n    image = tf.image.random_contrast(image, lower=0.8, upper=1.2)\n    image = tf.image.random_brightness(image,max_delta=0.1)\n    image = tf.image.random_saturation(image, lower=0.7, upper=1.3)\n    image = tfa.image.mean_filter2d(image, filter_shape = 3)\n    etc\n\nThis function is then used to map to a batch of images at the time images are fetched prior to model training:\n\n    if augmentation:\n        # map the data_augment function to each image in dataset prefetched during training\n        dataset = dataset.map(data_augment, num_parallel_calls=AUTO)   \n        \n\n## 2. Use Keras preprocessing layers (on batch of images)\n\nI explored a few image preprocessing layers for augmentation:\n\n    data_aug_layers = tf.keras.Sequential([\n        tf.keras.layers.experimental.preprocessing.RandomRotation(\n            0.125, fill_mode='constant'),\n        tf.keras.layers.experimental.preprocessing.RandomZoom(\n            (-0.17, -0.01), fill_mode='constant'),\n        tf.keras.layers.experimental.preprocessing.RandomFlip(mode='horizontal'),\n        tf.keras.layers.experimental.preprocessing.RandomTranslation(\n            (-.15,.15), (-.15,.15), fill_mode='constant')\n    ])\n\n* **Option 1**: Apply the preprocessing layers to the dataset at the time images are fetched prior to model training:\n\n        if augmentation:  \n            # apply data augmentation preprocessing layers in batch of images\n            dataset = dataset.map(lambda image, y: (data_aug_layers(image, training=True), y))  \n\n* **Option 2**: Make the preprocessing layers part of the model  \n\n        with strategy.scope():    \n        pretrained_model = efn.EfficientNetB7(\n            weights='imagenet', \n            include_top=False ,\n            input_shape=[*IMAGE_SIZE, 3]\n        )\n        pretrained_model.trainable = True # transfer learning\n        model = tf.keras.Sequential([\n            pretrained_model, \n            data_aug_layers,   # preprocessing layers as part of the model                  \n            tf.keras.layers.GlobalAveragePooling2D(),\n            tf.keras.layers.Dense(len(CLASSES), kernel_regularizer=regularizers.L2(0.005), \n                activation='softmax')\n        ])\n\n\n## 3. ImageDataGenerator (on single image)\n \nI am adding this third option to randomly tranform my training dataset.  This is great because one ImageDataGenerator with one random_transform method can do a lot of difference random augmentation on an image in one pass.  More info at [TensorFlow](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator).\n\n    tf.keras.preprocessing.image.ImageDataGenerator(\n        featurewise_center=False, samplewise_center=False,\n        featurewise_std_normalization=False, samplewise_std_normalization=False,\n        zca_whitening=False, zca_epsilon=1e-06, rotation_range=0, width_shift_range=0.0,\n        height_shift_range=0.0, brightness_range=None, shear_range=0.0, zoom_range=0.0,\n        channel_shift_range=0.0, fill_mode='nearest', cval=0.0,\n        horizontal_flip=False, vertical_flip=False, rescale=None,\n        preprocessing_function=None, data_format=None, validation_split=0.0, dtype=None\n    )\n     \nMy understanding is to first create the image.ImageDataGenerator with whatever data argumentation arguments you choose. Here is an example from TensorFlow:\n \n    img_gen = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255, rotation_range=20)\n\nThen, use the generator with the [random_transform method](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator#random_transform) to apply a random transformation to a single image according to the pre-set arguments/ranges of the generator.  Below, input x is a 3D tensor, single image. The output is a randomly transformed version of x of the same shape.\n\n    random_transform(\n        x, seed=None\n    )"},{"metadata":{},"cell_type":"markdown","source":"# Functions for data augmentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"##################### Keras preprocessing layers #####################\n\n# create image augmentation layers\n# 0.1 rotation = 360*0.15 = 36 deg\ndata_aug_layers = tf.keras.Sequential([\n    tf.keras.layers.experimental.preprocessing.RandomRotation(\n        0.125, fill_mode='constant'),\n])\n# these layers removed:\n#    tf.keras.layers.experimental.preprocessing.RandomZoom(\n#        (-0.17, -0.01), fill_mode='constant')   \n#    tf.keras.layers.experimental.preprocessing.RandomFlip(\n#        mode='horizontal'), \n#    tf.keras.layers.experimental.preprocessing.RandomTranslation(\n#        (-.15,.15),(-.15,.15), fill_mode='constant')\n\n############# ImageDataGenerator - random transformation #############\n\n# create an ImageDataGenerator \n# update this based on image augmenation exploration results\nimg_gen = tf.keras.preprocessing.image.ImageDataGenerator(\n    rotation_range=36, width_shift_range=0.15, height_shift_range=0.15,\n    brightness_range=None, zoom_range=[1.0, 1.25], fill_mode='constant', \n    horizontal_flip=True, preprocessing_function=None)\n\n# define data augmentation function with random_transform method \n# for dataset.map( ... )\ndef img_gen_random_transform(image, label):\n    # apply random_transform method to single image\n    image = img_gen.random_transform(image)\n    return image, label\n\n\n### tf.image; tfa.image; ImageDataGenerator random_transform method ###\n\n# define data augmentation function, one image at a time                  \ndef data_augment(image,  label):\n    \n    # using tf.image \n    image = tf.image.random_flip_left_right(image) \n    image = tf.image.random_contrast(image, lower=0.8, upper=1.2)\n    image = tf.image.random_brightness(image, max_delta=0.1) \n    image = tf.image.random_saturation(image, lower=0.7, upper=1.3)\n    # these commented out:\n    # Pad the image with a black, 90-pixel border\n    #image = tf.image.resize_with_crop_or_pad(\n    #            image, HEIGHT + 180, WIDTH + 180\n    #)\n    # Randomly crop to original size from the padded image\n    #image = tf.image.random_crop(image, size=[*IMAGE_SIZE,3])\n\n    # using tfa.image \n    #rdn = tf.random.normal([1], mean=0, stddev=1, dtype=tf.float32) \n    #if rdn > 2.0:  # blur 2.5% of the images (1 tail, 2 stddev above mean)\n    #    image = tfa.image.mean_filter2d(image, filter_shape = 3,\n    #                                   padding='constant')\n    \n    # using ImageDataGenerator random_transform method     \n    #image = img_gen.random_transform(image)  # didn't work\n        \n    return image, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Function to get training dataset  \n### With option for data augmentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"# get training datatset with augmentation option\ndef get_training_dataset(filenames, augmentation=False):\n    dataset = load_dataset(filenames, labeled=True, ordered=False)\n    if augmentation:\n        # map the data_augment function across images of dthe ataset \n        dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n        # map the img_gen_random_transform function across images of the dataset\n        #dataset = dataset.map(img_gen_random_transform, num_parallel_calls=AUTO)  # didn't work\n    \n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(buffer_size=1920)\n    dataset = dataset.batch(BATCH_SIZE)\n    \n    if augmentation:\n        # apply data augmentation preprocessing layers in batch of images\n        dataset = dataset.map(lambda image, y: (data_aug_layers(image, training=True), y))\n        \n    dataset = dataset.prefetch(AUTO)  # prefetch next batch while training\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Functions to visualize images in batches"},{"metadata":{"trusted":true},"cell_type":"code","source":"np.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(databatch):\n    images, labels = databatch\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    class_labels = []\n    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        class_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    else:\n        for num in enumerate(numpy_labels):\n            class_labels.append(CLASSES[num[1]])\n    return numpy_images, class_labels\n\ndef show_images(databatch, row=6, col=8):  # row, col of subplots\n    FIGSIZE = (col*3, row*3)  # 3X3 inch per image\n    plt.figure(figsize=FIGSIZE)\n    images, num_labl = batch_to_numpy_images_and_labels(databatch)\n    for j in range(row*col):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.title(num_labl[j])\n        plt.imshow(images[j])\n    plt.show()\n# ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image Augmentation Exploration\n#### get training dataset with `augmentation=False`"},{"metadata":{"trusted":true},"cell_type":"code","source":"# get training dataset with augmentation=False\nno_aug_train_set = get_training_dataset(TRAIN_FILENAMES, augmentation=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Explore image augmentation: tf.image & tfa.image"},{"metadata":{"trusted":true},"cell_type":"code","source":"# function to show image with random data augmentation\ndef show_data_aug(image):\n    ROW=len(images)\n    COL=7  # 1 no-aug plus 6 aug images\n    plt.figure(figsize=(COL*2,ROW*2))\n    i=0\n    for image in images:\n        plt.subplot(ROW,COL,i*COL+1)\n        plt.title('rdm flip L/R')\n        plt.axis('off')  \n        # augmented with random flip\n        plt.imshow(tf.image.random_flip_left_right(image))       \n\n        plt.subplot(ROW,COL,i*COL+2)\n        plt.title('resize & rdm crop')\n        plt.axis('off')    \n        # Pad the image with a black, 90-pixel border\n        image1 = tf.image.resize_with_crop_or_pad(\n            image, HEIGHT + 180, WIDTH + 180\n        )\n        # Randomly crop to original size from the padded image\n        image1 = tf.image.random_crop(image1, size=[*IMAGE_SIZE,3])\n        plt.imshow(image1)\n\n        plt.subplot(ROW,COL,i*COL+3)\n        plt.title('rdm contrast')\n        plt.axis('off')\n        # augmented with contrast\n        plt.imshow(tf.image.random_contrast(image, lower=0.8, upper=1.2))  \n\n        plt.subplot(ROW,COL,i*COL+4)\n        plt.title('rdm brightness')\n        plt.axis('off')\n        # augmented with brightness\n        plt.imshow(tf.image.random_brightness(image, max_delta=0.1))       \n\n        plt.subplot(ROW,COL,i*COL+5)\n        plt.title('no aug')\n        plt.axis('off')\n        plt.imshow(image)\n\n        plt.subplot(ROW,COL,i*COL+6)\n        plt.title('rdm saturation')\n        plt.axis('off')\n        # augmented with saturation\n        plt.imshow(tf.image.random_saturation(image, lower=0.7, upper=1.3))  \n\n        plt.subplot(ROW,COL,i*COL+7)\n        plt.title('rdm blur')\n        plt.axis('off')        \n        # ouput a rdm value from a normal distribtion \n        rdn = tf.random.normal([1], mean=0, stddev=1, dtype=tf.float32)              \n        if rdn > 2.0:  # 2 stddev above mean  \n            # blur 2.5% of the images\n            # using tfa.image mean filter\n            plt.imshow(\n                tfa.image.mean_filter2d(image, filter_shape = 3,\n                padding='constant')\n            )  \n        else:\n            plt.imshow(image)\n            \n        i+=1\n        \n    plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Re-run these codes to get the next batch of no aug training images\nno_aug_train_batch = (next(iter(no_aug_train_set.unbatch().batch(16)))) # get a batch for \nimages, _ = batch_to_numpy_images_and_labels(no_aug_train_batch)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Re-run this after adjusting image augmentation settings \n#   of the show_data_aug() function\n# compare no aug training images with random data augmentation\nprint('Training Dataset')\nprint('Image Augmentation with tf.image and tfa.image')\nshow_data_aug(images)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Explore image augmentation: for keras.layers"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Run these to visualize effects before implementing in\n#     tf.keras.layers.experimental.preprocessing.Random___()\nprint('Training Dataset')\nprint('Image Augmentation with tf.keras.preprocesing.image.random ...')\nROW=len(images)\nCOL=4  # 1 no-aug plus 3 aug images\nplt.figure(figsize=(COL*3,ROW*3))\ni=0\nfor image in images:\n    plt.subplot(ROW,COL,i*4+1)\n    plt.title('no aug')\n    plt.axis('off')\n    plt.imshow(image)\n    \n    plt.subplot(ROW,COL,i*4+2)\n    plt.title('rdm shift')\n    plt.axis('off')\n    # random shift on one numpy image tensor \n    # compared to tf.keras.layers.experimental.preprocessing.RandomTranslation(...)\n    image2 = tf.keras.preprocessing.image.random_shift(\n        image, wrg=0.15, hrg=0.15, row_axis=1, col_axis=2, channel_axis=2,\n        fill_mode='constant'\n    )    \n    plt.imshow(image2) \n\n    plt.subplot(ROW,COL,i*4+3)\n    plt.title('rdm 45-deg rotation')\n    plt.axis('off')\n    # random rotation on one numpy image tensor\n    # compared to tf.keras.layers.experimental.preprocessing.RandomRotation(...)\n    image3 = tf.keras.preprocessing.image.random_rotation(\n        image, rg=45, row_axis=1, col_axis=2, channel_axis=2, fill_mode='constant'\n    )\n    plt.imshow(image3)\n\n    plt.subplot(ROW,COL,i*4+4)\n    plt.title('rdm zoom')\n    plt.axis('off')\n    # random zoom on one numpy image tensor\n    # comapred to tf.keras.layers.experimental.preprocessing.RandomZoom(...)\n    image4 = tf.keras.preprocessing.image.random_zoom(\n        image, (.75, 1.0), row_axis=1, col_axis=2, channel_axis=2, fill_mode='constant'\n    )\n    plt.imshow(image4)\n    i+=1\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Explore image augmentation: ImageDataGenerator"},{"metadata":{"trusted":true},"cell_type":"code","source":"# create an ImageDataGenerator for random transformation\nexplore_img_gen = tf.keras.preprocessing.image.ImageDataGenerator(\n    rotation_range=45, width_shift_range=0.2, height_shift_range=0.2,\n    brightness_range=None, zoom_range=[0.75, 1.0], fill_mode='constant', \n    horizontal_flip=True, preprocessing_function=None\n)\n\nprint('Training Dataset')\nprint('Image Augmentation with random_transform method from ImageDataGenerator')\ni = 0\nROW=8  # rows of subplots\nCOL=4  # cols of subplots\nplt.figure(figsize=(COL*3.4,ROW*3))\nfor im in images:\n    plt.subplot(ROW,COL,i*2+1)\n    plt.title('no augmentation')\n    plt.axis('off')\n    plt.imshow(im)\n    plt.subplot(ROW,COL,i*2+2)\n    plt.title('rdm transorm from img_gen')\n    plt.axis('off')\n    plt.imshow(explore_img_gen.random_transform(im))\n    i+=1\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Decide on final settings for image augmentation\nAfter repeated trials of different settings for tf.image, tfa.image, tf.kersa.preprocessing.image and the random_transform method of ImageDataGenerator, I decided on the final settings for all image augmentation techniques.  Note: for demonstration purpose, some augmentations were actually implemented with keras.layers, where the final settings for augmentation were adjusted, too.  The following line get the training images from the GCS with `augmentation=True` with final augmenation setting prior to model training:   \n\n`training_dataset = get_training_dataset(TRAIN_FILENAMES, augmentation=True)` "},{"metadata":{},"cell_type":"markdown","source":"# Visualization Samples of All Datasets\n\n### observations  \n\nI ran the following visualization multiple times to compare the augmented training images with the validation images and with the test images, and to make sure that I had enough randomness in image augmentation but had not introduced characteristics that are not part of the test dataset.  \n\nThis also allowed me to get an idea about the three datasets.  I was surprised to see a picture that looks like a bridge (no flowers), a little girl (no flowers) and a tatoo of flowers on someone's leg in the training dataset.  There is a blank (white) picture (label=70) in the validation dataset.  There is a picture of a fountain (no flowers) in the test dataset.  There are pictures with humans, pets and insects in the picture and a hand holding a flower with flowers in all datasets.  Pictures taken from top view and side views, a single flower or flowers in bundles, close-up and from far in a meadow or along a riverside, zoomed-in and cropped, etc are common in all datasets."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get training dataset WITH all final data augmentation\ntraining_dataset   = get_training_dataset(TRAIN_FILENAMES, augmentation=True)\nvalidation_dataset = get_validation_dataset(VAL_FILENAMES)\ntest_dataset       = get_test_dataset(TEST_FILENAMES, ordered=False)  # not for prediction","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# you may run these lines multiple times to view different samples from the image sets\nR = 7     # rows of subplots/images\nC = 6     # cols of subplots/images\nB = R*C   # number of images in a batch\nprint('Training Images WITH random data augmentation')\nshow_images(next(iter(training_dataset.unbatch().batch(B))), row=R, col=C)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# you may run these lines multiple times to view different samples from the image sets\nprint('Validation Images')\nshow_images(next(iter(validation_dataset.unbatch().batch(B))), row=R, col=C)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Shuffle test dataset to sample for visualiation  \n\n    shuffle(\n        buffer_size, seed=None, reshuffle_each_iteration=None\n    )\n\n`test_dataset.shuffle( ... )`below particularly randomly shuffles the test dataset for visualization.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"# this needs TPU to run\n# you may run these lines multiple times to view different samples from the image sets\nprint('Test Images - shuffled')\nshow_images(next(iter(test_dataset.shuffle(buffer_size=NUM_TEST_IMAGES).unbatch().batch(B))), \n            row=R, col=C)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# get datasets for training\ntraining_dataset   = get_training_dataset(TRAIN_FILENAMES, augmentation=True) # with augmentation\nvalidation_dataset = get_validation_dataset(VAL_FILENAMES)\ntest_dataset       = get_test_dataset(TEST_FILENAMES, ordered=True)  \nprint('trainin dataset:    ', training_dataset)\nprint('validation dataset: ', validation_dataset)\nprint('test dataset:       ', test_dataset)\n\n'''  layers created earlier and applied to prefetched the training data prior to fitting to the model\n# make data_aug_layers part of the model\ntf.config.set_soft_device_placement(True)   \n\n# create image augmentation layers\n    data_aug_layers = tf.keras.Sequential([\n        tf.keras.layers.experimental.preprocessing.RandomRotation(\n            0.1, fill_mode='wrap'),\n        tf.keras.layers.experimental.preprocessing.RandomZoom(\n            (-0.17, -0.01), fill_mode='constant'),\n        tf.keras.layers.experimental.preprocessing.RandomFlip('horizontal'),\n        tf.keras.layers.experimental.preprocessing.RandomTranslation(\n            (-.15,.15),(-.15,.15), fill_mode='constant')\n    ])\n'''  \n\n# With pretrained model: DenseNet201\nwith strategy.scope():    \n    pretrained_model = tf.keras.applications.DenseNet201(\n        weights='imagenet', \n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    pretrained_model.trainable = True # transfer learning\n    model = tf.keras.Sequential([\n        pretrained_model,\n        # the following is commented out: layers applied to the training set instead \n        #data_aug_layers,                    \n        tf.keras.layers.Dropout(0.1),  \n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(104, kernel_regularizer=regularizers.L2(0.003), \n                              activation='softmax')\n    ])\n\n# display model summary, including all layers, the output shape and the number of parameters for each layer,  \n#  the number of trainable parameters and the number of non-trainable parameters. \nmodel.summary()\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Adversarial Regularization"},{"metadata":{},"cell_type":"markdown","source":"\n### wrap the model with adversarial regularization\n```\nadv_config = nsl.configs.make_adv_reg_config(\n    multiplier=0.2,\n    adv_step_size=0.05,\n    adv_grad_norm = 'infinity'  # from TensorFlow tutorial\n)\nadv_model = nsl.keras.AdversarialRegularization(\n    model,\n    adv_config=adv_config\n)\n```\n### compile the adv_model\n```\nadv_model.compile(\n    optimizer='adam',\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy']\n)\n```"},{"metadata":{},"cell_type":"markdown","source":"# Compile the model \n\n`optimizer='adam'` implements the Adam algorithm with some default values set for some arguments, e.g. learning_rate.  Adam optimization is a stochastic gradient descent method.\n\n`loss = 'sparse_categorical_crossentropy'` specifies that crossentropy metric is computed between the labels and predictions.  This metric is used when there are two or more label classes. Labels are expected to be provided as integers.  In this floser classification challenge, there are 104 different classes of flowers.  \n\n`metrics=['sparse_categorical_accuracy']`  \nInteger labels are used in the training, validation and test datasets. Thus metric is set to use sparse categorical accuracy, which calculates how often predictions matches the integer labels. "},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(\n    optimizer='adam',\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy']\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Learning Rate Scheduler callback  \nThe [Learning Rate Scheduler callback](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/LearningRateScheduler) gets the updated learning rate value from schedule function at the beginning of each epoch.  The schedule function takes an epoch index (integer, indexed from 0) and current learning rate (float) as inputs and returns a new learning rate as output (float).  \n\n    tf.keras.callbacks.LearningRateScheduler(\n        schedule, verbose=0\n    )"},{"metadata":{"trusted":true},"cell_type":"code","source":"# define a fine-tuned schedule for the Learning Rate Scheduler \ndef exponential_lr(epoch,\n                  start_lr=0.00001,min_lr=0.00001,max_lr=0.00005,\n                  rampup_epochs = 5, sustain_epochs = 0,\n                  exp_decay = 0.8):  # original exp_decay = 0.8\n    def lr(epoch, start_lr, min_lr,max_lr,rampup_epochs,sustain_epochs,\n          exp_decay):\n        # linear increase from start to rampup_epochs\n        if epoch < rampup_epochs:\n            lr= ((max_lr-start_lr)/\n                rampup_epochs * epoch + start_lr)\n        elif epoch < rampup_epochs + sustain_epochs:\n            lr = max_lr \n        else:\n            lr = ((max_lr - min_lr)* exp_decay ** (epoch-rampup_epochs-sustain_epochs)\n                  + min_lr)\n            \n        return lr\n    return lr(epoch,start_lr,min_lr,max_lr,rampup_epochs,sustain_epochs,exp_decay)\n\n# set learning rate scheduler for callback\nlr_callback = tf.keras.callbacks.LearningRateScheduler(schedule=exponential_lr,verbose=True)\n\n# learning rate chart\nepoch_rng = [i for i in range(EPOCHS)] \ny = [exponential_lr(x) for x in epoch_rng]\nplt.plot(epoch_rng,y)\nplt.xlim(-1, EPOCHS)\n\nprint(\"Learning rate schedule: start = {:.3g}; peak = {:.3g}; end = {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EarlyStopping callback\nReference: https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping    \nThe metric \"val_loss' will be monitored and allows training to be stopped early when the metric stops improving.  I set patience = 2, so training will be stopped after 2 epochs with no improvement.\n\n    tf.keras.callbacks.EarlyStopping(\n        monitor='val_loss', min_delta=0, patience=0, verbose=0,\n        mode='auto', baseline=None, restore_best_weights=False\n    )"},{"metadata":{"trusted":true},"cell_type":"code","source":"# set earlystopping for callback\n# Stop training when a monitored metric has stopped improving\nes_callback = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Checkpoint callback\nI referenced tutorials on TensorFlow on [Save and load models](https://www.tensorflow.org/tutorials/keras/save_and_load) and on [Save checkpoints during training](\nhttps://www.tensorflow.org/tutorials/keras/save_and_load#save_checkpoints_during_training)  \n\n    tf.keras.callbacks.ModelCheckpoint(\n        filepath, monitor='val_loss', verbose=0, save_best_only=False,\n        save_weights_only=False, mode='auto', save_freq='epoch',\n        options=None, **kwargs\n    )\n\nWith `tf.keraas.callbacks.ModelCheckpoint(..., save_best_only=True, save_weights_only=False, ...)`, I saved my best model after each epoch if it is considered the best at the time during training.  With the saved model, I have two options:  \n* After training, I may load my best model prior to computing predictions. (Trained model at the last epoch may not be the best.) \n* I may download my best model as pretrained base model to use in other platform.  When creating a notebook version using \"Save & Run All\", the model will be saved locally and preserved as output in my Kaggle working directory: `/kaggle/working/`.  I may download and use the model at another computing platform, like Google Colab. I may use the saved model as a base model to build on it by adding more layers, or make an adversarial model with it, etc., and see whether that makes a difference in the classification accuracy.\n\n## Writing checkpoints locally from a TPU model\nI spent a lot of time looking up for a solution for the UnimplementedError, so I want to capture what I found here. The following [example from Kaggle](https://www.kaggle.com/docs/tpu#tpu5a) shows an important argument for tf.keras.callbacks.ModelCheckpoint: `options=save_locally`.  And save_locally is configured to save model on `experimental_io_device='/job:localhost'`\n```\nsave_locally   = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')\ncheckpoints_cb = tf.keras.callbacks.ModelCheckpoint('./checkpoints', options=save_locally)\nmodel.fit( . . . , callbacks=[checkpoints_cb])\n```\nTensorflow has [API documentation](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint) that explains how to set related arguments to save weights only as opposed to save model locally. It look simple Tensorflow's argument table for the argument \"options\": \n\n```\nOptional tf.train.CheckpointOptions object if save_weights_only is true, or, \noptional tf.saved_model.SaveOptions object if save_weights_only is false.\n```\nThat means the following:  \n\n### **A)** If you **save weights only**, i.e. set `save_weights_only=True`:\n\n```\ntf.keras.callbacks.ModelCheckpoint(..., \n                                     save_weights_only=True, \n                                     options=save_locally,\n                                     ...)\n                                     \n```\nthen, `options= ...` must be a **tf.train.CheckpointOptions( ... ) object**, i.e. have `save_locally` pre-defined as below:  \n`save_locally = tf.train.CheckpointOptions(experimental_io_device='/job:localhost')`  \n\nNote: if not saving weights locally, set `options=None`   \n\n### **B)** If you s**ave the model**, i.e. set `save_weights_only=False`:\n```\ntf.keras.callbacks.ModelCheckpoint(...,\n                                   save_weights_only=False`,\n                                   options=save_locally,\n                                   ...)\n```                                   \nthen, `options= ...` must be a **tf.saved_model.SaveOptions( ... ) object**, i.e. have `save_locally` pre-defined as below:  \n`save_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')`\n\n\nIf options to save locally is not set up properly, the following error will occur when checkpoint files are to be saved during training:  \n```\nUnimplementedError - File system scheme '[local]' not implemented \n(file: '/kaggle/working/checkpoint_temp/variables/variables_temp/part-00000-of-00001') \nEncountered when executing an operation using EagerExecutor. \nThis error cancels all future operations and poisons their output tensors.\n```"},{"metadata":{},"cell_type":"markdown","source":"# Create checkpoint callback\nThe following is to save best model locally."},{"metadata":{"trusted":true},"cell_type":"code","source":"# set file path to save model\nBEST_MODEL_PATH = \"/kaggle/working/model_best.h5\"   # where filename='model_best.h5'\nFILE_DIR = os.path.dirname(BEST_MODEL_PATH)         # /kaggle/working/\n\n# Create a checkpoint callback that saves the best trained model locally \n#   save_best_only=True, \nsave_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')\ncp_callback  = tf.keras.callbacks.ModelCheckpoint(filepath=BEST_MODEL_PATH,      \n                   options=save_locally, monitor='val_loss', verbose=1,\n                   save_best_only=True, save_weights_only=False, mode='min')\n\n# show current entries saved in Kaggle output directory\nprint('list of entries contained in', FILE_DIR, tf.io.gfile.listdir(FILE_DIR))\n# the following two lines do the same as the above\nprint('list of entries contained in /kaggle/working/')  \n!ls {FILE_DIR}  # same as !ls \"/kaggle/working/\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train the model - with callbacks list\nNow the model is created and configured with losses and metrics with `model.compile(...)`, it is time to train the model with `model.fit()`, which outputs a History object.  \n\n    historical = model.fit( ...,                  \n                           callbacks=[..., ...]\n                           )                   \n                           \nAs the codes indicated, while the model is being trained, events are recorded into the History object named \"historical\".   This object's attribute, `historical.history` is then used to create plots to show accuracy and loss metrics in the next section.  \n\n### Callbacks\nThe callbacks argument `callbacks=[lr_callback, es_callback, cp_callback]` allows the listed callbacks to appy during training. Here is the list of [keras.callbacks.Callback](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks) instances.  \n\nThe cp_callback works well and saves the best model in the .h5 format.  This allows the best model to be loaded prior to doing predictions on test dataset.  (The trained model at the last epoch may not be necessarily the best model.)  "},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"# fit/train the model \n# save the History object to the variable \"historical\"\n# save checkpoints during training\nhistorical = model.fit(\n    training_dataset, \n    steps_per_epoch=STEPS_PER_EPOCH, \n    epochs=EPOCHS, \n    validation_data=validation_dataset,\n    callbacks=[lr_callback, es_callback, cp_callback]\n) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plots: accuracy and loss metrics\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create plots of loss and accuracy on the training and validation datasets.\n\nacc = historical.history['sparse_categorical_accuracy']\nval_acc = historical.history['val_sparse_categorical_accuracy']\n\nloss = historical.history['loss']\nval_loss = historical.history['val_loss']\n\nepochs_range = range(1, len(historical.history['loss'])+1)\n\nplt.figure(figsize=(14, 14))\nplt.subplot(2, 1, 1)\nplt.plot(epochs_range, acc, label='Training Accuracy')\nplt.plot(epochs_range, val_acc, label='Validation Accuracy')\nplt.legend(loc='lower right')\nplt.title('Training and Validation Accuracy')\nplt.xlabel('Epoch')\n\nplt.subplot(2, 1, 2)\nplt.plot(epochs_range, loss, label='Training Loss')\nplt.plot(epochs_range, val_loss, label='Validation Loss')\nplt.legend(loc='upper right')\nplt.title('Training and Validation Loss')\nplt.xlabel('Epoch')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load the best model saved during training"},{"metadata":{"trusted":true},"cell_type":"code","source":"# show files in local working directories \nprint('list of entries contained in', FILE_DIR, tf.io.gfile.listdir(FILE_DIR)) \n\n# load best model saved by cp_callback during training\nmodel = tf.keras.models.load_model(BEST_MODEL_PATH)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Compute predictions on the test dataset\nIf you are happy with the accuracy and loss charts above, it is time to use the trained model to make predictions with `model.predict(...)`.  \n\n`np.savetxt(...)` will create a file that can be submitted to the competition.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# predict probabilities and match to the most probable integer label for each image\nprint('Computing predictions...')\ntest_images_ds = test_dataset.map(lambda image, idnum: image)\nprobabilities = model.predict(test_images_ds)  \npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)\n\n# create file to submit to the competition\nprint('Generating submission.csv file...')\ntest_ids_ds = test_dataset.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\nnp.savetxt('submission.csv', np.rec.fromarrays([test_ids, predictions]), \n           fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}