{"cells":[{"metadata":{},"cell_type":"markdown","source":"[Keras-Tuner](https://github.com/keras-team/keras-tuner) is a convenient solution for hyperparameter tuning. It provides many built-in utilities with latest Keras APIs. \n\nIn this notebook I will show using a HyperModel available in Keras-Tuner that utilizes [Keras preprocessing layers (KPL)](https://keras.io/guides/preprocessing_layers/) for a hyperparameter tunable image augmentation, combined with a hypermodel for EfficientNet based on newly added implementation in [keras.applications](https://keras.io/api/applications/). Simply applying default setting yields good result.\n\nThis notebook requires TF>=2.3.0, and Keras-Tuner installed from [github head](https://github.com/keras-team/keras-tuner@master).\n\nIn general, hyperparameter searching requires large amount of resources. In the example you can choose `TESTING_LEVEL = 0` for a quick debug, `TESTING_LEVEL = 1` for a relatively short test to see some reasonable result (~0.9), or `TESTING_LEVEL = 2` and set your own search epochs limit for an extended run.\n\n*Note* Keras preprocessing layers is in experimental stage as of TF2.3. It is not fully supporting TPUs, so this notebook will only run on GPUs depite working on this TPU competition. A TPU enabled version that do not use KPL is [here](https://www.kaggle.com/fuyixing/flower-classification-on-tpu-with-keras-tuner).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Configuration\nTESTING_LEVEL = 0\n\nif TESTING_LEVEL == 0:\n    # For debugging\n    IMAGE_SIZE = [224, 224]\n    EPOCHS_SEARCH = 5\n    MAX_TRIALS = 3\n    EPOCHS_FINAL = 50\n    BATCH_SIZE_PER_REPLICA = 16\nelif TESTING_LEVEL == 1:\n    # For relatively short test to see some reasonable result\n    IMAGE_SIZE = [224, 224]\n    EPOCHS_SEARCH = 10\n    MAX_TRIALS = 5\n    EPOCHS_FINAL = 10\n    BATCH_SIZE_PER_REPLICA = 16\nelse:\n    # For an extended run.\n    # Can set even larger MAX_TRIALS and EPOCHS_SEARCH for even better result.\n    IMAGE_SIZE = [224, 224]\n    EPOCHS_SEARCH = 30\n    MAX_TRIALS = 10\n    EPOCHS_FINAL = 10\n    BATCH_SIZE_PER_REPLICA = 32\n    \n\n# load previous keras tuner result from output if possible\n!cp -r ../input/flower-kt-record/flower_classification_kt_kpl .","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q tensorflow==2.3.0\n!pip install -q git+https://github.com/keras-team/keras-tuner@master","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import random, re, math\nimport numpy as np, pandas as pd\nimport matplotlib.pyplot as plt\nimport tensorflow as tf, tensorflow.keras.backend as K\nfrom kaggle_datasets import KaggleDatasets\nprint('Tensorflow version ' + tf.__version__)\nimport kerastuner as kt\ntf.config.optimizer.set_jit(True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # Sync TPU version\n    from cloud_tpu_client import Client\n    c = Client()\n    c.configure_tpu_version(tf.__version__, restart_type='ifNeeded')\n    \n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n    \n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\nBATCH_SIZE = BATCH_SIZE_PER_REPLICA * strategy.num_replicas_in_sync","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we set a few over-all hyperparameters. These may also be searched through Keras-Tuner with customized tuner classes and model bulding function / HyperModels. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Data preparation\nThe code for data preparation are mostly adapted from this [notebook](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96) with some minor modification. Note that I am converting labels to one-hot encoding.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Data access\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('tpu-getting-started')\nprint(GCS_DS_PATH)\n\nGCS_PATH_SELECT = { # available image sizes\n    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'\n}\n\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') # predictions on this dataset should be submitted for the competition","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 102","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dataset functions\n\nAdapted from starter [kernel][1]. Notice that augmentation is not included as a part of data pipeline. Instead, it will be in the model.\n\n[1]: https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.data.experimental import AUTOTUNE\n\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) \n    # For keras.application.efficientnet, inputs range [0, 255]\n    image = tf.ensure_shape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef read_labeled_tfrecord(example, num_classes=len(CLASSES)):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"class\": tf.io.FixedLenFeature([], tf.int64),\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    label = tf.one_hot(label, num_classes)\n    return image, label\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"id\": tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords.\n    # When ordering is not needed, set `ordered=False` for faster loading.\n    \n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False \n\n    # automatically interleaves reads from multiple files\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTOTUNE)\n    # use data as soon as it streams in, rather than in its original order\n    dataset = dataset.with_options(ignore_order)\n    if labeled:\n        dataset = dataset.map(read_labeled_tfrecord,\n                              num_parallel_calls=AUTOTUNE)\n    else:\n        dataset = dataset.map(read_unlabeled_tfrecord,\n                              num_parallel_calls=AUTOTUNE)\n    return dataset\n\n\ndef get_training_dataset(filenames):\n    dataset = load_dataset(filenames, labeled=True, ordered=False)\n    dataset = dataset.repeat()\n    dataset = dataset.shuffle(2048)\n    # Drop remainder to ensure same batch size for all.\n    dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)\n    # Prefetch next batch while training\n    dataset = dataset.prefetch(AUTOTUNE) \n    return dataset\n\ndef get_validation_dataset(filenames):\n    dataset = load_dataset(filenames, labeled=True, ordered=False)\n    # Drop remainder to ensure same batch size for all.\n    dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)\n    # Prefer not prefetching for validation data on GPUs.\n    # dataset = dataset.prefetch(AUTOTUNE)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE, drop_remainder=False)\n    return dataset\n\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_val_samples = count_data_items(VALIDATION_FILENAMES)\nnum_train_samples = count_data_items(TRAINING_FILENAMES)\n\nnum_train_batches = num_train_samples // BATCH_SIZE\nnum_val_batches = num_val_samples // BATCH_SIZE\n\ntrain_ds = get_training_dataset(TRAINING_FILENAMES)\nvalidation_ds = get_validation_dataset(VALIDATION_FILENAMES)\nall_ds = get_training_dataset(TRAINING_FILENAMES + VALIDATION_FILENAMES)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Search Hyper-parameters with Keras-Tuner\n\nNow we will search hyperparameters with Keras-Tuner.\n\nA HyperModel in Keras-Tuner is class with a `build` method that creates a *compiled* Keras model using a set of hyperparameters for each trial. A tuner takes a [HyperModel](https://keras-team.github.io/keras-tuner/documentation/hypermodels/) or simply a model builder function, and tries the combinations of the hyperparameters for times depending on different tuning algorithms (defined by [Oracle](https://keras-team.github.io/keras-tuner/documentation/oracles/)). Each of the built-in [Tuner](https://keras-team.github.io/keras-tuner/documentation/tuners/) have corresponding oracle.\n\nFor simplicity, in this example I only using pre-built HyperModels and Tuner. It is also possible to create any HyperModel or model-building function, to create custom tuning algorithms by subclassing Oracles, and to use custom training loop by [subclassing Tuner](https://keras-team.github.io/keras-tuner/tutorials/subclass-tuner/). Customized training loop can help fully utilizing the pretrained weights by first training some epoches with pretrained layers frozen. \n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Data Augmentation with Keras-Tuner HyperModel\n\nKeras-Tuner provides a built-in image augmenting hypermodel in applications. This HyperModel, `HyperImageAugment`, can either be used as a stand-alone hypermodel or as a part of other HyperModels. It is especially convenient to be used with HyperModel in applications shipped with Keras-Tuner, as this notebook will later demonstrate. Here we first build some sample HyperModel for augmentation to visualize the augmentation.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# function for visualizing augmentation model\ndef visualize(aug_model):\n    row = 3; col = 4;\n    element = train_ds.unbatch().take(1)\n\n    for (img, label) in element:\n        img_batch = tf.repeat(tf.expand_dims(img, 0),row * col,axis=0)\n        img_augment = aug_model(img_batch)\n        plt.figure(figsize=(15, int(15 * row / col)))\n        for j in range(row * col):\n            plt.subplot(row,col, j + 1)\n            plt.axis('off')\n            plt.imshow(img_augment[j,] / 255.)\n        plt.show()\n        break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`HyperImageAugment` implements two types of image augmentations, controlled by choosing `augment_layers` to be 0 or a positive integer. When `augment_layers` is zero, it sequentially apply selected preprocessing layers; otherwise it implements a [RandAugment](https://arxiv.org/abs/1909.13719) style augmentation using the selected layers. \n\nFor demonstration purpose, we first build a hypermodel for augmentation that sequentially apply all layers, selecting only rotation and vertical translation. We set the factor for `rotate` and `translate_y` to be both 0.3 to 0.5, so for each trial, the augmentation factor for each layer is selected in the ranges defined in hypermodel. The augmentation factors for KPLs are mostly designed to be between 0 and 1, whereas each example are augmented with a random magnitude between 0 and the factor.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from kerastuner.applications import HyperImageAugment\n\nhm_aug = HyperImageAugment(\n    input_shape=[*IMAGE_SIZE, 3],\n    augment_layers=0,\n    rotate=[0.3, 0.5], # range of factor of rotation\n    translate_x=None, # horizontal translation is off\n    translate_y=[0.3, 0.5], # range of factor of vertical translation\n    contrast=None) # auto contrast is off","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This can both be seen directly from model summary, or through example data. Notice that all images are both translating and rotating. Also, the translation / rotation are as low as 0 because we are only setting the minimum of `factor` to be 0.3, not minimum of actual per-sample transform to 0.3. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from kerastuner.engine.hyperparameters import HyperParameters\nhp = HyperParameters()\naug_model = hm_aug.build(hp)\naug_model.summary()\n\nvisualize(aug_model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we build a hypermodel for RandAugment selecting rotation and vertical translation. We set the factor for `rotate` and `translate_y` to be both 0.3 to 0.5. For each trial, the augmentation factor for each layer is selected in the ranges defined in hypermodel. We set `augment_layers=[1, 1]` so that there is always one layer applied, either `rotate` or `translate_y` to each *sample*. Hence each figure is either rotated or translated (but not both in this case).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from kerastuner.applications import HyperImageAugment\n\nhm_aug = HyperImageAugment(\n    input_shape=[*IMAGE_SIZE, 3],\n    augment_layers=[1, 1], # only one layer of augmentation\n    rotate=[0.3, 0.5], # range of factor of rotation\n    translate_x=None, # horizontal translation is off\n    translate_y=[0.3, 0.5], # range of factor of vertical translation\n    contrast=None) # auto contrast is off\n\nhp = HyperParameters()\naug_model = hm_aug.build(hp)\naug_model.summary()\n\nvisualize(aug_model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For our searching and training, I will simply use the default options for HyperImageAugment. That is, up to 3 layers of RandAugment style augment with horizontal / vertial translation factor searched in range $[0, 0.4]$, rotation with factor in $[0, 0.5]$, random contrast with factor in $[0, 0.3]$.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from kerastuner.applications import HyperImageAugment\n# Use default setting\nhm_aug = HyperImageAugment(input_shape=[*IMAGE_SIZE, 3]) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The EfficientNet HyperModel\n\nKeras Tuner provides hypermodel for EfficientNet in applications. It utilizes EfficientNet from `tf.keras.applications`, and by default uses weights pretrained on ImageNet. Currently this hypermodel searches version of EfficientNet (B0-B7), type of pooling layer after feature extration (max pooling or average pooling) and initial learning rate. \n\n*A side note*: TF2.3 provides `experimental_steps_per_execution` keyword for `model.compile`. This greatly improves TPU efficiency, and may also slightly improve GPU efficiency.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define HyperModel using built-in application\nfrom kerastuner.applications.efficientnet import HyperEfficientNet\n\nhm = HyperEfficientNet(\n    input_shape=[*IMAGE_SIZE, 3],\n    classes=len(CLASSES),\n    # Augmentation model goes here. It can be HyperModel or Keras Model.\n    augmentation_model=hm_aug) \n\n# Optional: Restrict default hyperparameters.\n# To take effect, pass this `hp` instance when constructing tuner as `hyperparameters=hp`\nfrom kerastuner.engine.hyperparameters import HyperParameters\nhp = HyperParameters()\n# Restrict choice of EfficientNet version from B0-B7 to B0-B3\nhp.Choice('version', ['B0', 'B1', 'B2', 'B3']) \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To fully take advantage of pretrained weights, it is best to first freeze the feature extracting stage of the model; and fine tune the entire model for a few epochs. Here we show this work flow by subclassing the tuner. For a relatively easy task like this one, training EfficientNet from scratch is already capable of reaching good result; but utilizing pretrained weight can siginficantly accelerate convergence.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import copy\n\n# Helper function: re-compile with the same loss/metric/optimizer\ndef recompile(model):\n    metrics = model.compiled_metrics.metrics\n    metrics = [x.name for x in metrics]\n    model.compile(loss=model.loss,\n                  metrics=metrics,\n                  optimizer=model.optimizer)\n\nclass FineTuner(kt.engine.tuner.Tuner):\n    def run_trial(self, trial, *fit_args, **fit_kwargs):       \n        copied_fit_kwargs = copy.copy(fit_kwargs)\n        callbacks = fit_kwargs.pop('callbacks', [])\n        callbacks = self._deepcopy_callbacks(callbacks)\n        copied_fit_kwargs['callbacks'] = callbacks\n        \n        tf.keras.backend.clear_session()\n        model = self.hypermodel.build(trial.hyperparameters)\n        #dry run to build metrics\n        model.evaluate(*fit_args, steps=1, batch_size=1)\n        \n        # freeze pretrained feature extractor\n        for l in model.layers:\n            # For efficientnet implementation we use, layers in the\n            # Feature extraction part of model all have 'block', \n            # 'stem' or 'top_conv' in name.\n            if any(x in l.name for x in ['block', 'stem', 'top_conv']):\n                l.trainable = False\n            if isinstance(l, tf.keras.layers.BatchNormalization):\n                l.trainable = True\n        # it usually suggested to increase learning rate if running with\n        # multiple replica because of the increased batch size.\n        model.optimizer.lr = model.optimizer.lr * strategy.num_replicas_in_sync\n        recompile(model)\n        model.fit(*fit_args, **copied_fit_kwargs)\n        \n        for l in model.layers:\n            l.trainable = True\n        model.optimizer.lr = model.optimizer.lr / 10\n        recompile(model)\n        \n        # TunerCallback reports results to the `Oracle` and save the trained Model.\n        callbacks.append(kt.engine.tuner_utils.TunerCallback(self, trial))\n        \n        model.fit(*fit_args, **copied_fit_kwargs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When `Tuner` is customized, we need to initiate `Oracle` and pass the `Oracle` object to initiate the `Tuner` object. If using built-in tuners, we can directly initiate tuners such as `RandomSearch`. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define Oracle\noracle = kt.tuners.bayesian.BayesianOptimizationOracle(\n    objective='val_accuracy',\n    max_trials=MAX_TRIALS,\n    hyperparameters=hp,\n)\n\n# Initiate Tuner\ntuner = FineTuner(\n    hypermodel=hm,\n    oracle=oracle,\n    directory='flower_classification_kt_kpl',\n    project_name='bayesian_efficientnet',\n    # Distribution strategy is passed in here.\n    distribution_strategy=strategy, \n    # optimizer can be set here.\n    optimizer='adam',\n    metrics=['accuracy'],\n    )\ntuner.search_space_summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`tuner.search()` is called just like the way you would call `model.fit()`.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# tuner.search(train_ds,\n#              epochs=EPOCHS_SEARCH,\n#              validation_data=validation_ds,\n#              steps_per_epoch=num_train_batches,\n#              callbacks=[tf.keras.callbacks.ReduceLROnPlateau(),\n#                         tf.keras.callbacks.EarlyStopping(patience=5)],\n#              verbose=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As long as some trials are complete, we may move on to get the best result up to now even if search fail to finish. Also, as long as the project directory is not deleted, you may run the same code and it will pick up from where it stopped.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"tuner.results_summary()\nmodel = tuner.get_best_models()[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is usually good to fit the best model with all data including validation data after hyperparameter search is done.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train the best model with all data\nmodel.fit(all_ds,\n          epochs=EPOCHS_FINAL,\n          steps_per_epoch=num_train_batches + num_val_batches,\n          callbacks=[tf.keras.callbacks.ReduceLROnPlateau(monitor='loss')],\n          verbose=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prediction and create submission file","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ds_test = get_test_dataset(ordered=True)\n\nprint('Computing predictions...')\npredictions = []\n\nfor i, (test_img, test_id) in enumerate(ds_test):\n    print('Processing batch ', i)\n    probabilities = model(test_img)\n    prediction = np.argmax(probabilities, axis=-1)\n    predictions.append(prediction)\n\npredictions = np.concatenate(predictions)\nprint('Number of test examples predicted: ', predictions.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get image ids from test set and convert to unicode\nds_test_ids = ds_test.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(ds_test_ids.batch(np.iinfo(np.int64).max))).numpy().astype('U')\n\n# Write the submission file\nnp.savetxt(\n    'submission.csv',\n    np.rec.fromarrays([test_ids, predictions]),\n    fmt=['%s', '%d'],\n    delimiter=',',\n    header='id,label',\n    comments='',\n)\n\n# Look at the first few predictions\n!head submission.csv","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}