{"cells":[{"metadata":{},"cell_type":"markdown","source":"The TPUs provided on Kaggle makes training extremely fast, allowing us to explore more hyperparameters to get the best result. [Keras-Tuner](https://github.com/keras-team/keras-tuner) is a convenient solution for hyperparameter tuning.\n\nThis notebook serves a demonstration purpose, so 1) I use only landmarks with at least 250 images; 2) I use internet access to download pretrained weights and packages (for comptetition you need to load those as datasets). This notebook uses built-in tuning algorithms and hypermodels. In particular, I will use a hypermodel based on EfficientNet (shipped in [keras.applications](https://keras.io/api/applications/) since TF2.3) with random search tuning algorithm. We can see EfficientNet is capable to classify the landmarks when there are reasonably amount of image in each category, and hence potentially be a good backbone. However, for actual participation in the competition you may need to think of ways to work with the imbalanced dataset.\n\n*note*: Hyperparameter search is resource consuming by its nature. To get best result, you will want to allow each trial to reach convergence. This notebook, however, will only run the search for relatively fewer epoches for each trial, which effectively searches for the hyperparameters that converges fast instead of reaching best accuracy.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q tensorflow==2.3.0 # Use 2.3.0 for built-in EfficientNet\n!pip install -q git+https://github.com/keras-team/keras-tuner@master # Use github head for newly added TPU support\n!pip install -q cloud-tpu-client # Needed for sync TPU version\n!pip install -U tensorflow-gcs-config==2.3.0 # Needed for using private dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I preprocessed data as TF-record format. This is the suggested way of feeding data to TPUs. Here the TF-record data is savd as a private dataset, so the following lines are needed.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import random, re, math\nimport numpy as np, pandas as pd\nimport matplotlib.pyplot as plt\nimport tensorflow as tf, tensorflow.keras.backend as K\nfrom kaggle_datasets import KaggleDatasets\nprint('Tensorflow version ' + tf.__version__)\nimport kerastuner as kt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Configurations\n\nConfigure for TPU if TPU is available for use. In order to use TF2.3 on TPU, you need to manually configure TPU version using cloud-tpu-client. This is not yet officially supported. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # Sync TPU version\n    from cloud_tpu_client import Client\n    c = Client()\n    c.configure_tpu_version(tf.__version__, restart_type='ifNeeded')\n    \n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n    \n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# from kaggle_secrets import UserSecretsClient\n# user_secrets = UserSecretsClient()\n# user_credential = user_secrets.get_gcloud_credential()\n# user_secrets.set_tensorflow_credential(user_credential)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we set a few over-all hyperparameters. These may also be searched through Keras-Tuner with customized tuner classes and model bulding function / HyperModels. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Configuration\nIMAGE_SIZE = [256, 256]\nEPOCHS_SEARCH = 10\nEPOCHS_FINAL = 20\nSEED = 123\nBATCH_SIZE = 32 * strategy.num_replicas_in_sync\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data preparation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.data.experimental import AUTOTUNE\nbase_path = KaggleDatasets().get_gcs_path('gld-v2-256')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport functools\n\n\ndef create_dataset(file_pattern, allowed_labels, augmentation: bool = False, num_classes=None):\n    # Select only dataset within a list of allowed labels\n    if not num_classes:\n        raise ValueError('num_classses must be set.')\n\n    ignore_order = tf.data.Options()\n    ignore_order.experimental_deterministic = False\n    filenames = tf.io.gfile.glob(file_pattern)\n    filenames = filenames\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTOTUNE).shuffle(1000)\n\n    # Create a description of the features.\n    feature_description = {\n        'image/height': tf.io.FixedLenFeature([], tf.int64, default_value=0),\n        'image/width': tf.io.FixedLenFeature([], tf.int64, default_value=0),\n        'image/channels': tf.io.FixedLenFeature([], tf.int64, default_value=0),\n        'image/format': tf.io.FixedLenFeature([], tf.string, default_value=''),\n        'image/id': tf.io.FixedLenFeature([], tf.string, default_value=''),\n        'image/filename': tf.io.FixedLenFeature([], tf.string, default_value=''),\n        'image/encoded': tf.io.FixedLenFeature([], tf.string, default_value=''),\n        'image/class/label': tf.io.FixedLenFeature([], tf.int64, default_value=0),\n    }\n\n    parse_func = functools.partial(\n        _parse_example,\n        name_to_features=feature_description,\n        augmentation=augmentation\n    )\n    dataset = dataset.with_options(ignore_order)\n    dataset = dataset.map(parse_func, num_parallel_calls=AUTOTUNE)\n\n    def label_predicate(x, y):\n        return tf.greater(tf.reduce_sum(tf.cast(tf.equal(allowed_labels, y), tf.float32)), 0.)\n\n    def relabel(x, y):\n        y = tf.reduce_min(tf.where(tf.equal(allowed_labels, y)))\n        return x, tf.one_hot(y, num_classes)\n\n    dataset = dataset.filter(label_predicate)\n    dataset = dataset.map(relabel, num_parallel_calls=AUTOTUNE)\n    dataset = dataset.repeat()\n\n    dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)\n    dataset = dataset.prefetch(AUTOTUNE)\n    return dataset\n\ndef _parse_example(example, name_to_features, augmentation):\n    parsed_example = tf.io.parse_single_example(example, name_to_features)\n\n    image = parsed_example['image/encoded']\n    image = tf.io.decode_jpeg(image)\n    image = tf.cast(image, tf.float32)\n    image = tf.image.resize(image, IMAGE_SIZE)\n    image.set_shape([*IMAGE_SIZE, 3])\n\n    label = tf.cast(parsed_example['image/class/label'], tf.int64)\n    return image, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# original labelling\ntraining_csv_path = os.path.join(base_path, \"train.csv\")\ntrain_csv = pd.read_csv(str(training_csv_path))\n\n# original labelling\nclean_training_csv_path = os.path.join(base_path, \"train_clean.csv\")\nclean_train_csv = pd.read_csv(str(clean_training_csv_path))\n###\norig_unique_landmark_ids = clean_train_csv[\"landmark_id\"].tolist()\nprint('max label:', max(orig_unique_landmark_ids))\n###\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"landmark_ids_occurance = [len(x.split(\" \")) for x in clean_train_csv[\"images\"]]\n# The labelling used in tfrecord is compressed, corresponding to 0 based id of clean_csv\ncompressed_landmark_ids_to_occurance = list(enumerate(landmark_ids_occurance))\n\n#unique_landmark_ids = [x[0] for x in unique_landmark_ids_to_occurance]\n\nallowed_labels = [x[0] for x in compressed_landmark_ids_to_occurance if x[1] >= 250]\nallowed_labels = tf.convert_to_tensor(allowed_labels, dtype=tf.int64)\n\nnum_samples = sum([x for x in landmark_ids_occurance if x >= 250])\nNUM_CLASSES = len([x for x in landmark_ids_occurance if x >= 250])\n\n\n# unique_landmark_ids_occurance = tf.convert_to_tensor(unique_landmark_ids_occurance)\nprint(num_samples)\nsteps_per_epoch = int(num_samples / BATCH_SIZE)\n_num_samples = steps_per_epoch * BATCH_SIZE","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_tf_records = os.path.join(base_path, 'train*128')\nval_tf_records = os.path.join(base_path, 'val*128')\nall_tf_records = os.path.join(base_path, '*128')\n\nds_train = create_dataset(train_tf_records,\n                          allowed_labels,\n                          num_classes = NUM_CLASSES)\n\nds_val = create_dataset(val_tf_records,\n                        allowed_labels,\n                        num_classes = NUM_CLASSES)\n\nds_all = create_dataset(all_tf_records,\n                        allowed_labels,\n                        num_classes = NUM_CLASSES)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create dataset\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for img, lbl in ds_train.shuffle(10).take(1):\n    plt.imshow(tf.cast(img[0], tf.int32))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Augmentation\nCurrently Keras Preprocessing Layer (KPL) is under experimental stage and is not fully compatible with TPU. Hence augmentation functions adapted from [this notebook](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96) is used. \n\nWhen KPL layer become fully available, you will be able to use HyperModels for augmentation based on KPL layers that is shipped with Keras-Tuner. See [this notebook](https://www.kaggle.com/fuyixing/flower-classification-with-keras-tuner-and-kpl) for a GPU/CPU version of this notebook using KPL based tunable augmentation. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Visualizing examples","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Search Hyper-parameters with Keras-Tuner\n\nNow we search hyperparameters with Keras-Tuner.\n\nA HyperModel in Keras-Tuner is class with a `build` method that creates a *compiled* Keras model using a set of hyperparameters for each trial. A tuner takes a [HyperModel](https://keras-team.github.io/keras-tuner/documentation/hypermodels/) or simply a model builder function, and tries the combinations of the hyperparameters for times depending on different tuning algorithms (defined by [Oracle](https://keras-team.github.io/keras-tuner/documentation/oracles/)). Each of the built-in [Tuner](https://keras-team.github.io/keras-tuner/documentation/tuners/) have corresponding oracle.\n\nIn this example I only use pre-built HyperModel and Tuner. It is also possible to create any HyperModel or model building function, and to create custom tuning algorithms by subclassing Oracles, and to use custom training loop by [subclassing Tuner](https://keras-team.github.io/keras-tuner/tutorials/subclass-tuner/).\n\nTo fully utilize the advantage of pre-trained weight, it is usually good to first freeze most of the layers in training. This can be done by overriding `_compile` in `HyperEfficientNet` application; while for other cases you can always create your own HyperModel to specify desired behavior of the model building process.\n\nTF2.3 provides `experimental_steps_per_execution` keyword for `model.compile`. This greatly improves TPU efficiency. To use the feature in Keras Tuner, you will need to modify the model building function and then use the new subclass for HyperModel.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from kerastuner.applications.efficientnet import HyperEfficientNet\nclass MyHyperEfficientNet(HyperEfficientNet):\n    def _compile(self, model, hp):\n        \n        for l in model.layers:\n            # For efficientnet implementation we use, layers in the\n            # Feature extraction part of model all have 'block' in name.\n            if 'block' in l.name:\n                l.trainable = False\n                \n        super(MyHyperEfficientNet, self)._compile(model, hp)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define HyperModel using built-in application\nfrom kerastuner.applications.efficientnet import HyperEfficientNet\nhm = HyperEfficientNet(input_shape=[*IMAGE_SIZE, 3] , classes=NUM_CLASSES)\n\n# Optional: Restrict default hyperparameters.\n# To take effect, pass this `hp` instance when constructing tuner as `hyperparameters=hp`\nfrom kerastuner.engine.hyperparameters import HyperParameters\nhp = HyperParameters()\nhp.Choice('version', ['B0', 'B1', 'B2', 'B3']) #restrict choice of EfficientNet version from B0-B7 to B0-B4\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define Oracle\noracle = kt.tuners.randomsearch.RandomSearchOracle(\n    objective='val_accuracy',\n    max_trials=5,\n    hyperparameters=hp,\n)\n\n# Initiate Tuner\ntuner = kt.engine.tuner.Tuner(\n    hypermodel=hm,\n    oracle=oracle,\n    distribution_strategy=strategy, # This strategy's scope is used for building each model during the search.\n    directory='landmark',\n    project_name='randomsearch_efficientnet',\n)\ntuner.search_space_summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_split = 0.2\nnum_val_samples = int(num_samples * val_split)\nnum_train_samples = int(num_samples * (1 - val_split))\n\nnum_train_batches = num_train_samples // BATCH_SIZE\nnum_val_batches = num_val_samples // BATCH_SIZE","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tuner.search(ds_train,\n             epochs=EPOCHS_SEARCH,\n             validation_data=ds_val,\n             steps_per_epoch=num_train_batches,\n             validation_steps=num_val_batches,\n             verbose=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As long as some trials are complete, we may move on to get the best result up to now even if search fail to finish. Also, as long as the project directory is not deleted, you may run the same code and it will continue search from where it stopped.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"tuner.results_summary()\nmodel = tuner.get_best_models()[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is usually good to fit the best model with all data including validation data after hyperparameter search is done.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train the best model with all data\nmodel.fit(ds_all,\n          epochs=EPOCHS_FINAL,\n          steps_per_epoch=num_train_batches + num_val_batches,\n          callbacks=[tf.keras.callbacks.ReduceLROnPlateau()],\n          verbose=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}