{"cells":[{"metadata":{},"cell_type":"markdown","source":"I've been curious about TPUs since they were announced on Kaggle and this seems like a good opportunity to learn about them. It's also a good chance to learn a little about tfrecords and tensorflows data api.\n\nAs always, let's import the required libraries.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd \nimport tensorflow as tf\nimport matplotlib.pyplot as plt\n\nfrom kaggle_datasets import KaggleDatasets","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q efficientnet\nimport efficientnet.tfkeras as efn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Setup the TPU\n\nAs explained in notebooks such as this [one](https://www.kaggle.com/ryanholbrook/create-your-first-submission), TPUs are basically a bunch of GPU chips that are grouped together for one model to train on. By using a TPU we replicate the model eight times and split a batch of images to train on these eight models at the same time. Theoretically then, this gives us the speed of a GPU multiplied by eight.\n\nTo use a TPU we need a strategy. My understanding of a strategy is that it is like a set of instructions telling tensorflow how to replicate the model and assign these replicas to the eight GPUs. I presume this strategy also includes instructions on how to reconstruct the model from these eight trained replicas. Anyway, here is some code to form that strategy.","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The dataset also needs to be close to the TPU for the training meaning that a little extra logic is needed to get the path to the dataset. This line of code gets the path to the location that the data is kept in. I believe that this is a bucket in Google cloud.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_DS_PATH = KaggleDatasets().get_gcs_path('tpu-getting-started')\nprint(GCS_DS_PATH)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define Hyper-parameters\n\nTo make it easier to manage the models hyper-parameters I'll define them here as global variables.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_SIZE = 512\nEPOCHS = 35\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n\nNUM_TRAINING_IMAGES = 12753\nNUM_TEST_IMAGES = 7382\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data pipeline\n\nAs mentioned before the data for this challenge is kept in tfrecords rather than the usual csv or json files that many kaggle datasets are kept in. Luckily the tensorflow data api can easily read in the data from a tfrecord and setup a pipeline to feed the images into the model (for training, validation or testing). I'll start this pipeline then by reading in the tfrecords.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = tf.data.TFRecordDataset(\n    tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-' + str(IMAGE_SIZE) + 'x' + str(IMAGE_SIZE) + '/train/*.tfrec'),\n    num_parallel_reads = tf.data.experimental.AUTOTUNE\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As this is the training data pipeline it doesn't matter what order the images are inputted to the model. This is handy as ignoring any sort of order for a batch of images is easier and faster for the TPU to handle. The below config for the pipeline makes sure that the pipeline ignores any sort of order for the images, speeding up the training time.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# disable order and increase speed\nignore_order = tf.data.Options()\nignore_order.experimental_deterministic = False \ntrain_data = train_data.with_options(ignore_order)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next the images and labels need extracting from each tfrecord. A couple of helper functions are needed here to read in a tfrecord, extract the image and label from it and decode the jpeg image into a 3D numpy array of float32 data type (though technically it is a tensor data type which contains a numpy array).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_labeled_tfrecord(example):\n    tfrec_format = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"class\": tf.io.FixedLenFeature([], tf.int64), \n    }\n    \n    example = tf.io.parse_single_example(example, tfrec_format)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    \n    # returns a dataset of (image, label) pairs\n    return image, label \n\n\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  \n    image = tf.reshape(image, [IMAGE_SIZE, IMAGE_SIZE, 3])\n    \n    return image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then add those helper functions to the pipeline.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# logic to read a tfrecord, decode the image in the record and return as arrays\ntrain_data = train_data.map(read_labeled_tfrecord)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If an image classification model is to be trained well it needs to see a wide variety of the subject matter. Usually the training dataset alone isn't enough. To help with overfitting augmentation can be used to edit the images on the fly as they are fed into the model. This enables the model to see flowers from different angles and in different lighting.\n\nThe Tensorflow data api provides some handy functions to do this. They are limited in functionality compared to say the keras data generator or the albumentations library but for now, they will do.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def augment(image, label):\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n    \n    image = tf.image.random_brightness(image, max_delta=0.5)\n    image = tf.image.random_saturation(image, lower=0.2, upper=0.5)\n    \n    image = tf.image.random_crop(image, size=[IMAGE_SIZE, IMAGE_SIZE, 3])\n    image = tf.image.resize_with_crop_or_pad(image, IMAGE_SIZE, IMAGE_SIZE)\n    \n    return image, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = train_data.map(augment, num_parallel_calls=tf.data.experimental.AUTOTUNE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally add some config to the pipeline to help with training. Use repeat to ensure the pipeline goes back to the start of the dataset after it has finished one epoch of training. Shuffle ensures that the model learns the patterns in the images rather than just memorizing the order that the images come in while batch determines the size of a batch of images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = train_data.repeat()\ntrain_data = train_data.shuffle(2048)\ntrain_data = train_data.batch(BATCH_SIZE)\ntrain_data = train_data.prefetch(tf.data.experimental.AUTOTUNE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To see what the pipeline is inputting into the model let's have a look at the first five images in the first batch.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 5, figsize=(15, 5))\n\nfor images, labels in train_data.take(1):\n    for i in range(5):\n        axes[i].set_title('Label: {0}'.format(labels[i]))\n        axes[i].imshow(images[i])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And quickly put together a validation pipeline. Luckily kaggle has pre-split the dataset so it simply a case of adjusting the path to the images and removing anything that changes the order of the dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"val_data = tf.data.TFRecordDataset(\n    tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-' + str(IMAGE_SIZE) + 'x' + str(IMAGE_SIZE) + '/val/*.tfrec'),\n    num_parallel_reads = tf.data.experimental.AUTOTUNE\n)\n\nval_data = val_data.with_options(ignore_order)\n\nval_data = val_data.map(read_labeled_tfrecord, num_parallel_calls = tf.data.experimental.AUTOTUNE)\nval_data = val_data.batch(BATCH_SIZE)\nval_data = val_data.cache()\nval_data = val_data.prefetch(tf.data.experimental.AUTOTUNE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define model\n\nTo get a good start with the training I have loaded the EfficientNetB7 weights. It looks like Tensorflow will be making this available in their keras api in the next minor version. Until then I have loaded the weights through a pip library. The only addition to it is a final dense layer to prepare EfficientNets output for flower classification.\n\nI've also brought back that TPU strategy that was defined at the beginning of this notebook. By wrapping the model in the with statement I am asking Tensorflow to use the strategy to train the model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():    \n    enet = efn.EfficientNetB7(\n        input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3),\n        weights='imagenet',\n        include_top=False\n    )\n    \n    enet.trainable = True\n    \n    model = tf.keras.Sequential([\n        enet,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(104, activation='softmax', dtype='float32')\n    ])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Nothing fancy here. Compile the model with the adam optimiser and use the usual classification loss function (categorical crossentropy). Collect accuracy metrics as well to help evaluate the quality of the model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(\n    optimizer='adam',\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy']\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I've added callbacks to help the model when loss is struggling to trend downwards. The first reduces the learning rate as loss begins to plateau while the second early stops training if the model is making no more progress. This final one restores the model to the epoch where it performed best to ensure the best model is used for inference.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"callbacks = [\n    tf.keras.callbacks.ReduceLROnPlateau(monitor='loss', patience=2, verbose=1),\n    tf.keras.callbacks.EarlyStopping(monitor='loss', patience=5, verbose=1, restore_best_weights=True),\n]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, train the model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(\n    train_data, \n    validation_data = val_data,\n    steps_per_epoch = STEPS_PER_EPOCH, \n    epochs = EPOCHS,\n    callbacks = callbacks,\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluation\n\nLet's see how the model did.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 2, figsize=(15, 5))\n\naxes[0].set_title('Loss')\naxes[0].plot(history.history['loss'], label='Train')\naxes[0].plot(history.history['val_loss'], label='Validation')\naxes[0].legend()\n\naxes[1].set_title('Accuracy')\naxes[1].plot(history.history['sparse_categorical_accuracy'], label='Train')\naxes[1].plot(history.history['val_sparse_categorical_accuracy'], label='Validation')\naxes[1].legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission\n\nWith the model trained the final thing to do is to make predictions against the test set and submit the results. First a test pipeline will need to be defined. This is similar to the validation pipeline except that this time we won't need to load a label for each image. This adjusted helper function does this for us.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_unlabeled_tfrecord(example):\n    tfrec_format = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"id\": tf.io.FixedLenFeature([], tf.string),  \n    }\n    \n    example = tf.io.parse_single_example(example, tfrec_format)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    \n    return image, idnum","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data = tf.data.TFRecordDataset(\n    tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-' + str(IMAGE_SIZE) + 'x' + str(IMAGE_SIZE) + '/test/*.tfrec'),\n    num_parallel_reads = tf.data.experimental.AUTOTUNE\n)\n\ntest_data = test_data.with_options(tf.data.Options())\ntest_data = test_data.map(read_unlabeled_tfrecord, num_parallel_calls = tf.data.experimental.AUTOTUNE)\ntest_data = test_data.batch(BATCH_SIZE)\ntest_data = test_data.prefetch(tf.data.experimental.AUTOTUNE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now use the pipeline to make predictions against each image. The model outputs a probability for every possible class per image so argmax is used to get the most probable.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_images = test_data.map(lambda image, idnum: image)\n\nprobabilities = model.predict(test_images)\npredictions = np.argmax(probabilities, axis=-1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The starter notebooks for this competition had an alternative way of getting hold of the ids per image. I got some funny errors though that disconnected me from the bucket the data was in. As such I have used this alternative.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ids = []\n\nfor image, image_ids in test_data.take(NUM_TEST_IMAGES):\n    ids.append(image_ids.numpy())\n\nids = np.concatenate(ids, axis=None).astype(str)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Write the predictions and image ids to a file ready for submission.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame(data={'id': ids, 'label': predictions})\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And save the model in case it's needed in another notebook.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save('model.h5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}