{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Transfer Learning with TPU","metadata":{}},{"cell_type":"markdown","source":"## Classify Flowers on TPU","metadata":{}},{"cell_type":"markdown","source":"<div align=\"center\">\n    <img src=\"https://steemitimages.com/p/LcTxR7u1XKaa3e4T1EBuBP18JezPvjFFo8gNuE9CiKHBn31rR9T4NmjUUsoAbL1yPkCkAaa9b8ZRqtVUW1YjNvoeQvXGe7A5hyT7XJsyNYcAMhFFnV6LTx2ymSi954PiEQTnJsTzAhCaFQXfzHcPMjnWq?format=match&mode=fit\" alt=\"Spirited Away Flower Garden\" style=\"width: 700px;\"> \n</div>\n<div align=\"center\">\n  © Spirited Away (2001 film)\n</div>","metadata":{}},{"cell_type":"markdown","source":"First time using a TPU and don't know where to start?\n\nThis notebook will take you through the basics of using a **TPU** accelerator and handling TFRecord files. We'll also understand how **transfer learning** works using the ResNet152 V2. And if that's not enough, we'll take things up a notch by implementing **data augmentation** and **fine tuning** our pre-trained model. \n\nLet's get started!","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"### <font color=''>Table of contents<font><a class='anchor' id='top'></a>\n\n1. [Introduction](#section-one)  \n    \n2. [Get Data](#section-two)\n    \n3. [Visualize Data](#section-three)\n    \n4. [Prepare Data (Optional)](#section-four)\n    \n5. [Build Model](#section-five)\n    \n6. [Train Model](#section-six) \n\n7. [Fine Tune Model (Optional)](#section-seven)\n    \n8. [Test Model](#secion-eight)\n    \n9. [Conclusion](#section-nine)","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-one\"></a>\n## 1. Introduction\n\nThis notebook is a great starting point if you want to learn about **Tensor Processing Units** (TPUs). We'll also look at how to load, read and display iamges from **TFRecord** files. Finally, we'll use a fantastic model called the **Residual Network** (ResNet152) to classify different types of flowers.\n\n**How to use this notebook**: If you're new here or you simply want to jump to the results, I recommend skipping the two optional sections. If you're looking to gain a deeper understanding of how the code works, just look for the guides that I wrote. In any case, feel free to play around with the different parameters and settings.","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-two\"></a>\n## 2. Get Data\n\n* Use try-except block to detect TPU accelerator. If TPU is present, use the accelerator its distribution strategy.\n* Get the Google Cloud Storage (GCS) path to retrieve the files.\n* Load the TFRecord files and parse the images inside. Two functions are needed for both labeled and unlabeled data.","metadata":{}},{"cell_type":"code","source":"# Import libraries\n\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nimport tensorflow as tf\nfrom tensorflow.data.experimental import AUTOTUNE\nfrom tensorflow.keras import Model\nfrom tensorflow.keras.applications import ResNet152V2\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Input, Dense, GlobalAveragePooling2D\nfrom tensorflow.keras.layers import BatchNormalization, Dropout\nfrom tensorflow.keras.optimizers import Adam\n\nfrom kaggle_datasets import KaggleDatasets","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:55:57.310836Z","iopub.execute_input":"2023-02-18T04:55:57.311546Z","iopub.status.idle":"2023-02-18T04:55:59.256933Z","shell.execute_reply.started":"2023-02-18T04:55:57.311464Z","shell.execute_reply":"2023-02-18T04:55:59.256032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### A Guide to TPUs:\n\nIn order to detect the TPU accelerator, we need to use a try-except block. It goes like this: try to find TPU cluster to connect to, except when a ValueError occurs, then that means no TPU was found.\n\nIf TPU is found in the previous step, we connect the local Tensorflow instance to the TPU cluster. Then, we **initialize** the TPU and create a **distributed training strategy**. If TPU was not found, simply use the training strategy on CPU or GPU.","metadata":{}},{"cell_type":"code","source":"# Detect TPU, return appropriate distribution strategy\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:55:59.257939Z","iopub.execute_input":"2023-02-18T04:55:59.258134Z","iopub.status.idle":"2023-02-18T04:56:04.781226Z","shell.execute_reply.started":"2023-02-18T04:55:59.258111Z","shell.execute_reply":"2023-02-18T04:56:04.780656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get GCS path and select the file with 224x224 images\n\ngcs_ds_path = KaggleDatasets().get_gcs_path('tpu-getting-started')\ngcs_path = gcs_ds_path + '/tfrecords-jpeg-224x224'","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:56:04.782061Z","iopub.execute_input":"2023-02-18T04:56:04.782269Z","iopub.status.idle":"2023-02-18T04:56:05.188204Z","shell.execute_reply.started":"2023-02-18T04:56:04.782244Z","shell.execute_reply":"2023-02-18T04:56:05.187216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Set parameters\n\nBUFFER_SIZE = 60000\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nIMAGE_SIZE = [224, 224]\nHEIGHT = 224\nWIDTH = 224\n\nNUM_TRAINING_IMAGES = 12753\nNUM_TEST_IMAGES = 7382\nEPOCHS = 10\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:56:05.192244Z","iopub.execute_input":"2023-02-18T04:56:05.192476Z","iopub.status.idle":"2023-02-18T04:56:05.197531Z","shell.execute_reply.started":"2023-02-18T04:56:05.192450Z","shell.execute_reply":"2023-02-18T04:56:05.196637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### A Guide to TFRecords:\n\nTFRecords are simply a more efficient way of **storing** and **transferring** data. Each TFRecord file contains multiple data samples (called examples). Each example stores values for certain features.\n\nLet's look at the code below. Since the TFRecord files are stored in a folder, we need to use the **glob module** to access each individual file. \n\nNext, TFRecordDataset is used to load these files as bytes. Two dictionaries are created to describe the features for **labeled** and **unlabeled** datasets. We'll create a classes list with all the flower names.\n\nAfter that, we'll create two functions that will **parse** a single example from a TFRecord file and extract the data contained inside it. We will also **decode** and **resize** the images. Finally, we'll parse the labeled images from the training and validation datasets and parse the unlabeled images from the test dataset.","metadata":{}},{"cell_type":"code","source":"# Get the path to all the files within the tfrecords-jpeg-224x224 folder\n\ntraining_filepath = tf.io.gfile.glob(gcs_path + '/train/*.tfrec')\nvalidation_filepath = tf.io.gfile.glob(gcs_path + '/val/*.tfrec')\ntest_filepath = tf.io.gfile.glob(gcs_path + '/test/*.tfrec') ","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:56:05.198760Z","iopub.execute_input":"2023-02-18T04:56:05.199075Z","iopub.status.idle":"2023-02-18T04:56:05.423986Z","shell.execute_reply.started":"2023-02-18T04:56:05.199039Z","shell.execute_reply":"2023-02-18T04:56:05.423177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load TFRecord file from the folder as bytes\n\nraw_training_dataset = tf.data.TFRecordDataset(training_filepath)\nraw_validation_dataset = tf.data.TFRecordDataset(validation_filepath)\nraw_test_dataset = tf.data.TFRecordDataset(test_filepath)","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:56:05.424984Z","iopub.execute_input":"2023-02-18T04:56:05.425189Z","iopub.status.idle":"2023-02-18T04:56:05.451830Z","shell.execute_reply.started":"2023-02-18T04:56:05.425165Z","shell.execute_reply":"2023-02-18T04:56:05.451174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a dictionary describing the features\n\nlabeled_feature_description = {\n    'class': tf.io.FixedLenFeature([], tf.int64),\n    'image': tf.io.FixedLenFeature([], tf.string)\n}\n\nunlabeled_feature_description = {\n    'id': tf.io.FixedLenFeature([], tf.string),\n    'image': tf.io.FixedLenFeature([], tf.string)\n}","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:56:05.452661Z","iopub.execute_input":"2023-02-18T04:56:05.452868Z","iopub.status.idle":"2023-02-18T04:56:05.460033Z","shell.execute_reply.started":"2023-02-18T04:56:05.452843Z","shell.execute_reply":"2023-02-18T04:56:05.459010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Class name of flowers\n\nCLASSES = [\n    'pink primrose',        'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',      'wild geranium',         # 00-04\n    'tiger lily',           'moon orchid',               'bird of paradise', 'monkshood',      'globe thistle',         # 05-09\n    'snapdragon',           \"colt's foot\",               'king protea',      'spear thistle',  'yellow iris',           # 10-14\n    'globe-flower',         'purple coneflower',         'peruvian lily',    'balloon flower', 'giant white arum lily', # 15-19\n    'fire lily',            'pincushion flower',         'fritillary',       'red ginger',     'grape hyacinth',        # 20-24\n    'corn poppy',           'prince of wales feathers',  'stemless gentian', 'artichoke',      'sweet william',         # 25-29\n    'carnation',            'garden phlox',              'love in the mist', 'cosmos',         'alpine sea holly',      # 30-34\n    'ruby-lipped cattleya', 'cape flower',               'great masterwort', 'siam tulip',     'lenten rose',           # 35-39\n    'barberton daisy',      'daffodil',                  'sword lily',       'poinsettia',     'bolero deep blue',      # 40-44\n    'wallflower',           'marigold',                  'buttercup',        'daisy',          'common dandelion',      # 45-49\n    'petunia',              'wild pansy',                'primula',          'sunflower',      'lilac hibiscus',        # 50-54\n    'bishop of llandaff',   'gaura',                     'geranium',         'orange dahlia',  'pink-yellow dahlia',    # 55-59\n    'cautleya spicata',     'japanese anemone',          'black-eyed susan', 'silverbush',     'californian poppy',     # 60-64\n    'osteospermum',         'spring crocus',             'iris',             'windflower',     'tree poppy',            # 65-69\n    'gazania',              'azalea',                    'water lily',       'rose',           'thorn apple',           # 70-74\n    'morning glory',        'passion flower',            'lotus',            'toad lily',      'anthurium',             # 75-79\n    'frangipani',           'clematis',                  'hibiscus',         'columbine',      'desert-rose',           # 80-84\n    'tree mallow',          'magnolia',                  'cyclamen ',        'watercress',     'canna lily',            # 85-89\n    'hippeastrum ',         'bee balm',                  'pink quill',       'foxglove',       'bougainvillea',         # 90-94\n    'camellia',             'mallow',                    'mexican petunia',  'bromelia',       'blanket flower',        # 95-99\n    'trumpet creeper',      'blackberry lily',           'common tulip',     'wild rose'                                #100-103\n]","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:56:05.461711Z","iopub.execute_input":"2023-02-18T04:56:05.462052Z","iopub.status.idle":"2023-02-18T04:56:05.471785Z","shell.execute_reply.started":"2023-02-18T04:56:05.462012Z","shell.execute_reply":"2023-02-18T04:56:05.470823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a function to read and extract images from dataset\n\ndef _parse_labeled_image_function(example_proto):\n    example = tf.io.parse_single_example(example_proto, labeled_feature_description)\n    image = tf.io.decode_jpeg(example['image'])\n    image = tf.cast(image, tf.float32) / 255.\n    image = tf.image.resize(image, IMAGE_SIZE)\n    label = tf.cast(example['class'], tf.int32)\n    return image, label\n\ndef _parse_unlabeled_image_function(example_proto):\n    example = tf.io.parse_single_example(example_proto, unlabeled_feature_description)\n    image = tf.io.decode_jpeg(example['image'])\n    image = tf.cast(image, tf.float32) / 255.\n    image = tf.image.resize(image, IMAGE_SIZE)\n    idnum = example['id']\n    return image, idnum","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:56:05.473190Z","iopub.execute_input":"2023-02-18T04:56:05.473493Z","iopub.status.idle":"2023-02-18T04:56:05.486080Z","shell.execute_reply.started":"2023-02-18T04:56:05.473454Z","shell.execute_reply":"2023-02-18T04:56:05.485026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Parse and extract images\n\n# Parse labeled images, shuffle and batch\ntraining_dataset = (\n    raw_training_dataset\n    .map(_parse_labeled_image_function)\n    .repeat()\n    .shuffle(BUFFER_SIZE)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTOTUNE)\n)\n\n# Parse unlabeled images and batch\nvalidation_dataset = (\n    raw_validation_dataset\n    .map(_parse_labeled_image_function)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTOTUNE)\n)\n\n# Parse unlabeled images and batch\ntest_dataset = (\n    raw_test_dataset\n    .map(_parse_unlabeled_image_function)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTOTUNE)\n)","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:56:05.487797Z","iopub.execute_input":"2023-02-18T04:56:05.488151Z","iopub.status.idle":"2023-02-18T04:56:05.683907Z","shell.execute_reply.started":"2023-02-18T04:56:05.488110Z","shell.execute_reply":"2023-02-18T04:56:05.683048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-three\"></a>\n## 3. Visualize Data\n\n* Create a function called display_images and plot the images in a 5x5 grid.","metadata":{}},{"cell_type":"code","source":"# Display images in a 5x5 grid\n\nimage_batch, label_batch = next(iter(training_dataset))\n\ndef display_images(image_batch, label_batch):\n    plt.figure(figsize = [20,12])\n    for i in range(25):\n        plt.subplot(5,5,i+1)\n        plt.imshow(image_batch[i])\n        plt.title(CLASSES[label_batch[i].numpy()])\n        plt.axis('off')\n    plt.show()\n\ndisplay_images(image_batch, label_batch)","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:56:05.685260Z","iopub.execute_input":"2023-02-18T04:56:05.685579Z","iopub.status.idle":"2023-02-18T04:57:19.524530Z","shell.execute_reply.started":"2023-02-18T04:56:05.685539Z","shell.execute_reply":"2023-02-18T04:57:19.523641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-four\"></a>\n## 4. Prepare Data (Optional)\n\n* Perform data augmentation using tf.image to crop, flip and adjust the brightness, contrast and saturation.\n* Use the display_images function above to plot images in a 5x5 grid.","metadata":{}},{"cell_type":"markdown","source":"### A Guide to Image Augmentation:\n\nThis section is entirely optional. Generally, data augmentation does lead to an increase in  the model's performance.\n\nWe'll start by creating a function called augment_image. We'll use tf.image to apply **random transformations** to the training set. There are 5 transformations we'll make: **crop, flip, brightness, contrast and saturation**. \n\nFor all of these tf.image functions, you simply need to call the function and apply it to the image. The crop function needs an extra step: you need to resize the image before you can crop it. In the example below, we add 10px to the height and width. \n\nAs for the other tf.image functions, the numbers after the image variable are basically adjustment values. We're increasing brightness by 20% and adjusting contrast and saturation between a range of 80% and 120%.","metadata":{}},{"cell_type":"code","source":"# Create a function to augment brightness, contrast, flip and crop images\n\ndef augment_image(image, label):\n    \n    # Add 10px padding and random crop\n    image = tf.image.resize_with_crop_or_pad(image, HEIGHT+10, WIDTH+10)\n    image = tf.image.random_crop(image, size=[*IMAGE_SIZE, 3])\n    \n    # Random flip\n    image = tf.image.random_flip_left_right(image)\n    \n    # Random brightness\n    image = tf.image.random_brightness(image, 0.2)\n    \n    # Random contrast \n    image = tf.image.random_contrast(image, lower=0.8, upper=1.2)\n    \n    # Random saturation\n    image = tf.image.random_saturation(image, lower=0.8, upper=1.2)\n    \n    return image, label","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:57:19.526014Z","iopub.execute_input":"2023-02-18T04:57:19.526458Z","iopub.status.idle":"2023-02-18T04:57:19.532507Z","shell.execute_reply.started":"2023-02-18T04:57:19.526426Z","shell.execute_reply":"2023-02-18T04:57:19.531749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Parse unlabeled images, augment, shuffle and batch\n\ntraining_dataset_augmented = (\n    raw_training_dataset\n    .map(_parse_labeled_image_function)\n    .map(augment_image)\n    .repeat()\n    .shuffle(BUFFER_SIZE)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTOTUNE)\n)","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:57:19.536314Z","iopub.execute_input":"2023-02-18T04:57:19.536820Z","iopub.status.idle":"2023-02-18T04:57:19.748662Z","shell.execute_reply.started":"2023-02-18T04:57:19.536787Z","shell.execute_reply":"2023-02-18T04:57:19.747710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display images in a 5x5 grid\n\nimage_batch_augmented, label_batch_augmented = next(iter(training_dataset_augmented))\n\ndisplay_images(image_batch_augmented, label_batch_augmented)","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:57:19.749840Z","iopub.execute_input":"2023-02-18T04:57:19.750098Z","iopub.status.idle":"2023-02-18T04:59:42.451818Z","shell.execute_reply.started":"2023-02-18T04:57:19.750070Z","shell.execute_reply":"2023-02-18T04:59:42.451040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-five\"></a>\n## 5. Build Model\n\n* Create a function called build_model that uses the ResNet152V2 architecture as the base.\n* Freeze the model and add GlobalAveragePooling2D, BatchNormalization, Dropout and Dense layers on top of the base model.\n* Compile the new model using the Adam optimizer and sparse categorical cross-entropy loss.","metadata":{}},{"cell_type":"markdown","source":"### A Guide to Transfer Learning:\n\nTransfer learning works by taking what a model has learnt on one problem and applying it to another, similar problem. Thankfully, Keras makes it easy for us to use the models that other people created. These models are called **pre-trained models**.\n\nTransfer Learning can be completed in 4 simple steps. First, we need to take the layers from a pretrained model. In the example below, we used the **ResNet152 V2** with pre-trained weights from ImageNet. Also, we did not include the classifier on top. Step 2 is to **freeze the model**. We can simply set trainable to false.\n\nThe third step involves adding a few new layers on top. We added a **global average pooling** layer, followed by **batch normalization** and then a **dropout layer**. Right on top of the model, we added a **fully connected** (dense) layer with 104 units (since there are 104 classes of flowers). A softmax activation function is also used here.\n\nLastly, we need to **compile and train** the new model. We used the adam optimizer and the sparse categorical cross-entropy loss. And that's it! In the next section, we'll proceed to train the model.","metadata":{}},{"cell_type":"code","source":"# Create a function to build the model\n\ndef build_model():\n    inputs = Input(shape=(HEIGHT, WIDTH, 3))\n    model = ResNet152V2(include_top=False, input_tensor=inputs, weights=\"imagenet\")\n\n    # Freeze the pretrained weights\n    model.trainable = False\n\n    # Rebuild top\n    x = GlobalAveragePooling2D()(model.output)\n    x = BatchNormalization()(x)\n    x = Dropout(0.2)(x)\n    outputs = Dense(104, activation=\"softmax\")(x)\n\n    # Compile\n    model = Model(inputs, outputs)\n    model.compile(optimizer=Adam(learning_rate=1e-2), \n                  loss=\"sparse_categorical_crossentropy\", \n                  metrics=[\"sparse_categorical_accuracy\"])\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:59:42.453253Z","iopub.execute_input":"2023-02-18T04:59:42.453538Z","iopub.status.idle":"2023-02-18T04:59:42.461664Z","shell.execute_reply.started":"2023-02-18T04:59:42.453506Z","shell.execute_reply":"2023-02-18T04:59:42.460640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-six\"></a>\n## 6. Train Model\n\n* Call the build_model function that was created in the previous section.\n* Train the model on the augmented training dataset and test using the validation dataset.","metadata":{}},{"cell_type":"code","source":"# Train the model\n\nwith strategy.scope():\n    model = build_model()\n\nhist = model.fit(training_dataset_augmented, \n                 epochs=EPOCHS*2, \n                 validation_data=validation_dataset, \n                 steps_per_epoch=STEPS_PER_EPOCH)","metadata":{"execution":{"iopub.status.busy":"2023-02-18T04:59:42.463240Z","iopub.execute_input":"2023-02-18T04:59:42.463723Z","iopub.status.idle":"2023-02-18T05:13:14.966713Z","shell.execute_reply.started":"2023-02-18T04:59:42.463688Z","shell.execute_reply":"2023-02-18T05:13:14.965846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-seven\"></a>\n## 7. Fine Tune Model (Optional)\n\n* Create a function to unfreeze the top 20 layers of the pretrained model.\n* Compile the model using a lower learning rate and train the model.","metadata":{}},{"cell_type":"markdown","source":"### A Guide to Fine Tuning:\n\nThis section is entirely optional. It can either improve your model or result in overfitting, so you'd definitely need to experiment a'lil with this one!\n\nOnce the top layers have been trained, we can try unfreezing a part of the base model and retraining the whole model. In this case, we will **unfreeze the top 20 layers**. \n\nTake a look at the code right below. We will create a for loop for the top 20 layers and only train the layers that aren't batch normalization layers. You may be wondering why this is the case. That's because **batch normalization** layers contain **non-trainable weights** and when we train those layers, we essentially update those weights which causes the model to lose what it had learnt.\n\nAfter the for loop, we will compile the model similar to what we did above. The only difference is that we're setting a much **lower learning rate** so as to **prevent overfitting**. Finally, we'll proceed to train the top 20 layers with half the number of epochs compared to the training step above.","metadata":{}},{"cell_type":"code","source":"# Create a function to unfreeze the model the top 20 layers\n# But, we'll keep BatchNormalization layers frozen\n\ndef unfreeze_model(model):\n    for layer in model.layers[-20:]:\n        if not isinstance(layer, BatchNormalization):\n            layer.trainable = True\n\n    model.compile(optimizer=Adam(learning_rate=1e-4), \n                  loss=\"sparse_categorical_crossentropy\", \n                  metrics=[\"sparse_categorical_accuracy\"])","metadata":{"execution":{"iopub.status.busy":"2023-02-18T05:13:14.967966Z","iopub.execute_input":"2023-02-18T05:13:14.968438Z","iopub.status.idle":"2023-02-18T05:13:14.975171Z","shell.execute_reply.started":"2023-02-18T05:13:14.968405Z","shell.execute_reply":"2023-02-18T05:13:14.974024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Unfreeze and train the model\n\nunfreeze_model(model)\n\nhist = model.fit(training_dataset_augmented, \n                 epochs=EPOCHS, \n                 validation_data=validation_dataset, \n                 steps_per_epoch=STEPS_PER_EPOCH)","metadata":{"execution":{"iopub.status.busy":"2023-02-18T05:13:14.976456Z","iopub.execute_input":"2023-02-18T05:13:14.976790Z","iopub.status.idle":"2023-02-18T05:21:10.014630Z","shell.execute_reply.started":"2023-02-18T05:13:14.976761Z","shell.execute_reply":"2023-02-18T05:21:10.013730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-eight\"></a>\n## 8. Test Model\n\n* Test the model on the test dataset and output the predictions.\n* Compile all the predictions together in a file and save it for submission.","metadata":{}},{"cell_type":"code","source":"# Predict images from test set\n\ntest_images = test_dataset.map(lambda image, idnum: image)\nprob = model.predict(test_images)\npred = np.argmax(prob, axis=-1)\nprint(pred)","metadata":{"execution":{"iopub.status.busy":"2023-02-18T05:21:10.016667Z","iopub.execute_input":"2023-02-18T05:21:10.016903Z","iopub.status.idle":"2023-02-18T05:21:40.498506Z","shell.execute_reply.started":"2023-02-18T05:21:10.016876Z","shell.execute_reply":"2023-02-18T05:21:40.497540Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Prepare file for submission\n\ntest_ids_ds = test_dataset.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n\nnp.savetxt(\n    'submission.csv',\n    np.rec.fromarrays([test_ids, pred]),\n    fmt=['%s', '%d'],\n    delimiter=',',\n    header='id,label',\n    comments='',\n)","metadata":{"execution":{"iopub.status.busy":"2023-02-18T05:21:40.499636Z","iopub.execute_input":"2023-02-18T05:21:40.499887Z","iopub.status.idle":"2023-02-18T05:21:47.962286Z","shell.execute_reply.started":"2023-02-18T05:21:40.499860Z","shell.execute_reply":"2023-02-18T05:21:47.961413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-nine\"></a>\n## 9. Conclusion\n\nIf you made it all the way here, congrats! \n\nIn this notebook, we learnt how to use TPUs to **speed up the training** of machine learning models. Next, we looked at how to load and parse images from **TFRecord** files. Finally, we used the pre-trained **ResNet152 V2** model to classify our images, while also tweaking the learning rate and trainable layers.\n\nIf you enjoyed this notebook, please consider giving it an upvote. That way, others can discover it too! \n\nCheers :)","metadata":{}},{"cell_type":"markdown","source":"### References:\n\n* [Use TPUs](https://www.tensorflow.org/guide/tpu)\n* [Data Augmentation](https://www.tensorflow.org/tutorials/images/data_augmentation)\n* [Image classification via fine-tuning with EfficientNet](https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/#:~:text=Keras%20implementation%20of%20EfficientNet&text=This%20model%20takes%20input%20images,as%20part%20of%20the%20model.)\n* [Transfer Learning and Fine Tuning](https://www.tensorflow.org/guide/keras/transfer_learning)\n* [Computer Vision - Petals to the Metal](https://www.kaggle.com/code/georgezoto/computer-vision-petals-to-the-metal)","metadata":{}}]}