{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n# Introduction #\n\nWelcome to the [**Petals to the Metal**](https://www.kaggle.com/c/tpu-getting-started) competition! In this competition, you’re challenged to build a machine learning model to classify 104 types of flowers based on their images.\n\nIn this tutorial notebook, you'll learn how to build an image classifier in Keras and train it on a [Tensor Processing Unit (TPU)](https://www.kaggle.com/docs/tpu). At the end, you'll have a complete project you can build off of with ideas of your own.\n\n
\n Fork This Notebook!
\nCreate your own editable copy of this notebook by clicking on the Copy and Edit button in the top right corner.\n
","metadata":{}},{"cell_type":"markdown","source":"# Step 1: Imports #\n\nWe begin by importing several Python packages.","metadata":{}},{"cell_type":"code","source":"import gc # garbage collection, to free up some memory\nimport math, re, os\nimport numpy as np\nimport tensorflow as tf\n\nfrom matplotlib import pyplot as plt\n\n# check the package version\nprint('Tensorflow version ' + tf.__version__)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 2: Distribution Strategy #\n\nA TPU has eight different *cores* and each of these cores acts as its own accelerator. (A TPU is sort of like having eight GPUs in one machine.) We tell TensorFlow how to make use of all these cores at once through a **distribution strategy**. Run the following cell to create the distribution strategy that we'll later apply to our model.","metadata":{}},{"cell_type":"code","source":"# detect TPU, return appropriate distribution strategy\ntry:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n print('Running on TPU ', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n strategy = tf.distribute.get_strategy() \n\nprint('REPLICAS: ', strategy.num_replicas_in_sync)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll use the distribution strategy when we create our neural network model. Then, TensorFlow will distribute the training among the eight TPU cores by creating eight different *replicas* of the model, one for each core.\n\n# Step 3: Loading the Competition Data #\n\n## Get GCS Path ##\n\nWhen used with TPUs, datasets need to be stored in a [Google Cloud Storage bucket](https://cloud.google.com/storage/). You can use data from any public GCS bucket by giving its path just like you would data from `'/kaggle/input'`. The following will retrieve the GCS path for this competition's dataset.","metadata":{}},{"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\n\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('tpu-getting-started')\n\n# what do gcs paths look like?\nprint(GCS_DS_PATH)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can use data from any public dataset here on Kaggle in just the same way. If you'd like to use data from one of your private datasets, see [here](https://www.kaggle.com/docs/tpu#tpu3pt5).\n\n## Load Data ##\n\nWhen used with TPUs, datasets are often serialized into [TFRecords](https://www.kaggle.com/ryanholbrook/tfrecords-basics). This is a format convenient for distributing data to each of the TPUs cores. We've hidden the cell that reads the TFRecords for our dataset since the process is a bit long. You could come back to it later for some guidance on using your own datasets with TPUs.","metadata":{}},{"cell_type":"code","source":"IMAGE_SIZE = [224, 224]\nGCS_PATH = GCS_DS_PATH + '/tfrecords-jpeg-224x224'\nAUTO = tf.data.experimental.AUTOTUNE\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') \n\nCLASSES = ['pink primrose', 'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea', 'wild geranium', 'tiger lily', 'moon orchid', 'bird of paradise', 'monkshood', 'globe thistle', # 00 - 09\n 'snapdragon', \"colt's foot\", 'king protea', 'spear thistle', 'yellow iris', 'globe-flower', 'purple coneflower', 'peruvian lily', 'balloon flower', 'giant white arum lily', # 10 - 19\n 'fire lily', 'pincushion flower', 'fritillary', 'red ginger', 'grape hyacinth', 'corn poppy', 'prince of wales feathers', 'stemless gentian', 'artichoke', 'sweet william', # 20 - 29\n 'carnation', 'garden phlox', 'love in the mist', 'cosmos', 'alpine sea holly', 'ruby-lipped cattleya', 'cape flower', 'great masterwort', 'siam tulip', 'lenten rose', # 30 - 39\n 'barberton daisy', 'daffodil', 'sword lily', 'poinsettia', 'bolero deep blue', 'wallflower', 'marigold', 'buttercup', 'daisy', 'common dandelion', # 40 - 49\n 'petunia', 'wild pansy', 'primula', 'sunflower', 'lilac hibiscus', 'bishop of llandaff', 'gaura', 'geranium', 'orange dahlia', 'pink-yellow dahlia', # 50 - 59\n 'cautleya spicata', 'japanese anemone', 'black-eyed susan', 'silverbush', 'californian poppy', 'osteospermum', 'spring crocus', 'iris', 'windflower', 'tree poppy', # 60 - 69\n 'gazania', 'azalea', 'water lily', 'rose', 'thorn apple', 'morning glory', 'passion flower', 'lotus', 'toad lily', 'anthurium', # 70 - 79\n 'frangipani', 'clematis', 'hibiscus', 'columbine', 'desert-rose', 'tree mallow', 'magnolia', 'cyclamen ', 'watercress', 'canna lily', # 80 - 89\n 'hippeastrum ', 'bee balm', 'pink quill', 'foxglove', 'bougainvillea', 'camellia', 'mallow', 'mexican petunia', 'bromelia', 'blanket flower', # 90 - 99\n 'trumpet creeper', 'blackberry lily', 'common tulip', 'wild rose'] # 100 - 102\n\n\ndef decode_image(image_data):\n image = tf.image.decode_jpeg(image_data, channels=3)\n image = tf.cast(image, tf.float32) / 255.0 # convert image to floats in [0, 1] range\n image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n return image\n\ndef read_labeled_tfrecord(example):\n LABELED_TFREC_FORMAT = {\n \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n \"class\": tf.io.FixedLenFeature([], tf.int64), # shape [] means single element\n }\n example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n image = decode_image(example['image'])\n label = tf.cast(example['class'], tf.int32)\n return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n UNLABELED_TFREC_FORMAT = {\n \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n \"id\": tf.io.FixedLenFeature([], tf.string), # shape [] means single element\n # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n }\n example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n image = decode_image(example['image'])\n idnum = example['id']\n return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n # Read from TFRecords. For optimal performance, reading from multiple files at once and\n # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n ignore_order = tf.data.Options()\n if not ordered:\n ignore_order.experimental_deterministic = False # disable order, increase speed\n\n dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n return dataset","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create Data Pipelines ##\n\nIn this final step we'll use the `tf.data` API to define an efficient data pipeline for each of the training, validation, and test splits.","metadata":{}},{"cell_type":"code","source":"# add data augmentation sequence\ndata_augmentation = tf.keras.Sequential(\n [\n# tf.keras.layers.Resizing(224, 224), # resize the image to match base model input requirement\n tf.keras.layers.Rescaling(255), # rescale back the pixel value to 255, required by the EfficientNet input\n tf.keras.layers.RandomFlip('horizontal_and_vertical'), # flip the image\n tf.keras.layers.RandomRotation(0.25, fill_mode='reflect'), # rotation, factor of 2*pi\n tf.keras.layers.RandomZoom(0.1, fill_mode='reflect'), # apply random zoom in and out\n tf.keras.layers.RandomTranslation(height_factor=0.1, \n width_factor=0.1, \n fill_mode='reflect') # randomly shift the image\n ]\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_training_dataset():\n dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n dataset = dataset.repeat() # the training dataset must repeat for several epochs\n dataset = dataset.shuffle(2048)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n return dataset\n\ndef get_validation_dataset(ordered=False):\n dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTO)\n return dataset\n\ndef get_test_dataset(ordered=False):\n dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTO)\n return dataset\n\ndef count_data_items(filenames):\n # the number of data items is written in the name of the .tfrec\n # files, i.e. flowers00-230.tfrec = 230 data items\n n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n return np.sum(n)\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\n\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This next cell will create the datasets that we'll use with Keras during training and inference. Notice how we scale the size of the batches to the number of TPU cores.","metadata":{}},{"cell_type":"code","source":"# Define the batch size. This will be 16 with TPU off and 128 (=16*8) with TPU on\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n\nds_train = get_training_dataset()\nds_valid = get_validation_dataset()\nds_test = get_test_dataset()\n\nprint('Training:', ds_train)\nprint ('Validation:', ds_valid)\nprint('Test:', ds_test)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These datasets are `tf.data.Dataset` objects. You can think about a dataset in TensorFlow as a *stream* of data records. The training and validation sets are streams of `(image, label)` pairs.","metadata":{}},{"cell_type":"code","source":"print ('Training data shapes:')\nfor image, label in ds_train.take(3):\n print (image.numpy().shape, label.numpy().shape)\n\nprint ()\nprint ('Training data label examples:', label.numpy())\n\n# delete unused variables\ndel image, label","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The test set is a stream of `(image, idnum)` pairs; `idnum` here is the unique identifier given to the image that we'll use later when we make our submission as a `csv` file.","metadata":{}},{"cell_type":"code","source":"print ('Test data shapes:')\nfor image, idnum in ds_test.take(3):\n print(image.numpy().shape, idnum.numpy().shape)\n\nprint ()\nprint ('Test data IDs:', idnum.numpy().astype('U')) # U=unicode string\n\n# delete unused variables\ndel image, idnum","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 4: Explore Data #\n\nLet's take a moment to look at some of the images in the dataset.","metadata":{}},{"cell_type":"code","source":"def batch_to_numpy_images_and_labels(data):\n images, labels = data\n numpy_images = images.numpy()\n numpy_labels = labels.numpy()\n if numpy_labels.dtype == object: # binary string in this case,\n # these are image ID strings\n numpy_labels = [None for _ in enumerate(numpy_images)]\n # If no labels, only image IDs, return None for labels (this is\n # the case for test data)\n return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n if correct_label is None:\n return CLASSES[label], True\n correct = (label == correct_label)\n return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n plt.subplot(*subplot)\n plt.axis('off')\n plt.imshow(image)\n if len(title) > 0:\n plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n return (subplot[0], subplot[1], subplot[2]+1)\n \ndef display_batch_of_images(databatch, predictions=None):\n \"\"\"This will work with:\n display_batch_of_images(images)\n display_batch_of_images(images, predictions)\n display_batch_of_images((images, labels))\n display_batch_of_images((images, labels), predictions)\n \"\"\"\n # data\n images, labels = batch_to_numpy_images_and_labels(databatch)\n if labels is None:\n labels = [None for _ in enumerate(images)]\n \n # auto-squaring: this will drop data that does not fit into square\n # or square-ish rectangle\n rows = int(math.sqrt(len(images)))\n cols = len(images)//rows\n \n # size and spacing\n FIGSIZE = 13.0\n SPACING = 0.1\n subplot=(rows,cols,1)\n if rows < cols:\n plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n else:\n plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n \n # display\n for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n title = '' if label is None else CLASSES[label]\n correct = True\n if predictions is not None:\n title, correct = title_from_label_and_target(predictions[i], label)\n dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n \n #layout\n plt.tight_layout()\n if label is None and predictions is None:\n plt.subplots_adjust(wspace=0, hspace=0)\n else:\n plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n plt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can display a single batch of images from a dataset with another of our helper functions. The next cell will turn the dataset into an iterator of batches of 20 images.\n\nUse the Python `next` function to pop out the next batch in the stream and display it with the helper function.","metadata":{}},{"cell_type":"code","source":"ds_iter = iter(ds_train.unbatch().batch(20))\n\none_batch = next(ds_iter)\ndisplay_batch_of_images(one_batch)\n\n# delete unused variables\ndel ds_iter, one_batch","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By defining `ds_iter` and `one_batch` in separate cells, you only need to rerun the cell above to see a new batch of images.","metadata":{}},{"cell_type":"code","source":"# delete unused variables and apply garbage collection\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 5: Define Model #\n\nNow we're ready to create a neural network for classifying images! We'll use what's known as **transfer learning**. With transfer learning, you reuse part of a pretrained model to get a head-start on a new dataset.\n\nFor this project, we'll to use a model called **EfficientNet** pretrained on [ImageNet](http://image-net.org/)). Later, you might want to experiment with [other models](https://www.tensorflow.org/api_docs/python/tf/keras/applications) included with Keras.\n\nThe distribution strategy we created earlier contains a [context manager](https://docs.python.org/3/reference/compound_stmts.html#with), `strategy.scope`. This context manager tells TensorFlow how to divide the work of training among the eight TPU cores. When using TensorFlow with a TPU, it's important to define your model in a `strategy.scope()` context.","metadata":{}},{"cell_type":"code","source":"# EfficientNet model\ndef make_model(num_classes):\n\n from tensorflow.keras.applications import EfficientNetB4\n\n # input layer\n inputA = tf.keras.Input(shape=(224, 224, 3))\n\n # image augmentation block\n x = data_augmentation(inputA)\n\n # Use EfficientNet as convolutional blocks.\n # This pre-trained model takes input images of shape (224, 224, 3), and the input data should range [0, 255].\n # Normalization is included as part of the model.\n base_modelA = EfficientNetB4(include_top=False, # <- do not include the ImageNet classifier at the top.\n weights='imagenet', \n drop_connect_rate=0.4,\n classifier_activation=None,\n input_shape=(224, 224, 3),\n pooling='avg')\n\n # freeze the base_model\n base_modelA.trainable = False\n\n # When you unfreeze a model that contains BatchNormalization layers in order to do fine-tuning later, \n # you should keep the BatchNormalization layers in inference mode by passing training = False when calling the base model. \n # Otherwise, the updates applied to the non-trainable weights will destroy what the model has learned.\n x = base_modelA(x, training=False)\n\n # rebuild the top layer\n x = tf.keras.layers.BatchNormalization()(x) # <- changed from the very first model\n x = tf.keras.layers.Dropout(0.2)(x)\n\n # output block\n x = tf.keras.layers.Dense(num_classes, activation='softmax')(x)\n\n return tf.keras.Model(inputs=inputA, outputs=x)\n\nwith strategy.scope():\n \n # make model\n model = make_model(num_classes=len(CLASSES))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The `'sparse_categorical'` versions of the loss and metrics are appropriate for a classification task with more than two labels, like this one.","metadata":{}},{"cell_type":"code","source":"# compile the model\nmodel.compile(\n optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),\n loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),\n metrics=['accuracy'],\n)\n\n# check the summary\nmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# show the model architecture\ntf.keras.utils.plot_model(\n model,\n to_file='model.pdf',\n show_shapes=True,\n show_dtype=False,\n show_layer_names=True,\n rankdir='TB',\n expand_nested=True,\n dpi=100,\n layer_range=None,\n# show_layer_activations=True,\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 6: Training","metadata":{}},{"cell_type":"code","source":"# add list of callbacks\ncallbacks = [\n tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=5) # -> early stopping, add if needed min_delta=1e-4 \n]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Fit Model ##\n\nAnd now we're ready to train the model. After defining a few parameters, we're good to go!","metadata":{}},{"cell_type":"code","source":"# Define training epochs\nEPOCHS = 200\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\n\nhistory = model.fit(\n ds_train,\n validation_data=ds_valid,\n epochs=EPOCHS,\n steps_per_epoch=STEPS_PER_EPOCH,\n callbacks=callbacks,\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This next cell shows how the loss and metrics progressed during training. Thankfully, it converges!","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(8, 6))\n\nplt.subplot(2, 1, 1)\nplt.plot(history.history['accuracy'],\n lw=3, color='tab:blue')\nplt.plot(history.history['val_accuracy'],\n lw=3, color='tab:orange')\nplt.ylabel('Accuracy')\nplt.xticks([])\n\nplt.subplot(2, 1, 2)\nplt.plot(history.history['loss'], label='Training',\n lw=3, color='tab:blue')\nplt.plot(history.history['val_loss'], label = 'Validation',\n lw=3, color='tab:orange')\n\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.legend(loc='upper right', fontsize='x-large')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Fine tuning the model\n\nThe second step is to unfreeze a number of layers and fit the model using smaller learning rate. In this example we show unfreezing all layers, but depending on specific dataset it may be desireble to only unfreeze a fraction of all layers.\n\nWhen the feature extraction with pretrained model works good enough, this step would give a very limited gain on validation accuracy. In our case we only see a small improvement, as ImageNet pretraining already exposed the model to a good amount of flowers.\n\nOn the other hand, when we use pretrained weights on a dataset that is more different from ImageNet, this fine-tuning step can be crucial as the feature extractor also needs to be adjusted by a considerable amount. Such a situation can be demonstrated if choosing CIFAR-100 dataset instead, where fine-tuning boosts validation accuracy by about 10% to pass 80% on EfficientNetB0. In such a case the convergence may take more than 50 epochs.\n\nA side note on freezing/unfreezing models: setting trainable of a Model will simultaneously set all layers belonging to the Model to the same trainable attribute. Each layer is trainable only if both the layer itself and the model containing it are trainable. Hence when we need to partially freeze/unfreeze a model, we need to make sure the trainable attribute of the model is set to True.","metadata":{}},{"cell_type":"code","source":"# first and foremost, we need to make the base_model trainable\n# this subsequently will enable us to freeze the layers inside that function\n# if we keep the base_model untrainable, we will not be able to do anything\n\n# get the base_model name and unfreeze all layers inside it\nmodel.get_layer('efficientnetb4').trainable = True","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create function to freeze model layers\n# define the fraction of layers to be unfreezed\ndef unfreeze_model(model, frac=0):\n '''\n Function to unfreeze layers inside a pre-trained (sub-)model.\n Take the main model and freeze fraction as inputs.\n '''\n \n # keep track the number of processed layers after certain loop\n layer_counter = 1\n \n # calculate the layer size of the submodel target\n submodel_size = len(model.get_layer('efficientnetb4').layers)\n \n \n # in case of EfficientNet algorithm, we need to keep the BatchNorm layers frozen\n for layer in model.get_layer('efficientnetb4').layers:\n \n # freeze the BatchNorm layers\n if isinstance(layer, tf.keras.layers.BatchNormalization):\n layer.trainable = False\n \n # if not BatchNorm, keep it frozen until the targeted freeze fraction\n elif layer_counter < int(frac*submodel_size):\n layer.trainable = False\n\n # start unfreezing after reaching the number of designated layer fraction \n else:\n layer.trainable = True\n \n # increase the layer counter\n layer_counter+=1\n \n # check trainable status, if necessary\n #print (layer, layer.trainable)\n\n # recompile the fine-tuned model\n model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-5),\n loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),\n metrics=['accuracy'])\n\n# make the submodel (or base_model) trainable\nunfreeze_model(model)\n\n# check the summary\nmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Importantly, although the base model becomes trainable, it is still running in inference mode since we passed training=False when calling it when we built the model. This means that the batch normalization layers inside won't update their batch statistics. If they did, they would wreck havoc on the representations learned by the model so far.","metadata":{}},{"cell_type":"code","source":"# continue training the model\nhistory_fine = model.fit(\n ds_train,\n validation_data=ds_valid,\n epochs=len(history.epoch)+EPOCHS,\n steps_per_epoch=STEPS_PER_EPOCH,\n callbacks=callbacks,\n initial_epoch=len(history.epoch) # <- start from the last epoch of previous training + 1\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visualize the fine-tuned model\nplt.figure(figsize=(8, 6))\n\nplt.subplot(2, 1, 1)\nplt.plot(history.history['accuracy'] + history_fine.history['accuracy'], # -> combine the results from previous training\n lw=3, color='tab:blue')\nplt.plot(history.history['val_accuracy'] + history_fine.history['val_accuracy'],\n lw=3, color='tab:orange')\nplt.ylabel('Accuracy')\nplt.xticks([])\n\nplt.axvline(x=len(history.epoch), ymin=0, ymax=5, ls='--',\n label='Start Fine Tuning', lw=3, color='tab:gray')\n\nplt.subplot(2, 1, 2)\nplt.plot(history.history['loss'] + history_fine.history['loss'], label='Training',\n lw=3, color='tab:blue')\nplt.plot(history.history['val_loss'] + history_fine.history['val_loss'], label = 'Validation',\n lw=3, color='tab:orange')\n\nplt.axvline(x=len(history.epoch), ymin=0, ymax=5, ls='--',\n label='Start Fine Tuning', lw=3, color='tab:gray')\n\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.legend(loc='upper right', fontsize='x-large')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 7: Evaluate Predictions #\n\nBefore making your final predictions on the test set, it's a good idea to evaluate your model's predictions on the validation set. This can help you diagnose problems in training or suggest ways your model could be improved. We'll look at two common ways of validation: plotting the **confusion matrix** and **visual validation**.","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\n\ndef display_confusion_matrix(cmat, score, precision, recall):\n plt.figure(figsize=(15,15))\n ax = plt.gca()\n ax.matshow(cmat, cmap='Reds')\n ax.set_xticks(range(len(CLASSES)))\n ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n ax.set_yticks(range(len(CLASSES)))\n ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n titlestring = \"\"\n if score is not None:\n titlestring += 'f1 = {:.3f} '.format(score)\n if precision is not None:\n titlestring += '\\nprecision = {:.3f} '.format(precision)\n if recall is not None:\n titlestring += '\\nrecall = {:.3f} '.format(recall)\n if len(titlestring) > 0:\n ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n plt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Confusion Matrix ##\n\nA [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) shows the actual class of an image tabulated against its predicted class. It is one of the best tools you have for evaluating the performance of a classifier.\n\nThe following cell does some processing on the validation data and then creates the matrix with the `confusion_matrix` function included in [`scikit-learn`](https://scikit-learn.org/stable/index.html).","metadata":{}},{"cell_type":"code","source":"cmdataset = get_validation_dataset(ordered=True)\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\n\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy()\ncm_probabilities = model.predict(images_ds)\ncm_predictions = np.argmax(cm_probabilities, axis=-1)\n\nlabels = range(len(CLASSES))\ncmat = confusion_matrix(\n cm_correct_labels,\n cm_predictions,\n labels=labels,\n)\ncmat = (cmat.T / cmat.sum(axis=1)).T # normalize","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You might be familiar with metrics like [F1-score](https://en.wikipedia.org/wiki/F1_score) or [precision and recall](https://en.wikipedia.org/wiki/Precision_and_recall). This cell will compute these metrics and display them with a plot of the confusion matrix. (These metrics are defined in the Scikit-learn module `sklearn.metrics`; we've imported them in the helper script for you.)","metadata":{}},{"cell_type":"code","source":"score = f1_score(\n cm_correct_labels,\n cm_predictions,\n labels=labels,\n average='macro',\n)\nprecision = precision_score(\n cm_correct_labels,\n cm_predictions,\n labels=labels,\n average='macro',\n)\nrecall = recall_score(\n cm_correct_labels,\n cm_predictions,\n labels=labels,\n average='macro',\n)\n\n# display_confusion_matrix(cmat, score, precision, recall)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print (score, precision, recall)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visual Validation ##\n\nIt can also be helpful to look at some examples from the validation set and see what class your model predicted. This can help reveal patterns in the kinds of images your model has trouble with.\n\nThis cell will set up the validation set to display 20 images at a time -- you can change this to display more or fewer, if you like.","metadata":{}},{"cell_type":"code","source":"dataset = get_validation_dataset()\ndataset = dataset.unbatch().batch(20)\nbatch = iter(dataset)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And here is a set of flowers with their predicted species. Run the cell again to see another set.","metadata":{}},{"cell_type":"code","source":"images, labels = next(batch)\nprobabilities = model.predict(images)\npredictions = np.argmax(probabilities, axis=-1)\ndisplay_batch_of_images((images, labels), predictions)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 8: Make Test Predictions #\n\nOnce you're satisfied with everything, you're ready to make predictions on the test set.","metadata":{}},{"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True)\n\nprint ('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = model.predict(test_images_ds)\npredictions = np.argmax(probabilities, axis=-1)\nprint (predictions)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll generate a file `submission.csv`. This file is what you'll submit to get your score on the leaderboard.","metadata":{}},{"cell_type":"code","source":"print('Generating submission.csv file...')\n\n# Get image ids from test set and convert to unicode\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n\n# Write the submission file\nnp.savetxt(\n 'submission.csv',\n np.rec.fromarrays([test_ids, predictions]),\n fmt=['%s', '%d'],\n delimiter=',',\n header='id,label',\n comments='',\n)\n\n# Look at the first few predictions\n!head submission.csv","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 9: Make a submission #\n\nIf you haven't already, create your own editable copy of this notebook by clicking on the **Copy and Edit** button in the top right corner. Then, submit to the competition by following these steps:\n\n1. Begin by clicking on the blue **Save Version** button in the top right corner of the window. This will generate a pop-up window. \n2. Ensure that the **Save and Run All** option is selected, and then click on the blue **Save** button.\n3. This generates a window in the bottom left corner of the notebook. After it has finished running, click on the number to the right of the **Save Version** button. This pulls up a list of versions on the right of the screen. Click on the ellipsis **(...)** to the right of the most recent version, and select **Open in Viewer**. This brings you into view mode of the same page. You will need to scroll down to get back to these instructions.\n4. Click on the **Output** tab on the right of the screen. Then, click on the file you would like to submit, and click on the blue **Submit** button to submit your results to the leaderboard.\n\nYou have now successfully submitted to the competition!\n\nIf you want to keep working to improve your performance, select the blue **Edit** button in the top right of the screen. Then you can change your code and repeat the process. There's a lot of room to improve, and you will climb up the leaderboard as you work.\n","metadata":{}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161321) to chat with other Learners.*","metadata":{}},{"cell_type":"code","source":"print ('All process finished!')","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}