{"cells":[{"metadata":{},"cell_type":"markdown","source":"# EfficientNet+Augmentation for Cassava Disease Classification using TF.Keras"},{"metadata":{},"cell_type":"markdown","source":"This notebook presents a full pipeline to load the data, apply advanced data augmentation, train an EfficientNet and use the model to predict over the test images. To make it possible to run within the allocated time for notebooks, this notebook will only present a single fold with a split of 80% for training and 20% for validation. Due to the original image size of 600x800 pixels, we will randomly crop 512x512 images from original images in order to keep the highest image resolution possible for our model training. Previous versions of this notebook used resized images and the results were extremely poor in comparison (~0.42 accuracy)."},{"metadata":{},"cell_type":"markdown","source":"Below are some version notes which were written at version 20 so I only included what I could remember."},{"metadata":{},"cell_type":"markdown","source":"Version notes:\n* 27: Fix bug in the TTA process.\n* 26: Move from using `ImageDataGenerator` to `tf.data` to load data into the model. This allows to tune the Normalization layer in the pretrained EfficientNetB3 model provided by `tf.keras`. By default, images are normalized using the `imagenet` dataset's mean and standard deviation. *- score: 0.888*\n* 24: Test CosineDecay instead of ReduceLRonPlateau and increase the dropout rate within the EfficientNetB3 model. *- score: 0.885*\n* 22: Replace the custom generator and data augmentation using `imgaug` with the new Keras preprocessing layers. Also increase the image size from 300x300 to 512x512. *- score: 0.880*\n* 21: *- score: 0.883*\n* 20: Fix bug activating the \"training mode\" by default in the custom generator, even during validation. Removed data standardization and class weights, simplify the bottleneck layers of the model, and remove the dropout from the data augmentation techniques. *- score: 0.873*\n* 17: Additional data augmentation techniques. *- score: 0.821*\n* 16: Tighter scan of the images at test time. *- score: 0.857*\n* 15: Add data standardization. (Cannot remember if there was a bug but the score was abnormaly low) *- score: 0.692*\n* 11: Train the model by randomly cropping 300x300pixel tiles from the original images *- score: 0.847*\n* 9: Classification from resized images. *- score: 0.421* "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom PIL import Image\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom tqdm import tqdm\nfrom sklearn.utils import shuffle\nfrom sklearn.utils import class_weight\nfrom sklearn.preprocessing import minmax_scale\nimport random\nimport cv2\nfrom imgaug import augmenters as iaa\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport tensorflow as tf\nfrom tensorflow.keras.models import Sequential, Model\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.layers import Dense, Dropout, Activation, Input, BatchNormalization, GlobalAveragePooling2D\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.experimental import CosineDecay\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.applications import EfficientNetB3\nfrom tensorflow.keras.layers.experimental.preprocessing import RandomCrop,CenterCrop, RandomRotation","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prepare the training and validation data generators"},{"metadata":{"trusted":true},"cell_type":"code","source":"training_folder = '../input/cassava-leaf-disease-classification/train_images/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"samples_df = pd.read_csv(\"../input/cassava-leaf-disease-classification/train.csv\")\nsamples_df = shuffle(samples_df, random_state=42)\nsamples_df[\"filepath\"] = training_folder+samples_df[\"image_id\"]\nsamples_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I keep 80% of the data provided for training and retain the other 20% for validation during my training process."},{"metadata":{"trusted":true},"cell_type":"code","source":"training_percentage = 0.8\ntraining_item_count = int(len(samples_df)*training_percentage)\nvalidation_item_count = len(samples_df)-int(len(samples_df)*training_percentage)\ntraining_df = samples_df[:training_item_count]\nvalidation_df = samples_df[training_item_count:]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Initially, I had set the image size to 300x300 in order to fit the original input size to the EfficientnetB3 model. However, as we are randomly cropping these images from the original 600x800-pixel images, it appears that using a dimension of 512x512 pixels leads to better results."},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 8\nimage_size = 512\ninput_shape = (image_size, image_size, 3)\ndropout_rate = 0.4\nclasses_to_predict = sorted(training_df.label.unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The code below allows to load the data from the dataframes using `tf.data`."},{"metadata":{"trusted":true},"cell_type":"code","source":"training_data = tf.data.Dataset.from_tensor_slices((training_df.filepath.values, training_df.label.values))\nvalidation_data = tf.data.Dataset.from_tensor_slices((validation_df.filepath.values, validation_df.label.values))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_image_and_label_from_path(image_path, label):\n    img = tf.io.read_file(image_path)\n    img = tf.image.decode_jpeg(img, channels=3)\n    return img, label\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n\ntraining_data = training_data.map(load_image_and_label_from_path, num_parallel_calls=AUTOTUNE)\nvalidation_data = validation_data.map(load_image_and_label_from_path, num_parallel_calls=AUTOTUNE)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"training_data_batches = training_data.shuffle(buffer_size=1000).batch(batch_size).prefetch(buffer_size=AUTOTUNE)\nvalidation_data_batches = validation_data.shuffle(buffer_size=1000).batch(batch_size).prefetch(buffer_size=AUTOTUNE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I also prepare a special dataset that will be fed to the Normalization layer. The EfficientnetB3 provided by `tf.keras` includes an out-of-the-box Normalization layer fit onto the `imagenet` dataset. Therefore, we can pull that layer and use the `adapt` function to refit it to the Cassava Disease dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"adapt_data = tf.data.Dataset.from_tensor_slices(training_df.filepath.values)\ndef adapt_mode(image_path):\n    img = tf.io.read_file(image_path)\n    img = tf.image.decode_jpeg(img, channels=3)\n    img = layers.experimental.preprocessing.Rescaling(1.0 / 255)(img)\n    return img\n\nadapt_data = adapt_data.map(adapt_mode, num_parallel_calls=AUTOTUNE)\nadapt_data_batches = adapt_data.shuffle(buffer_size=1000).batch(batch_size).prefetch(buffer_size=AUTOTUNE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The data augmentation preprocessing layers below will be used when training the model but disabled in inference mode."},{"metadata":{"trusted":true},"cell_type":"code","source":"data_augmentation_layers = tf.keras.Sequential(\n    [\n        layers.experimental.preprocessing.RandomCrop(height=image_size, width=image_size),\n        layers.experimental.preprocessing.RandomFlip(\"horizontal_and_vertical\"),\n        layers.experimental.preprocessing.RandomRotation(0.25),\n        layers.experimental.preprocessing.RandomZoom((-0.2, 0)),\n        layers.experimental.preprocessing.RandomContrast((0.2,0.2))\n    ]\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Simply reusing some of the code from [this tutorial](https://www.tensorflow.org/tutorials/images/data_augmentation) to show what our augmentations look like. I add the image previously opened to a batch and pass it through the data augmentation layers."},{"metadata":{"trusted":true},"cell_type":"code","source":"image = Image.open(\"../input/cassava-leaf-disease-classification/train_images/3412658650.jpg\")\nplt.imshow(image)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image = tf.expand_dims(np.array(image), 0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10, 10))\nfor i in range(9):\n  augmented_image = data_augmentation_layers(image)\n  ax = plt.subplot(3, 3, i + 1)\n  plt.imshow(augmented_image[0])\n  plt.axis(\"off\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build the model"},{"metadata":{},"cell_type":"markdown","source":"I am using an EfficientNetB3 on top of which I add some outputs layers to predict our 5 disease classes. I decided to load the imagenet pretrained weights locally to keep the internet off (part of the requirements to submit a kernel to this competition)."},{"metadata":{},"cell_type":"markdown","source":"At this point, you may have noticed that I have not used any kind of normalization or rescaling. I recently discovered that there is a Normalization layer included in Keras'pretrained EfficientNet, as mentioned [here](https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/#keras-implementation-of-efficientnet)."},{"metadata":{"trusted":true},"cell_type":"code","source":"efficientnet = EfficientNetB3(weights=\"../input/efficientnetb3-notop/efficientnetb3_notop.h5\", \n                              include_top=False, \n                              input_shape=input_shape, \n                              drop_connect_rate=dropout_rate)\n\ninputs = Input(shape=input_shape)\naugmented = data_augmentation_layers(inputs)\nefficientnet = efficientnet(augmented)\npooling = layers.GlobalAveragePooling2D()(efficientnet)\ndropout = layers.Dropout(dropout_rate)(pooling)\noutputs = Dense(len(classes_to_predict), activation=\"softmax\")(dropout)\nmodel = Model(inputs=inputs, outputs=outputs)\n    \nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The 3rd layer of the Efficientnet is the Normalization layer, which can be tuned to our new dataset instead of `imagenet`. Be patient on this one, it does take a bit of time as we're going through the entire training set."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nmodel.get_layer('efficientnetb3').get_layer('normalization').adapt(adapt_data_batches)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I wanted to try the new `CosineDecay` function implemented in `tf.keras` as it seemed promising and I struggled to find the right settings (if there were any) for the `ReduceLROnPlateau`."},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = 8\ndecay_steps = int(round(len(training_df)/batch_size))*epochs\ncosine_decay = CosineDecay(initial_learning_rate=1e-4, decay_steps=decay_steps, alpha=0.3)\n\ncallbacks = [ModelCheckpoint(filepath='best_model.h5', monitor='val_loss', save_best_only=True)]\n\nmodel.compile(loss=\"sparse_categorical_crossentropy\", optimizer=tf.keras.optimizers.Adam(cosine_decay), metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(training_data_batches,\n                  epochs = epochs, \n                  validation_data=validation_data_batches,\n                  callbacks=callbacks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Verification of the training process"},{"metadata":{},"cell_type":"markdown","source":"First, we will check that we perform on similar level on both the training and validation. The training curve will also tell us if we stopped training too early or may have overfitted in comparison to the validation data."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('Loss over epochs')\nplt.ylabel('Loss')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Validation'], loc='best')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We load the best weights that were kept from the training phase. Just to check how our model is performing, we will attempt predictions over the validation set. This can help to highlight any classes that will be consistently miscategorised."},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_weights(\"best_model.h5\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prediction on test images"},{"metadata":{"trusted":true},"cell_type":"code","source":"def scan_over_image(img_path, crop_size=512):\n    '''\n    Will extract 512x512 images covering the whole original image\n    with some overlap between images\n    '''\n    \n    img = Image.open(img_path)\n    img_height, img_width = img.size\n    img = np.array(img)\n    \n    y = random.randint(0,img_height-crop_size)\n    x = random.randint(0,img_width-crop_size)\n\n    x_img_origins = [0,img_width-crop_size]\n    y_img_origins = [0,img_height-crop_size]\n    img_list = []\n    for x in x_img_origins:\n        for y in y_img_origins:\n            img_list.append(img[x:x+crop_size , y:y+crop_size,:])\n  \n    return np.array(img_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_samples(img_path):\n    '''\n    Display all 512x512 images extracted from original images\n    '''\n    \n    img_list = scan_over_image(img_path)\n    sample_number = len(img_list)\n    fig = plt.figure(figsize = (8,sample_number))\n    for i in range(0,sample_number):\n        ax = fig.add_subplot(2, 4, i+1)\n        ax.imshow(img_list[i])\n        ax.set_title(str(i))\n    plt.tight_layout()\n    plt.show()\n\ndisplay_samples(\"../input/cassava-leaf-disease-classification/train_images/3412658650.jpg\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I apply some very basic test time augmentation to every local image extracted from the original 600x800 image. We know we can do some fancy augmentation with `imgaug` or `albumentations` but I wanted to do that exclusively with Keras' preprocessing layers to keep the \"cleanest\" pipeline possible."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_time_augmentation_layers = tf.keras.Sequential(\n    [\n        layers.experimental.preprocessing.RandomFlip(\"horizontal_and_vertical\"),\n        layers.experimental.preprocessing.RandomZoom((-0.2, 0)),\n        layers.experimental.preprocessing.RandomContrast((0.2,0.2))\n    ]\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def predict_and_vote(image_filename, folder, TTA_runs=4):\n    '''\n    Run the model over 4 local areas of the given image,\n    before making a decision depending on the most predicted\n    disease.\n    '''\n    \n    #apply TTA to each of the 4 images and sum all predictions for each local image\n    localised_predictions = []\n    local_image_list = scan_over_image(folder+image_filename)\n    for local_image in local_image_list:\n        duplicated_local_image = tf.convert_to_tensor(np.array([local_image for i in range(TTA_runs)]))\n        augmented_images = test_time_augmentation_layers(duplicated_local_image)\n        \n        predictions = model.predict(augmented_images)\n        localised_predictions.append(np.sum(predictions, axis=0))\n    \n    #sum all predictions from all 4 images and retrieve the index of the highest value\n    global_predictions = np.sum(np.array(localised_predictions),axis=0)\n    final_prediction = np.argmax(global_predictions)\n    \n    return final_prediction","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_predictions_over_image_list(image_list, folder):\n    predictions = []\n    with tqdm(total=len(image_list)) as pbar:\n        for image_filename in image_list:\n            pbar.update(1)\n            predictions.append(predict_and_vote(image_filename, folder))\n    return predictions","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First, I test my entire prediction pipeline on the validation set as we have little visibility over the test set."},{"metadata":{"trusted":true},"cell_type":"code","source":"validation_df[\"results\"] = run_predictions_over_image_list(validation_df[\"image_id\"], training_folder)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!cat ../input/cassava-leaf-disease-classification/label_num_to_disease_map.json","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"validation_df[:20]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"true_positives = 0\nprediction_distribution_per_class = {\"0\":{\"0\": 0, \"1\": 0, \"2\":0, \"3\":0, \"4\":0},\n                                     \"1\":{\"0\": 0, \"1\": 0, \"2\":0, \"3\":0, \"4\":0},\n                                     \"2\":{\"0\": 0, \"1\": 0, \"2\":0, \"3\":0, \"4\":0},\n                                     \"3\":{\"0\": 0, \"1\": 0, \"2\":0, \"3\":0, \"4\":0},\n                                     \"4\":{\"0\": 0, \"1\": 0, \"2\":0, \"3\":0, \"4\":0}}\nnumber_of_images = len(validation_df)\nfor idx, pred in validation_df.iterrows():\n    if int(pred[\"label\"]) == pred.results:\n        true_positives+=1\n    prediction_distribution_per_class[str(pred[\"label\"])][str(pred.results)] += 1\nprint(\"accuracy: {}%\".format(true_positives/number_of_images*100))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"prediction_distribution_per_class","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also have a better understanding of where this new model misclassifies diseases by plotting a heatmap from the results. Each row in this heatmap is normalised to highlight the classification distribution per disease without being bothered by the fact that the dataset is imbalanced."},{"metadata":{"trusted":true},"cell_type":"code","source":"heatmap_df = pd.DataFrame(columns={\"groundtruth\",\"prediction\",\"value\"})\nfor key in prediction_distribution_per_class.keys():\n    for pred_key in prediction_distribution_per_class[key].keys():\n        value = prediction_distribution_per_class[key][pred_key]/validation_df.query(\"label==@key\").count()[0]\n        heatmap_df = heatmap_df.append({\"groundtruth\":key,\"prediction\":pred_key,\"value\":value}, ignore_index=True)   \n\nheatmap = heatmap_df.pivot(index='groundtruth', columns='prediction', values='value')\nsns.heatmap(heatmap,cmap=\"Blues\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_folder = '../input/cassava-leaf-disease-classification/test_images/'\nsubmission_df = pd.DataFrame(columns={\"image_id\",\"label\"})\nsubmission_df[\"image_id\"] =  os.listdir(test_folder)\nsubmission_df[\"label\"] = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df[\"label\"] = run_predictions_over_image_list(submission_df[\"image_id\"], test_folder)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Thanks for reading this notebook! If you found this notebook helpful, please give it an upvote. It is always greatly appreciated!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}