{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Noisy Labels Lead To Incorrect Predictions\n\n![](https://miro.medium.com/max/2720/0*ZGkRWDDZQGHJCvBT)\n\nHi everyone, today I'll be going through with you my method of denoising the data in this competition. As you know, a significant portion of images in this competition was mislabelled, which is what we know as \"label noise\". With label noise, it is very likely that your model will pick up the wrong information and make incorrect predictions. For example, if it correctly predicts a dog as a dog, but we tell it that the dog is actually a cat, then it will learn (incorrectly) that features of a dog are supposed to be features of a cat.\n\nHere is the brief overview of this walkthrough:\n\n1. Importing Libraries And Data\n2. Image Augmentation\n3. Model Building\n4. Stratified 5-Fold Cross Validation\n5. Retrieving Out-Of-Fold Accuracy and Predictions\n6. Test Time Augmentation\n7. Denoising The Data\n8. Re-Fitting The Model \n\n*Disclaimer: This method is not backed by any research or empirical evidence. It worked for me and I just wanted to share with the community something I felt would be helpful.*"},{"metadata":{},"cell_type":"markdown","source":"# Importing Libraries and Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Seed everything for reproducible results\nimport random\nfrom numpy.random import seed\nfrom tensorflow.random import set_seed\n\nseed_value = 42\nrandom.seed(seed_value)\nseed(seed_value)\nset_seed(seed_value)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport warnings\nimport random\nimport os\nimport cv2\nimport sys\nfrom pylab import rcParams\nfrom PIL import Image\nfrom tqdm import tqdm\nwarnings.filterwarnings('ignore')\n\nimport tensorflow as tf\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import Input\nfrom tensorflow.keras.models import Model, load_model\nfrom tensorflow.keras.layers import Dense, Flatten, Dropout, Activation, Input, GlobalAveragePooling2D\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.applications import EfficientNetB0\nfrom tensorflow.keras.mixed_precision import experimental as mixed_precision\nfrom sklearn.model_selection import StratifiedKFold","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"policy = mixed_precision.Policy('mixed_float16')\nmixed_precision.set_policy(policy) #shortens training time by 2x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(\"../input/cassava-leaf-disease-classification/train.csv\")\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train[\"label\"] = df_train[\"label\"].astype(str) #convert to str as we want to use cross entropy loss later\ndf_train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image Augmentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size=32\nimage_size=300\n\ninput_shape = (image_size, image_size, 3)\ntarget_size = (image_size, image_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_augmentation = tf.keras.Sequential(\n    [\n        tf.keras.layers.experimental.preprocessing.RandomCrop(image_size, image_size),\n        tf.keras.layers.experimental.preprocessing.RandomFlip(\"horizontal_and_vertical\"),\n        tf.keras.layers.experimental.preprocessing.RandomRotation(0.25),\n        tf.keras.layers.experimental.preprocessing.RandomContrast(0.2)\n    ])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"path = \"../input/cassava-leaf-disease-classification/train_images/\"\nfiles = df_train[\"image_id\"].tolist()\nfile = random.choice(files)\nimage = Image.open(path + file)\nplt.imshow(image)\nplt.axis(\"off\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Augmentation is necessary as it helps our model generalize better. Let's see what our images can look like after augmentation. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"image = tf.expand_dims(np.array(image), 0)\n\nplt.figure(figsize=(14, 14))\nfor i in range(9):\n    augmented_image = img_augmentation(image)\n    ax = plt.subplot(3, 3, i + 1)\n    plt.imshow(augmented_image[0])\n    plt.axis(\"off\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def DataGenerator(train_set, val_set):\n    \n    train_datagen = ImageDataGenerator().flow_from_dataframe(\n                  dataframe = train_set,\n                  directory='../input/cassava-leaf-disease-classification/train_images',\n                  x_col='image_id',\n                  y_col='label',\n                  target_size=target_size,\n                  batch_size=batch_size,\n                  shuffle=True,\n                  class_mode='sparse',\n                  seed=seed_value)\n\n    val_datagen = ImageDataGenerator().flow_from_dataframe(\n                dataframe = val_set,\n                directory='../input/cassava-leaf-disease-classification/train_images',\n                x_col='image_id',\n                y_col='label',\n                target_size=target_size,\n                batch_size=batch_size,\n                shuffle=False,\n                class_mode='sparse',\n                seed=seed_value)\n    \n    return train_datagen, val_datagen","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Building"},{"metadata":{},"cell_type":"markdown","source":"I selected EfficientNetB0 for this notebook as it's the smallest and fastest pre-trained model I could find. For your own purposes, you should select your best performing architecture. We will use a Cosine Decay Learning Rate Schedule for our model (more intuition on that below)."},{"metadata":{},"cell_type":"markdown","source":"![](https://www.jeremyjordan.me/content/images/2018/02/Screen-Shot-2018-02-24-at-11.47.09-AM.png)"},{"metadata":{},"cell_type":"markdown","source":"We want a relatively high initial learning rate as we want the loss function to converge to a global minimum (visualized on a 2D plane as the bottom of a curve) as soon as possible. However, as it converges, the gradient might change, requiring a lower learning rate to ensure an optimal learning rate for convergence. Hence, we can't have a fixed learning rate as the initial learning rate may become too high, preventing convergence (as shown in the diagram above). So, we will implement a decay schedule for our learning rate, so that it decreases over time to ensure convergence. "},{"metadata":{},"cell_type":"markdown","source":"![](https://miro.medium.com/max/1266/1*2NAuh6DbcrrMv4Voq5yG9A.png)"},{"metadata":{},"cell_type":"markdown","source":"And this is how our learning rate will look like over time after implementing the decay."},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = 3\ntotal_steps = (int(len(df_train)*0.8/batch_size)+1)*epochs\n\nlr = tf.keras.experimental.CosineDecay(initial_learning_rate=1e-3, decay_steps=total_steps)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model():\n    base_model = EfficientNetB0(include_top=False, weights=\"imagenet\", input_shape=input_shape)\n\n    # Rebuild top\n    inputs = Input(shape=input_shape)\n    base = base_model(inputs)\n    pooling = GlobalAveragePooling2D()(base)\n    outputs = Dense(5, activation=\"softmax\", dtype='float32')(pooling) #necessary for mixed-precision training to work properly\n\n    # Compile\n    model = Model(inputs=inputs, outputs=outputs)\n    optimizer = tf.keras.optimizers.Adam(learning_rate=lr)\n    model.compile(optimizer=optimizer, loss=\"sparse_categorical_crossentropy\", metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Stratified 5-Fold Cross Validation"},{"metadata":{},"cell_type":"markdown","source":"As we are using out-of-fold (OOF) accuracy as the baseline to denoise our data, this part is especially important. Make sure you use 5-fold CV to ensure that your model has a 80/20 training/validation split, with each validation split being a different subset of the entire data. So after 5 folds, your model would have used all the data as validation data, which we can then use to retrieve our OOF accuracy. Stratified KFold helps preserve the class imbalance across all folds, standardizing the difficulty."},{"metadata":{"trusted":true},"cell_type":"code","source":"fold_number = 0\nn_splits = 5\noof_accuracy = []\n\ntf.keras.backend.clear_session()\nskf = StratifiedKFold(n_splits=n_splits, random_state=seed_value)\nfor train_index, val_index in skf.split(df_train[\"image_id\"], df_train[\"label\"]):\n    train_set = df_train.loc[train_index]\n    val_set = df_train.loc[val_index]\n    train_datagen, val_datagen = DataGenerator(train_set, val_set)\n    model = build_model()\n    print(\"Training fold no.: \" + str(fold_number+1))\n\n    model_name = \"effnetb0 \"\n    fold_name = \"fold.h5\"\n    filepath = model_name + str(fold_number+1) + fold_name\n    callbacks = [ModelCheckpoint(filepath=filepath, monitor='val_accuracy', save_best_only=True)]\n\n    history = model.fit(train_datagen, epochs=epochs, validation_data=val_datagen, callbacks=callbacks)\n    oof_accuracy.append(max(history.history[\"val_accuracy\"]))\n    fold_number += 1\n    if fold_number == n_splits:\n        print(\"Training finished!\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Retrieving Out-Of-Fold Accuracy\nAfter this, we can see what's our average OOF accuracy:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(\"Average Out-Of-Fold Accuracy: {:.2f}\".format(np.mean(oof_accuracy)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now to retrieve our OOF predictions, we have to load each model and get them to predict the validation data from their fold."},{"metadata":{"trusted":true},"cell_type":"code","source":"# First we load our models\nmodels = [] \nfor i in range(5):\n    effnet = load_model(\"./effnetb0 \" + str(i+1) + \"fold.h5\")\n    models.append(effnet)\n\nmodel_one = models[0]\nmodel_two = models[1]\nmodel_three = models[2]\nmodel_four = models[3]\nmodel_five = models[4]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Then we get our validation data\ndf = pd.read_csv(\"../input/cassava-leaf-disease-classification/train.csv\")\nval_list = []\n\nskf = StratifiedKFold(n_splits=5, random_state=seed_value)\nfor train_index, val_index in skf.split(df[\"image_id\"], df[\"label\"]):\n    val_list.append(val_index)\n\none_fold = df.loc[val_list[0]]\ntwo_fold = df.loc[val_list[1]]\nthree_fold = df.loc[val_list[2]]\nfour_fold = df.loc[val_list[3]]\nfive_fold = df.loc[val_list[4]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Test Time Augmentation\n\nTest Time Augmentation (TTA), is as the name suggests: performing augmentations on your test images before getting your model to predict them. The augmentations are usually (part of) the augmentations you used during training time. This generally leads to a slight performance boost. Here's a brief intuition why: \n\nWhen we use softmax cross entropy for multi-class classification, the predictions are usually produced in the form of one-hot labels. That is to say, in the context of this competition, if my model predicts with an 60% confidence that an image is label 3, 30% confidence that the image is label 4, and 10% that the image is label 1, then the one-hot prediction would be like this: [0, 0.1, 0.0, 0.6, 0.3]\n\nDuring TTA, we duplicate the original image and perform augmentation and get the model to aggregrate predictions across those images, leading to better generalization (refer to figure below).\n\n![](https://stepup.ai/content/images/2020/08/test_time_augmentation_concept.png)\n\nGreat, now that we have the intuition, let's define some custom functions for this to work."},{"metadata":{"trusted":true},"cell_type":"code","source":"tta = tf.keras.Sequential(\n    [\n        tf.keras.layers.experimental.preprocessing.RandomFlip(\"horizontal_and_vertical\"),\n        tf.keras.layers.experimental.preprocessing.RandomRotation(0.25),\n        tf.keras.layers.experimental.preprocessing.RandomContrast(0.2)\n    ]\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def duplicate_image(img_path, image_size=image_size, tta_runs=2):\n\n    img = Image.open(img_path)\n    img = img.resize((image_size, image_size))\n    img_height, img_width = img.size\n    img = np.array(img)\n    \n    img_list = []\n    for i in range(tta_runs):\n        img_list.append(img)\n  \n    return np.array(img_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def predict_with_tta(image_filename, folder, tta_runs=2):\n    \n    #apply TTA to each of the 3 images and sum all predictions for each local image\n    localised_predictions = []\n    local_image_list = duplicate_image(folder+image_filename)\n    for local_image in local_image_list:\n        local_image = tf.expand_dims(local_image,0)\n        augmented_images = [tta(local_image) for i in range(tta_runs)]\n        predictions = model.predict(np.array(augmented_images[0]))\n        localised_predictions.append(np.sum(predictions, axis=0))\n    \n    #sum all predictions from all 3 images and retrieve the index of the highest value\n    global_predictions = np.sum(np.array(localised_predictions),axis=0)\n    max_value = max(global_predictions)\n    final_prediction = np.argmax(global_predictions)\n    \n    return [final_prediction, max_value, global_predictions]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at how this works out!"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_folder = \"../input/cassava-leaf-disease-classification/train_images/\"\ntrain_image = \"1000015157.jpg\"\npredictions = predict_with_tta(train_image, train_folder)\n\nprint(\"Predicted Label: \", predictions[0])\nprint(\"Predicted Label Value: \", predictions[1])\nprint(\"Predicted One-Hot Label: \", predictions[2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Very nice. If you realize, the predicted label value is >1 because we implemented 2x TTA. This pushes the max label value (100% confidence) to 2. If you implement 3x TTA, it will be 3 and so on.\n\nAlso, this is a percentage estimated as a float. If you want to know how confident in the prediction the model is, simply take the label value/number of tta*100. For example.."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Confidence Level: {:.2f}\".format(predictions[1]/2*100), \"%\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Retrieving Out-Of-Fold Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def predict_image_list(image_list, folder):\n    predictions = []\n    values = []\n    with tqdm(total=len(image_list)) as pbar:\n        for image_filename in image_list:\n            pbar.update(1)\n            predictions.append(predict_with_tta(image_filename, folder)[0])\n            values.append(predict_with_tta(image_filename, folder)[1])\n    return [predictions, values]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"model = model_one\nplaceholder = predict_image_list(one_fold[\"image_id\"], train_folder)\none_fold[\"pred\"] = placeholder[0]\none_fold[\"value\"] = placeholder[1]\none_fold.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"model = model_two\nplaceholder = predict_image_list(two_fold[\"image_id\"], train_folder)\ntwo_fold[\"pred\"] = placeholder[0]\ntwo_fold[\"value\"] = placeholder[1]\ntwo_fold.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"model = model_three\nplaceholder = predict_image_list(three_fold[\"image_id\"], train_folder)\nthree_fold[\"pred\"] = placeholder[0]\nthree_fold[\"value\"] = placeholder[1]\nthree_fold.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"model = model_four\nplaceholder = predict_image_list(four_fold[\"image_id\"], train_folder)\nfour_fold[\"pred\"] = placeholder[0]\nfour_fold[\"value\"] = placeholder[1]\nfour_fold.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"model = model_five\nplaceholder = predict_image_list(five_fold[\"image_id\"], train_folder)\nfive_fold[\"pred\"] = placeholder[0]\nfive_fold[\"value\"] = placeholder[1]\nfive_fold.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Denoising The Data\n\nSo what we need to do now is to set a confidence level (or threshold) for our model to clean the labels. Here is how we are going to do it:\n\n1. Identify the images which our model predicted incorrectly\n2. Determine if the confidence level is above our threshold\n3. If it is, remove or relabel the data\n4. If not, leave it as it is\n\nThe brief intuition behind why this works is because your model is already predicting wrong labels with very high confidence for those images. Hence, it is unlikely to predict it correctly even if you continue to feed it the image for training (as it is unable to get it right anyway). Thus, by removing these noisy labels, it can then focus on the correct labels and strengthen its predictions on the clean data. \n\nOf course, you do not want to remove so much data that it is unable to \"learn\" the \"difficult\" examples, or worse, change the distribution of the dataset. For demonstration purposes, I will set this threshold at 80% confidence."},{"metadata":{"trusted":true},"cell_type":"code","source":"threshold = 2*0.8","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will then retrieve the index of the selected images to be \"denoised\" and combine them into a list."},{"metadata":{"trusted":true},"cell_type":"code","source":"mask1 = (one_fold[\"label\"] != one_fold[\"pred\"]) & (one_fold[\"value\"] >= threshold)\none_list = one_fold[mask1].index.to_list()\n\nmask2 = (two_fold[\"label\"] != two_fold[\"pred\"]) & (two_fold[\"value\"] >= threshold)\ntwo_list = two_fold[mask2].index.to_list()\n\nmask3 = (three_fold[\"label\"] != three_fold[\"pred\"]) & (three_fold[\"value\"] >= threshold)\nthree_list = three_fold[mask3].index.to_list()\n\nmask4 = (four_fold[\"label\"] != four_fold[\"pred\"]) & (four_fold[\"value\"] >= threshold)\nfour_list = four_fold[mask4].index.to_list()\n\nmask5 = (five_fold[\"label\"] != five_fold[\"pred\"]) & (five_fold[\"value\"] >= threshold)\nfive_list = five_fold[mask5].index.to_list()\n\ncombined_list = list(np.unique(one_list + two_list + three_list + four_list + five_list))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at the distribution of this list."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"temp = df_train.iloc[combined_list]\ntemp[\"label\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"pct = len(temp)/len(df_train)*100\nprint(\"Percentage of Data To Be Removed: {:.2f}\".format(pct), \"%\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Re-Fitting The Model\nWe will now drop these images from our dataset and see how it affects the performance of our model."},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df_train.drop(combined_list, axis=\"index\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.reset_index(drop=True, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fold_number = 0\nn_splits = 5\noof_accuracy = []\n\ntf.keras.backend.clear_session()\nskf = StratifiedKFold(n_splits=n_splits, random_state=seed_value)\nfor train_index, val_index in skf.split(df[\"image_id\"], df[\"label\"]):\n    train_set = df.loc[train_index]\n    val_set = df.loc[val_index]\n    train_datagen, val_datagen = DataGenerator(train_set, val_set)\n    model = build_model()\n    print(\"Training fold no.: \" + str(fold_number+1))\n\n    model_name = \"denoised effnetb0 \"\n    fold_name = \"fold.h5\"\n    filepath = model_name + str(fold_number+1) + fold_name\n    callbacks = [ModelCheckpoint(filepath=filepath, monitor='val_accuracy', save_best_only=True)]\n\n    history = model.fit(train_datagen, epochs=epochs, validation_data=val_datagen, callbacks=callbacks)\n    oof_accuracy.append(max(history.history[\"val_accuracy\"]))\n    fold_number += 1\n    if fold_number == n_splits:\n        print(\"Training finished!\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(\"Average Out-Of-Fold Accuracy: {:.2f}\".format(np.mean(oof_accuracy)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see, just by removing less than 5% of the data, our CV increased significantly. Of course, a fairer comparison would be to keep the original validation splits, but that would require several more lines of code. Personally, I got a 0.01 LB increase after denoising the labels this way, so I hope this will help you too! If you liked this notebook, please give it an upvote. It motivates me to continue to make high-quality notebooks. Do check out my other notebook where I cover an end-to-end solution in Keras for this competition too: https://www.kaggle.com/junyingsg/end-to-end-cassava-disease-classification-in-keras\n\nSome of my other notebooks unrelated to this competition:\n- https://www.kaggle.com/junyingsg/in-depth-analysis-of-women-s-e-commerce-reviews\n- https://www.kaggle.com/junyingsg/a-clean-analysis-of-covid-19-data\n- https://www.kaggle.com/junyingsg/a-beginner-s-comprehensive-guide-to-using-xgboost\n\nThanks for the support & happy kaggling!"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}