{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Albumentations\n![](https://neurohive.io/wp-content/uploads/2019/03/Screenshot-from-2019-03-14-00-53-12.png)"},{"metadata":{},"cell_type":"markdown","source":"## + CutMix(up)\n![](https://miro.medium.com/max/4176/1*IR3uTsclxKdzKIXDlTiVgg.png)"},{"metadata":{},"cell_type":"markdown","source":"Hi everyone, welcome back to another Tensorflow implementation of state-of-the-art image augmentations today. I previously shared two notebooks with augmentations done through ImageDataGenerator, but that was limited in flexibility. Today, I'm going to show you how to utilize the customizability of tf.data to implementation Albumentations with CutMix(up).\n\nBefore I start, do checkout my previous notebooks for this competition if you haven't as they will provide good background:\n- https://www.kaggle.com/junyingsg/step-by-step-guide-to-denoising-your-labels\n- https://www.kaggle.com/junyingsg/end-to-end-cassava-disease-classification-in-keras\n\nI also want to give a shoutout to the notebook who provided the implementation of CutMix(up) I drew from:\n- https://www.kaggle.com/itsuki9180/efficientnet-and-cutmixup-with-tpu-train-phase"},{"metadata":{},"cell_type":"markdown","source":"# Importing libraries and data"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom PIL import Image\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom tqdm import tqdm\nfrom sklearn.utils import shuffle\nfrom sklearn.utils import class_weight\nimport random\nimport cv2\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import Model, Sequential, load_model\nfrom tensorflow.keras.layers import Dense, Flatten, Dropout, Activation, Input\nfrom tensorflow.keras.layers import BatchNormalization, GlobalAveragePooling2D\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.applications import EfficientNetB0\nfrom sklearn.model_selection import StratifiedShuffleSplit, StratifiedKFold\nimport tensorflow_addons as tfa\nimport albumentations as A\nfrom functools import partial\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.mixed_precision import experimental as mixed_precision\npolicy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\nmixed_precision.set_policy(policy)\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\nfrom numpy.random import seed\nfrom tensorflow.random import set_seed\n\nseed_value = 42\nrandom.seed(seed_value)\nseed(seed_value)\nset_seed(seed_value)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(\"../input/cassava-leaf-disease-classification/train.csv\")\ntraining_folder = '../input/cassava-leaf-disease-classification/train_images/'\ndf_train[\"filepath\"] = training_folder+df_train[\"image_id\"]\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Converting data to tf.data\n\nSo, some of you might be wondering why go through all the trouble to convert our data to a tf.data implementation when we can just use the built-in tensorflow/keras functions and avoid all this hassle and complications? Here are two main reasons:\n\n1. Python executes code eagerly (sequentially) by default, as do pandas and numpy. This means that the ability to run code in parallel with a GPU (multi-processing) is not utilized fully if you don't specify. TF.data maximizes parallel code execution by converting your code into Tensorflow graphs and tensors (in a nutshell), so it runs faster.\n\n2. Like I mentioned earlier, tf.data offers much more functionality, flexibility and customization for your code than in-built functions. In our case, you can manipulate images much more intricately with tf.data than you can with in-built functions. We are able to implement CutMix(up) exactly because of this. \n\nHere are some links in case you want to reaed up more:\n- https://www.tensorflow.org/guide/data_performance\n- https://stackoverflow.com/questions/54894799/why-should-i-use-tf-data\n\n*Disclaimer: I'm not making full use of tf.data functionality in this notebook. The problem with many notebooks utilizing tf.data is that they are too complex and not beginner-friendly. I aim to change that in this notebook and offer a introductory example of tf.data for further exploration. That is to say, tf.data can be implemented much more efficiently and as a result; run faster. This notebook only has barebones functionality to accomodate the necessary image augmentations.*"},{"metadata":{},"cell_type":"markdown","source":"I'll be using EfficientNetB0 and a smaller image size for demonstration purposes. Feel free to scsale up both the model and the image size with additional tweaks for better results."},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 16\nimage_size = 224\ninput_shape = (image_size, image_size, 3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"skf = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=seed_value)\nfor train_index, val_index in skf.split(df_train[\"image_id\"], df_train[\"label\"]):\n    train_data = df_train.loc[train_index]\n    val_data = df_train.loc[val_index]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_image_and_label_from_path(image_path, label):\n    img = tf.io.read_file(image_path)\n    img = tf.image.decode_jpeg(img, channels=3)\n    return img, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![](https://storage.googleapis.com/jalammar-ml/tf.data/images/tf.data-pipeline-1.png)\n\nThe first step is to read in data from our csv file and images, convert them into tf.tensors with the features and labels assigned. You can think of tensors as small data blocks, whose sizes we can specify with \"batch size\"."},{"metadata":{"trusted":true},"cell_type":"code","source":"training_data = tf.data.Dataset.from_tensor_slices((train_data[\"filepath\"].values, train_data[\"label\"].values))\nvalidation_data = tf.data.Dataset.from_tensor_slices((val_data[\"filepath\"].values, val_data[\"label\"].values))\n\ntraining_data = training_data.map(load_image_and_label_from_path, num_parallel_calls=AUTOTUNE)\nvalidation_data = validation_data.map(load_image_and_label_from_path, num_parallel_calls=AUTOTUNE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image augmentation\nNext, we still specify our image augmentations in Albumentations and map them to our tf.tensors."},{"metadata":{"trusted":true},"cell_type":"code","source":"def augment_train_data(train_ds):\n    transforms = A.Compose([\n            A.RandomResizedCrop(image_size, image_size),\n            A.Transpose(p=0.5),\n            A.HorizontalFlip(p=0.5),\n            A.VerticalFlip(p=0.5),\n            A.ShiftScaleRotate(p=0.5),\n            A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n            A.RandomBrightnessContrast(brightness_limit=(-0.1, 0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n            A.CoarseDropout(p=0.5),\n            A.Cutout(p=0.5),\n            ], p=1)\n    \n    def aug_fn(image):\n        data = {\"image\":image}\n        aug_data = transforms(**data)\n        aug_img = aug_data[\"image\"]\n        aug_img = tf.cast(aug_img, tf.float32)\n        return aug_img\n\n    def process_data(image, label):\n        aug_img = tf.numpy_function(func=aug_fn, inp=[image], Tout=tf.float32)\n        return aug_img, label\n    \n    def set_shapes(img, label, img_shape=(image_size,image_size,3)):\n        img.set_shape(img_shape)\n        label.set_shape([])\n        return img, label\n    \n    ds_alb = train_ds.map(partial(process_data), num_parallel_calls=AUTOTUNE).prefetch(AUTOTUNE)\n    ds_alb = ds_alb.map(set_shapes, num_parallel_calls=AUTOTUNE)\n    ds_alb = ds_alb.repeat()\n    ds_alb = ds_alb.batch(batch_size)\n    return ds_alb","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def augment_val_data(val_ds):\n    transforms = A.Compose([\n                A.CenterCrop(image_size, image_size),\n                ], p=1)\n    \n    def aug_fn(image):\n        data = {\"image\":image}\n        aug_data = transforms(**data)\n        aug_img = aug_data[\"image\"]\n        aug_img = tf.cast(aug_img, tf.float32)\n        return aug_img\n\n    def process_data(image, label):\n        aug_img = tf.numpy_function(func=aug_fn, inp=[image], Tout=tf.float32)\n        return aug_img, label\n    \n    def set_shapes(img, label, img_shape=(image_size, image_size,3)):\n        img.set_shape(img_shape)\n        label.set_shape([])\n        return img, label\n    \n    ds_alb = val_ds.map(partial(process_data), num_parallel_calls=AUTOTUNE).prefetch(AUTOTUNE)\n    ds_alb = ds_alb.map(set_shapes, num_parallel_calls=AUTOTUNE).batch(batch_size)\n    return ds_alb","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_alb = augment_train_data(training_data)\nval_alb = augment_val_data(validation_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def view_image(ds):\n    image, label = next(iter(ds)) # extract 1 batch from the dataset\n    image = image.numpy()/255\n    label = label.numpy()\n\n    fig = plt.figure(figsize=(22, 22))\n    for i in range(batch_size):\n        ax = fig.add_subplot(4, 4, i+1, xticks=[], yticks=[])\n        ax.imshow(image[i])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### We can view the images post-augmentation with our \"view image\" function."},{"metadata":{"trusted":true},"cell_type":"code","source":"view_image(train_alb)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Check that validation set remains unchanged!"},{"metadata":{"trusted":true},"cell_type":"code","source":"view_image(val_alb)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# CutMix(up)\n\nGreat! The primary augmentations through Albumentations are done. Time to implement CutMix(up) on top of them for better generalization and performance of our model."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"IMAGE_SIZE = [image_size, image_size]\nAUG_BATCH = batch_size","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def cutmix(image, label, PROBABILITY=0.5):\n    # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n    # output - a batch of images with cutmix applied\n    DIM = IMAGE_SIZE[0]\n    CLASSES = 5\n    \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n        # DO CUTMIX WITH PROBABILITY DEFINED ABOVE\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.int32)\n        # CHOOSE RANDOM IMAGE TO CUTMIX WITH\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n        # CHOOSE RANDOM LOCATION\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        b = tf.random.uniform([],0,1) # this is beta dist with alpha=1.0\n        WIDTH = tf.cast( DIM * tf.math.sqrt(1-b),tf.int32) * P\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n        # MAKE CUTMIX IMAGE\n        one = image[j,ya:yb,0:xa,:]\n        two = image[k,ya:yb,xa:xb,:]\n        three = image[j,ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        img = tf.concat([image[j,0:ya,:,:],middle,image[j,yb:DIM,:,:]],axis=0)\n        imgs.append(img)\n        # MAKE CUTMIX LABEL\n        a = tf.cast(WIDTH*WIDTH/DIM/DIM,tf.float32)\n        if len(label.shape)==1:\n            lab1 = tf.one_hot(label[j],CLASSES)\n            lab2 = tf.one_hot(label[k],CLASSES)\n        else:\n            lab1 = label[j,]\n            lab2 = label[k,]\n        labs.append((1-a)*lab1 + a*lab2)\n            \n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image2,label2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def mixup(image, label, PROBABILITY=0.5):\n    # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n    # output - a batch of images with mixup applied\n    DIM = IMAGE_SIZE[0]\n    CLASSES = 5\n    \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n        # DO MIXUP WITH PROBABILITY DEFINED ABOVE\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.float32)\n        # CHOOSE RANDOM\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n        a = tf.random.uniform([],0,1)*P # this is beta dist with alpha=1.0\n        # MAKE MIXUP IMAGE\n        img1 = image[j,]\n        img2 = image[k,]\n        imgs.append((1-a)*img1 + a*img2)\n        # MAKE CUTMIX LABEL\n        if len(label.shape)==1:\n            lab1 = tf.one_hot(label[j],CLASSES)\n            lab2 = tf.one_hot(label[k],CLASSES)\n        else:\n            lab1 = label[j,]\n            lab2 = label[k,]\n        labs.append((1-a)*lab1 + a*lab2)\n            \n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image2,label2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def transform(image1,label):\n    # THIS FUNCTION APPLIES BOTH CUTMIX AND MIXUP\n    DIM = IMAGE_SIZE[0]\n    CLASSES = 5\n    SWITCH = 0.5\n    CUTMIX_PROB = 0.66\n    MIXUP_PROB = 0.66\n    # FOR SWITCH PERCENT OF TIME WE DO CUTMIX AND (1-SWITCH) WE DO MIXUP\n\n    image2, label2 = cutmix(image1, label, CUTMIX_PROB)\n    image3, label3 = mixup(image1, label, MIXUP_PROB)\n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n        P = tf.cast( tf.random.uniform([],0,1)<=SWITCH, tf.float32)\n        imgs.append(P*image2[j,]+(1-P)*image3[j,])\n        labs.append(P*label2[j,]+(1-P)*label3[j,])\n\n    image4 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label4 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image4,label4","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def onehot(image,label):\n    CLASSES = 5\n    return image,tf.one_hot(label,CLASSES)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def optimize_data(train_alb, val_alb):\n\n    def configure_for_train(ds):\n        ds = ds.map(transform, num_parallel_calls=AUTOTUNE)\n        ds = ds.unbatch()\n        ds = ds.shuffle(buffer_size=1024)\n        ds = ds.batch(batch_size)\n        ds = ds.prefetch(buffer_size=AUTOTUNE)\n        return ds\n    \n    def configure_for_val(ds):\n        ds = ds.map(onehot, num_parallel_calls=AUTOTUNE)\n        ds = ds.prefetch(buffer_size=AUTOTUNE)\n        return ds\n\n    train_ds = configure_for_train(train_alb)\n    val_ds = configure_for_val(val_alb)\n    \n    return train_ds, val_ds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ds, val_ds = optimize_data(train_alb, val_alb)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Let's see how our images look like after applying CutMix and Mixup:"},{"metadata":{"trusted":true},"cell_type":"code","source":"view_image(train_ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Again, make sure the validation set is untouched."},{"metadata":{"trusted":true},"cell_type":"code","source":"view_image(val_ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model creation & training"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_model():\n    # Model creation\n    base_model = EfficientNetB0(include_top=False, weights=\"imagenet\", input_shape=input_shape)\n    \n    # Rebuild top\n    inputs = Input(shape=input_shape)\n    aug_model = base_model(inputs)\n    pooling = GlobalAveragePooling2D()(aug_model)\n    dropout = Dropout(0.5)(pooling)\n    outputs = Dense(5, activation=\"softmax\", dtype='float32')(dropout)\n\n    # Compile\n    model = Model(inputs=inputs, outputs=outputs)\n    optimizer = tfa.optimizers.RectifiedAdam()\n    loss = tf.keras.losses.CategoricalCrossentropy(label_smoothing=0.01)\n\n    model.compile(optimizer=optimizer, loss=loss, metrics=[\"accuracy\"])\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is where tf.data works its magic. Consider the following image:\n### Naive pipeline\n![](https://dominikschmidt.xyz/tensorflow-data-pipeline/assets/feed_dict_pipeline.png)\n\nThis is the typical workflow of a naive data pipeline, there is always some idle time and overhead due to the inefficiency of sequential execution.\n\nIn contrast, consider:\n### tf.data pipeline\n![](https://dominikschmidt.xyz/tensorflow-data-pipeline/assets/tf_data_pipeline.png)\n\nThis is the workflow of a tf.data pipeline. As you can see, downtime and waiting around is minimized while processing is maximized through parallel execution."},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_val_split(train_data, val_data):\n    training_data = tf.data.Dataset.from_tensor_slices((train_data[\"filepath\"].values, train_data[\"label\"].values))\n    validation_data = tf.data.Dataset.from_tensor_slices((val_data[\"filepath\"].values, val_data[\"label\"].values))\n\n    training_data = training_data.map(load_image_and_label_from_path, num_parallel_calls=AUTOTUNE)\n    validation_data = validation_data.map(load_image_and_label_from_path, num_parallel_calls=AUTOTUNE)\n    \n    return (training_data, validation_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = 10\ntotal_steps = (int(len(df_train)*0.8/batch_size)+1)\nfold_number = 4\nn_splits = 5\ntrain_list = []\nval_list = []\n\nskf = StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=seed_value)\nfor train_index, val_index in skf.split(df_train[\"image_id\"], df_train[\"label\"]):\n    train_list.append(train_index)\n    val_list.append(val_index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(n_splits-fold_number): \n    tf.keras.backend.clear_session()\n    gc.collect()\n    train_set = df_train.loc[train_list[fold_number]]\n    val_set = df_train.loc[val_list[fold_number]]\n    train_data, val_data = train_val_split(train_set, val_set)\n    train_alb = augment_train_data(train_data)\n    val_alb = augment_val_data(val_data)\n    train_final, val_final = optimize_data(train_alb, val_alb)\n\n    model = create_model()\n    print(\"Training fold no.: \" + str(fold_number+1))\n\n    model_name = \"effnetb3 \"\n    fold_name = \"fold.h5\"\n    filepath = model_name + str(fold_number+1) + fold_name\n    callbacks = [ReduceLROnPlateau(monitor='val_loss', patience=1, verbose=1, factor=0.2),\n                 EarlyStopping(monitor='val_loss', patience=2, verbose=1, restore_best_weights=True),\n                 ModelCheckpoint(filepath=filepath, monitor='val_accuracy', save_best_only=True)]\n\n    history = model.fit(train_final, steps_per_epoch=total_steps, epochs=epochs, validation_data=val_final, callbacks=callbacks)\n    fold_number += 1\n    if fold_number == n_splits:\n        print(\"Training finished!\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And we're done! Hope you enjoyed this walkthrough and it motivates you to learn and try using tf.data. Please upvote this notebook if you liked it. It motivates me to continue producing high-quality notebooks. Thanks and stay tuned!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}