{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Overview\n\nIn this notebook, we illustrate some commonly used augmentations for images in various libraries, and perform inference with both TensorFlow and PyTorch pipelines.\n\nA key augmentation that we will like to illustrate is the insect augmentation, or rather an augmentation that performs an overlay of small artifacts onto the images with a certain probability. A variant first surfaced in the Melanoma Detection competition recently, created by Roman. I've also adapted it in a recent Global Wheat Detection competition. "},{"metadata":{},"cell_type":"markdown","source":"## Version Notes\n\n- version 1 to 13: uses images from Casava leaf classification competition\n- version 14 onwards: uses images from Plant Pathology 2021, Herbarium 2021, and iWildcam 2021 competitions"},{"metadata":{"_kg_hide-output":true,"execution":{"iopub.execute_input":"2020-12-27T04:06:56.338289Z","iopub.status.busy":"2020-12-27T04:06:56.337578Z","iopub.status.idle":"2020-12-27T04:07:07.85439Z","shell.execute_reply":"2020-12-27T04:07:07.853469Z"},"papermill":{"duration":11.541639,"end_time":"2020-12-27T04:07:07.854521","exception":false,"start_time":"2020-12-27T04:06:56.312882","status":"completed"},"tags":[],"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#!pip install imutils\nimport sys\nsys.path.append('../input/imutils/imutils-0.5.3')","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.020598,"end_time":"2020-12-27T04:07:07.896698","exception":false,"start_time":"2020-12-27T04:07:07.8761","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Imports"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2020-12-27T04:07:07.946375Z","iopub.status.busy":"2020-12-27T04:07:07.945105Z","iopub.status.idle":"2020-12-27T04:07:16.771431Z","shell.execute_reply":"2020-12-27T04:07:16.770182Z"},"papermill":{"duration":8.853842,"end_time":"2020-12-27T04:07:16.771559","exception":false,"start_time":"2020-12-27T04:07:07.917717","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import os\nimport random\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport matplotlib.pyplot as plt\n\nimport torch\nimport torchvision\nfrom torchvision import transforms\nimport tensorflow as tf\nimport albumentations as A\nimport imgaug.augmenters as iaa\nfrom imgaug import parameters as iap","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Configs"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.execute_input":"2020-12-27T04:07:16.818768Z","iopub.status.busy":"2020-12-27T04:07:16.81782Z","iopub.status.idle":"2020-12-27T04:07:16.822742Z","shell.execute_reply":"2020-12-27T04:07:16.822162Z"},"papermill":{"duration":0.03028,"end_time":"2020-12-27T04:07:16.822873","exception":false,"start_time":"2020-12-27T04:07:16.792593","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"Cassava = False\nFGVC8_HERB = False # herbarium 2021\nFGVC8_CAM = False # iwildcam2021\nFGVC8_PLANT = True # plant pathology 2021\n\nif Cassava:\n    DIR = \"../input/cassava-leaf-disease-classification/train_images\"\n    image_path = f'{DIR}/100042118.jpg'\nif FGVC8_PLANT:\n    DIR = \"../input/plant-pathology-2021-fgvc8/train_images\"\n    image_path = f'{DIR}/800113bb65efe69e.jpg'\nif FGVC8_HERB:\n    DIR = \"../input/herbarium-2021-fgvc8/train/images/000/00\"\n    image_path = f'{DIR}/1360648.jpg'\nif FGVC8_CAM:\n    DIR = \"../input/iwildcam2021-fgvc8/train\"\n    image_path = f'{DIR}/86760c00-21bc-11ea-a13a-137349068a90.jpg'","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.022514,"end_time":"2020-12-27T04:07:16.866096","exception":false,"start_time":"2020-12-27T04:07:16.843582","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Read Sample Image without Augmentations"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T04:07:16.913462Z","iopub.status.busy":"2020-12-27T04:07:16.912759Z","iopub.status.idle":"2020-12-27T04:07:17.270071Z","shell.execute_reply":"2020-12-27T04:07:17.269543Z"},"papermill":{"duration":0.382762,"end_time":"2020-12-27T04:07:17.270201","exception":false,"start_time":"2020-12-27T04:07:16.887439","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"chosen_image = cv2.imread(image_path)\nplt.imshow(chosen_image)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.025289,"end_time":"2020-12-27T04:07:17.321478","exception":false,"start_time":"2020-12-27T04:07:17.296189","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Albumentations Augmentations\n\n- Albumentations part adapted from my good friend Hongnan's notebbok in the Global Wheat Detection competition (https://www.kaggle.com/reighns/augmentations-data-cleaning-and-bounding-boxes#Bounding-Boxes-with-Albumentations)\n- Added more augmentations which may be useful\n- Added TensorFlow and Torchvision versions of the augmentations"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T04:07:17.387251Z","iopub.status.busy":"2020-12-27T04:07:17.386285Z","iopub.status.idle":"2020-12-27T04:07:17.389197Z","shell.execute_reply":"2020-12-27T04:07:17.388684Z"},"papermill":{"duration":0.042418,"end_time":"2020-12-27T04:07:17.389339","exception":false,"start_time":"2020-12-27T04:07:17.346921","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"albumentation_list = [A.RandomSunFlare(p=1), \n                      A.RandomFog(p=1), \n                      A.RandomBrightness(p=1),\n                      A.RandomCrop(p=1,height = 512, width = 512), \n                      A.Rotate(p=1, limit=90),\n                      A.RGBShift(p=1), \n                      A.RandomSnow(p=1),\n                      A.HorizontalFlip(p=1), \n                      A.VerticalFlip(p=1), \n                      A.RandomContrast(limit = 0.5,p = 1),\n                      A.HueSaturationValue(p=1,hue_shift_limit=20, sat_shift_limit=30, val_shift_limit=50),\n                      A.Cutout(p=1),\n                      A.Transpose(p=1), \n                      A.JpegCompression(p=1),\n                      A.CoarseDropout(p=1),\n                      A.IAAAdditiveGaussianNoise(loc=0, scale=(2.5500000000000003, 12.75), per_channel=False, p=1),\n                      A.IAAAffine(scale=1.0, translate_percent=None, translate_px=None, rotate=0.0, shear=0.0, order=1, cval=0, mode='reflect', p=1),\n                      A.IAAAffine(rotate=90., p=1),\n                      A.IAAAffine(rotate=180., p=1)]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-12-27T04:07:17.455011Z","iopub.status.busy":"2020-12-27T04:07:17.454306Z","iopub.status.idle":"2020-12-27T04:07:21.74616Z","shell.execute_reply":"2020-12-27T04:07:21.746724Z"},"papermill":{"duration":4.331741,"end_time":"2020-12-27T04:07:21.746865","exception":false,"start_time":"2020-12-27T04:07:17.415124","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"img_matrix_list = []\nbboxes_list = []\nfor aug_type in albumentation_list:\n    img = aug_type(image = chosen_image)['image']\n    img_matrix_list.append(img)\n\nimg_matrix_list.insert(0,chosen_image)    \n\ntitles_list = [\"Original\",\"RandomSunFlare\",\"RandomFog\",\"RandomBrightness\",\n               \"RandomCrop\",\"Rotate\", \"RGBShift\", \"RandomSnow\",\"HorizontalFlip\", \"VerticalFlip\", \"RandomContrast\",\"HSV\",\n               \"Cutout\",\"Transpose\",\"JpegCompression\",\"CoarseDropout\",\"IAAAdditiveGaussianNoise\",\"IAAAffine\",\"IAAAffineRotate90\",\"IAAAffineRotate180\"]\n\ndef plot_multiple_img(img_matrix_list, title_list, ncols, nrows=5,  main_title=\"\"):\n    fig, myaxes = plt.subplots(figsize=(20, 15), nrows=nrows, ncols=ncols, squeeze=False)\n    fig.suptitle(main_title, fontsize = 30)\n    fig.subplots_adjust(wspace=0.3)\n    fig.subplots_adjust(hspace=0.3)\n    for i, (img, title) in enumerate(zip(img_matrix_list, title_list)):\n        myaxes[i // ncols][i % ncols].imshow(img)\n        myaxes[i // ncols][i % ncols].set_title(title, fontsize=15)\n    plt.show()\n    \nplot_multiple_img(img_matrix_list, titles_list, ncols = 4,main_title=\"Different Types of Augmentations with Albumentations\")","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.05159,"end_time":"2020-12-27T04:07:21.848924","exception":false,"start_time":"2020-12-27T04:07:21.797334","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# OpenCV-Based Custom Augmentations\n\n# Insect (Bee) Augmentation\n\n- Originally, I implemented this for the recent Global Wheat Detection competition at https://www.kaggle.com/khoongweihao/insect-augmentation-with-efficientdet-d6, inspired by Roman's hair augmentation from the recent Melanoma competition\n- Motivations for this approach:\n    - the intuition was to mimic as close to as possible, actual scenarios in the wild where insects are present and at times visible in images similar to those in this and similar competitions\n    - insects can be found roaming on leaves in reality\n    - bees were used as an example. Other small artifacts such as raindrops overlayed with the images or other insect species should suffice. Main idea is to mimic what we can observe in reality, in the form of augmentations"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T04:07:21.954203Z","iopub.status.busy":"2020-12-27T04:07:21.95304Z","iopub.status.idle":"2020-12-27T04:07:21.971876Z","shell.execute_reply":"2020-12-27T04:07:21.972442Z"},"papermill":{"duration":0.07362,"end_time":"2020-12-27T04:07:21.972588","exception":false,"start_time":"2020-12-27T04:07:21.898968","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def insect_augmentation(image, n_insects=2, dark_insect=False, p=0.5, insects_folder='../input/bee-augmentation'):\n    aug_prob = random.random()\n    if aug_prob < p:\n        height, width, _ = image.shape  # target image width and height\n        insects_images = [im for im in os.listdir(insects_folder) if 'bee' in im]\n        img_shape = image.shape\n\n        for _ in range(n_insects):\n            insect = cv2.cvtColor(cv2.imread(os.path.join(insects_folder, random.choice(insects_images))), cv2.COLOR_BGR2RGB)\n            insect = cv2.flip(insect, random.choice([-1, 0, 1]))\n            insect = cv2.rotate(insect, random.choice([0, 1, 2]))\n            insect = cv2.resize(insect, (width, height))\n\n            h_height, h_width, _ = insect.shape  # insect image width and height\n            roi_ho = random.randint(0, image.shape[0] - insect.shape[0])\n            roi_wo = random.randint(0, image.shape[1] - insect.shape[1])\n            roi = image[roi_ho:roi_ho + h_height, roi_wo:roi_wo + h_width]\n\n            # Creating a mask and inverse mask \n            img2gray = cv2.cvtColor(insect, cv2.COLOR_BGR2GRAY)\n            ret, mask = cv2.threshold(img2gray, 10, 255, cv2.THRESH_BINARY)\n            #mask_inv = cv2.cvtColor(cv2.bitwise_not(mask),cv2.COLOR_BGR2GRAY)\n            mask_inv = cv2.bitwise_not(mask)\n\n            # Now black-out the area of insect in ROI\n            img_bg = cv2.bitwise_and(roi, roi, mask=mask_inv)\n\n            # Take only region of insect from insect image.\n            if dark_insect:\n                img_bg = cv2.bitwise_and(roi, roi, mask=mask_inv)\n                insect_fg = cv2.bitwise_and(img_bg, img_bg, mask=mask)\n            else:\n                insect_fg = cv2.bitwise_and(insect, insect, mask=mask)\n\n            # Put insect in ROI and modify the target image\n            dst = cv2.add(img_bg, insect_fg, dtype=cv2.CV_64F)\n\n            image[roi_ho:roi_ho + h_height, roi_wo:roi_wo + h_width] = dst\n\n    return image","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.049402,"end_time":"2020-12-27T04:07:22.071953","exception":false,"start_time":"2020-12-27T04:07:22.022551","status":"completed"},"tags":[]},"cell_type":"markdown","source":"### Original Bees"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T04:07:22.175202Z","iopub.status.busy":"2020-12-27T04:07:22.174172Z","iopub.status.idle":"2020-12-27T04:07:22.563615Z","shell.execute_reply":"2020-12-27T04:07:22.56292Z"},"papermill":{"duration":0.442252,"end_time":"2020-12-27T04:07:22.563798","exception":false,"start_time":"2020-12-27T04:07:22.121546","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"chosen_image = cv2.imread(image_path)\naug_image = insect_augmentation(chosen_image, n_insects=2, dark_insect=False, p=1.0)\nplt.imshow(aug_image)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.064964,"end_time":"2020-12-27T04:07:22.709164","exception":false,"start_time":"2020-12-27T04:07:22.6442","status":"completed"},"tags":[]},"cell_type":"markdown","source":"### Dark Bees"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T04:07:22.824587Z","iopub.status.busy":"2020-12-27T04:07:22.823857Z","iopub.status.idle":"2020-12-27T04:07:23.158259Z","shell.execute_reply":"2020-12-27T04:07:23.158785Z"},"papermill":{"duration":0.394708,"end_time":"2020-12-27T04:07:23.15893","exception":false,"start_time":"2020-12-27T04:07:22.764222","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"chosen_image = cv2.imread(image_path)\naug_image = insect_augmentation(chosen_image, n_insects=2, dark_insect=True, p=1.0)\nplt.imshow(aug_image)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.058382,"end_time":"2020-12-27T04:07:23.276648","exception":false,"start_time":"2020-12-27T04:07:23.218266","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# `imgaug` Based Augmentations"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T04:07:23.397689Z","iopub.status.busy":"2020-12-27T04:07:23.396953Z","iopub.status.idle":"2020-12-27T04:07:23.413535Z","shell.execute_reply":"2020-12-27T04:07:23.412851Z"},"papermill":{"duration":0.078432,"end_time":"2020-12-27T04:07:23.413654","exception":false,"start_time":"2020-12-27T04:07:23.335222","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"ia_trans_list = [iaa.blend.BlendAlpha(factor=(0.2, 0.8),\n                                      foreground=iaa.Affine(rotate=(-30, 30)),\n                                      per_channel=True),\n                 iaa.Fliplr(1.),\n                 iaa.Flipud(1.),\n                 iaa.SimplexNoiseAlpha(iaa.Multiply(iap.Choice([0.5, 1.5]), per_channel=True)),\n                 iaa.Crop(percent=(0., 0.3)),\n                ]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-12-27T04:07:23.541104Z","iopub.status.busy":"2020-12-27T04:07:23.54044Z","iopub.status.idle":"2020-12-27T04:07:27.638646Z","shell.execute_reply":"2020-12-27T04:07:27.639206Z"},"papermill":{"duration":4.166558,"end_time":"2020-12-27T04:07:27.639363","exception":false,"start_time":"2020-12-27T04:07:23.472805","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"img_matrix_list = []\nbboxes_list = []\nfor aug_type in ia_trans_list:\n    # convert to tensor\n    chosen_image = cv2.imread(image_path)\n    iaa_seq = iaa.Sequential([aug_type])\n    trans_img = iaa_seq.augment_images(chosen_image)\n    img_matrix_list.append(trans_img)\n\nimg_matrix_list.insert(0, chosen_image)    \n\ntitles_list = [\"Original\",\"Ghost Aug\",\"Flip Left Right\",\"Flip Up Down\",\"SimplexNoiseAlpha\", \"Crop\"]\n\nplot_multiple_img(img_matrix_list, titles_list, ncols = 3, nrows=2, main_title=\"Different Types of Augmentations with Albumentations\")","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.086359,"end_time":"2020-12-27T04:07:27.811143","exception":false,"start_time":"2020-12-27T04:07:27.724784","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# PyTorch-Based Augmentations (Torchvision)"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T04:07:27.988423Z","iopub.status.busy":"2020-12-27T04:07:27.987689Z","iopub.status.idle":"2020-12-27T04:07:27.990809Z","shell.execute_reply":"2020-12-27T04:07:27.990065Z"},"papermill":{"duration":0.094323,"end_time":"2020-12-27T04:07:27.990926","exception":false,"start_time":"2020-12-27T04:07:27.896603","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"torch_trans_list = [transforms.CenterCrop((178, 178)),\n                    transforms.Resize(128),\n                    transforms.RandomRotation(45),\n                    transforms.RandomAffine(35),\n                    transforms.RandomCrop(128),\n                    transforms.RandomHorizontalFlip(p=1),\n                    transforms.RandomPerspective(p=1),\n                    transforms.RandomVerticalFlip(p=1)]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-12-27T04:07:28.167133Z","iopub.status.busy":"2020-12-27T04:07:28.166458Z","iopub.status.idle":"2020-12-27T04:07:30.715911Z","shell.execute_reply":"2020-12-27T04:07:30.716519Z"},"papermill":{"duration":2.642407,"end_time":"2020-12-27T04:07:30.716668","exception":false,"start_time":"2020-12-27T04:07:28.074261","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"img_matrix_list = []\nbboxes_list = []\nfor aug_type in torch_trans_list:\n    # convert to tensor\n    chosen_image = cv2.imread(image_path)\n    chosen_tensor = transforms.Compose([transforms.ToTensor()])(chosen_image)\n    chosen_tensor = transforms.Compose([aug_type])(chosen_tensor)\n    trans_img = transforms.ToPILImage()(chosen_tensor)\n    img_matrix_list.append(trans_img)\n\nimg_matrix_list.insert(0, chosen_image)    \n\ntitles_list = [\"Original\",\"CenterCrop\",\"Resize\",\"RandomRotation\",\"RandomAffine\",\"RandomCrop\",\"RandomHorizontalFlip\",\"RandomPerspective\",\n               \"RandomVerticalFlip\"]\n\nplot_multiple_img(img_matrix_list, titles_list, ncols = 3, nrows=3, main_title=\"Different Types of Augmentations with Albumentations\")","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.118618,"end_time":"2020-12-27T04:07:30.95481","exception":false,"start_time":"2020-12-27T04:07:30.836192","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# TensorFlow-Based Augmentations"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T04:07:31.203845Z","iopub.status.busy":"2020-12-27T04:07:31.20303Z","iopub.status.idle":"2020-12-27T04:07:31.450489Z","shell.execute_reply":"2020-12-27T04:07:31.449355Z"},"papermill":{"duration":0.375892,"end_time":"2020-12-27T04:07:31.450625","exception":false,"start_time":"2020-12-27T04:07:31.074733","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"chosen_image = cv2.imread(image_path)\n\ntf_trans_list = [\n    tf.image.rot90(chosen_image, k=1), # 90 degrees counter-clockwise\n    tf.image.rot90(chosen_image, k=2), # 180 degrees counter-clockwise\n    tf.image.rot90(chosen_image, k=3), # 270 degrees counter-clockwise\n    tf.image.random_brightness(chosen_image, 0.5), \n    tf.image.random_contrast(chosen_image, 0.2, 0.5), \n    tf.image.random_flip_left_right(chosen_image, seed=42),\n    tf.image.random_flip_up_down(chosen_image, seed=42),\n    tf.image.random_hue(chosen_image, 0.5),\n    tf.image.random_jpeg_quality(chosen_image, 35, 50), \n    tf.image.random_saturation(chosen_image, 5, 10), \n    tf.image.transpose(chosen_image),\n]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-12-27T04:07:31.703777Z","iopub.status.busy":"2020-12-27T04:07:31.702923Z","iopub.status.idle":"2020-12-27T04:07:34.261534Z","shell.execute_reply":"2020-12-27T04:07:34.262064Z"},"papermill":{"duration":2.692804,"end_time":"2020-12-27T04:07:34.262212","exception":false,"start_time":"2020-12-27T04:07:31.569408","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"img_matrix_list = []\nbboxes_list = []\nfor aug_image in tf_trans_list:\n    img_matrix_list.append(aug_image)\n\nimg_matrix_list.insert(0, chosen_image)    \n\ntitles_list = [\"Original\",\"Rotate90\",\"Rotate180\",\"Rotate270\",\"RandomBrightness\",\"RandomContrast\",\"RandomLeftRightFlip\",\"RandomUpDownFlip\",\n               \"RandomHue\",\"RandomJPEGQuality\",\"RandomSaturation\",\"Transpose\"]\n\nplot_multiple_img(img_matrix_list, titles_list, ncols = 3, nrows=4, main_title=\"Different Types of Augmentations with Albumentations\")","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.141103,"end_time":"2020-12-27T04:07:34.550556","exception":false,"start_time":"2020-12-27T04:07:34.409453","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Model Pipeline with Insect (Bee) Augmentation"},{"metadata":{},"cell_type":"markdown","source":"## TensorFlow-Keras\n\n- modified from @dimitreoliveira's inference notebook to perform inference for weights obtained with insect augmentation: https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tpu-v2-pods-inference"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!pip install --quiet /kaggle/input/kerasapplications\n!pip install --quiet /kaggle/input/efficientnet-git","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.141277,"end_time":"2020-12-27T04:07:34.834406","exception":false,"start_time":"2020-12-27T04:07:34.693129","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import math, os, re, warnings, random, glob\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.backend as K\nfrom tensorflow.keras import Sequential, Model\nimport efficientnet.tfkeras as efn\n\ndef seed_everything(seed=0):\n    random.seed(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    os.environ['TF_DETERMINISTIC_OPS'] = '1'\n\nseed = 0\nseed_everything(seed)\nwarnings.filterwarnings('ignore')\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print(f'Running on TPU {tpu.master()}')\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')\n\nBATCH_SIZE = 16 * REPLICAS\nHEIGHT = 512\nWIDTH = 512 \nCHANNELS = 3\nN_CLASSES = 5\nTTA_STEPS = 8 # Do TTA if > 0 \n\ndef data_augment(image, label):\n    p_spatial = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_pixel_1 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_pixel_2 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_pixel_3 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n            \n    # Flips\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n    if p_spatial > .75:\n        image = tf.image.transpose(image)\n        \n    # Rotates\n    if p_rotate > .75:\n        image = tf.image.rot90(image, k=3) # rotate 270º\n    elif p_rotate > .5:\n        image = tf.image.rot90(image, k=2) # rotate 180º\n    elif p_rotate > .25:\n        image = tf.image.rot90(image, k=1) # rotate 90º\n        \n    # Pixel-level transforms\n    if p_pixel_1 >= .4:\n        image = tf.image.random_saturation(image, lower=.7, upper=1.3)\n    if p_pixel_2 >= .4:\n        image = tf.image.random_contrast(image, lower=.8, upper=1.2)\n    if p_pixel_3 >= .4:\n        image = tf.image.random_brightness(image, max_delta=.1)\n        \n    # Crops\n    if p_crop > .7:\n        if p_crop > .9:\n            image = tf.image.central_crop(image, central_fraction=.7)\n        elif p_crop > .8:\n            image = tf.image.central_crop(image, central_fraction=.8)\n        else:\n            image = tf.image.central_crop(image, central_fraction=.9)\n    elif p_crop > .4:\n        crop_size = tf.random.uniform([], int(HEIGHT*.8), HEIGHT, dtype=tf.int32)\n        image = tf.image.random_crop(image, size=[crop_size, crop_size, CHANNELS])\n    \n    return image, label\n\ndef get_name(file_path):\n    parts = tf.strings.split(file_path, os.path.sep)\n    name = parts[-1]\n    return name\n\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    \n    return image\n\ndef center_crop(image):\n    image = tf.reshape(image, [600, 800, CHANNELS]) # Original shape\n    \n    h, w = image.shape[0], image.shape[1]\n    if h > w:\n        image = tf.image.crop_to_bounding_box(image, (h - w) // 2, 0, w, w)\n    else:\n        image = tf.image.crop_to_bounding_box(image, 0, (w - h) // 2, h, h)\n        \n    image = tf.image.resize(image, [HEIGHT, WIDTH]) # Expected shape\n    return image\n\ndef resize_image(image, label):\n    image = tf.image.resize(image, [HEIGHT, WIDTH])\n    image = tf.reshape(image, [HEIGHT, WIDTH, CHANNELS])\n    return image, label\n\ndef process_path(file_path):\n    name = get_name(file_path)\n    img = tf.io.read_file(file_path)\n    img = decode_image(img)\n    return img, name\n\ndef get_dataset(files_path, shuffled=False, tta=False, extension='jpg'):\n    dataset = tf.data.Dataset.list_files(f'{files_path}*{extension}', shuffle=shuffled)\n    dataset = dataset.map(process_path, num_parallel_calls=AUTO)\n    if tta:\n        dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.map(resize_image, num_parallel_calls=AUTO)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\ndatabase_base_path = '/kaggle/input/cassava-leaf-disease-classification/'\nsubmission = pd.read_csv(f'{database_base_path}sample_submission.csv')\ndisplay(submission.head())\n\nTEST_FILENAMES = tf.io.gfile.glob(f'{database_base_path}test_tfrecords/ld_test*.tfrec')\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nprint(f'GCS: test: {NUM_TEST_IMAGES}')\n\nmodel_path_list = glob.glob('../input/casava-efficientnet-tpu-weights/*.h5')\nmodel_path_list.sort()\n\nprint('Models to predict:')\nprint(*model_path_list, sep='\\n')\n\ndef model_fn(input_shape, N_CLASSES):\n    inputs = L.Input(shape=input_shape, name='input_image')\n    base_model = efn.EfficientNetB4(input_tensor=inputs, \n                                    include_top=False, \n                                    weights=None, \n                                    pooling='avg')\n\n    x = L.Dropout(.5)(base_model.output)\n    output = L.Dense(N_CLASSES, activation='softmax', name='output')(x)\n    model = Model(inputs=inputs, outputs=output)\n\n    return model\n\nwith strategy.scope():\n    model = model_fn((None, None, CHANNELS), N_CLASSES)\n    \nfiles_path = f'{database_base_path}test_images/'\ntest_size = len(os.listdir(files_path))\ntest_preds = np.zeros((test_size, N_CLASSES))\n\n\nfor model_path in model_path_list:\n    print(model_path)\n    K.clear_session()\n    model.load_weights(model_path)\n\n    if TTA_STEPS > 0:\n        test_ds = get_dataset(files_path, tta=True).repeat()\n        ct_steps = TTA_STEPS * ((test_size/BATCH_SIZE) + 1)\n        preds = model.predict(test_ds, steps=ct_steps, verbose=1)[:(test_size * TTA_STEPS)]\n        preds = np.mean(preds.reshape(test_size, TTA_STEPS, N_CLASSES, order='F'), axis=1)\n        test_preds += preds / len(model_path_list)\n    else:\n        test_ds = get_dataset(files_path, tta=False)\n        x_test = test_ds.map(lambda image, image_name: image)\n        test_preds += model.predict(x_test) / len(model_path_list)\n    \ntest_preds = np.argmax(test_preds, axis=-1)\ntest_names_ds = get_dataset(files_path)\nimage_names = [img_name.numpy().decode('utf-8') for img, img_name in iter(test_names_ds.unbatch())]\n\nsubmission = pd.DataFrame({'image_id': image_names, 'label': test_preds})\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.141593,"end_time":"2020-12-27T04:07:35.127617","exception":false,"start_time":"2020-12-27T04:07:34.986024","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Useful References\n\n- https://www.tensorflow.org/api_docs/python/tf/image/\n- https://albumentations.ai/docs/api_reference/augmentations/transforms/\n- https://pytorch.org/docs/stable/torchvision/transforms.html\n\n# Some Remarks\n\n- the needle augmentation implementation still remains buggy with the TensorFlow-Keras pipeline\n- in Torch, it seems to work just fine\n- after the competition has concluded and after reviewing private and public LB scores, it does seem that using insect augmentations led to a closer private and public LB score relationship! But sadly no medal this time round.. Sobs...\n\n# Cases Where Insect Augmentation Works\n\n- improved CV and LB scores observed when applied to medical image use cases, e.g. in RANZCR competition\n\n## All the best Kagglers!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}