{"cells":[{"metadata":{},"cell_type":"markdown","source":"![image](https://raw.githubusercontent.com/Lexie88rus/augmentation-packages-overview/master/images/art-assorted-background.jpg)"},{"metadata":{},"cell_type":"markdown","source":"_Photo by rawpixel.com from Pexels_"},{"metadata":{},"cell_type":"markdown","source":"# Image Augmentation for Deep Learning\n### _Overview of popular augmentation packages and PyTorch examples_"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# Import utilities\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Import Image manipulation\nfrom PIL import Image\n\n# Import data visualization\nfrom matplotlib import pyplot as plt\nimport matplotlib.patches as patches\nimport matplotlib\n\n# Import PyTorch\nimport torch\nfrom torch import nn\nimport torch.nn.functional as F\nimport torchvision.transforms.functional as TF\nfrom torch.utils.data import Dataset, DataLoader","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"! pip install augmentor","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"! pip install albumentations","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Introduction"},{"metadata":{},"cell_type":"markdown","source":"Having a large dataset is crucial for performance of the deep learning model. However, we can improve the performance of the model by [augmenting the data we already have](https://bair.berkeley.edu/blog/2019/06/07/data_aug/). Deep learning frameworks usually have built-in data augmentation utilities (like torchvision transformations in PyTorch), but those can be unefficient or lacking some required functionality (like mask augmentation).\n\nIn this kernel I would like to make an overview of most popular data augmentation packages, designed specifically for machine learning, and demonstrate how to use these packages with PyTorch."},{"metadata":{},"cell_type":"markdown","source":"## Load Demo Images"},{"metadata":{},"cell_type":"markdown","source":"Let's load and look at the sample image for the demonstration:"},{"metadata":{},"cell_type":"markdown","source":"`1. Image for classification:`"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# load sample dog image\ndog_img = Image.open('../input/image1.jpg')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# show the demo image\nplt.title('Sample image')\nplt.axis('off')\nplt.imshow(dog_img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`2. Image and mask for segmentation:`"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# load mask for sample dog image\ndog_mask = Image.open('../input/mask1.png')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# define a function plot image and mask\ndef plot_image_and_mask(image, mask):\n    '''\n    Function to plot a single prediction:\n    INPUT:\n        image - PIL image \n        mask - PIL image with corresponding mask\n    '''\n    fig, axs = plt.subplots(1, 3, figsize=(15,5))\n\n    #plot the original data\n    axs[0].imshow(image) \n    axs[0].axis('off')\n    axs[0].set_title('Image')\n\n    #plot the mask\n    axs[1].imshow(mask)\n    axs[1].axis('off')   \n    axs[1].set_title('Mask')\n    \n    #plot image and add the mask\n    axs[2].imshow(image)\n    axs[2].imshow(mask, alpha = 0.5, cmap = \"Reds\")\n    axs[2].axis('off')   \n    axs[2].set_title('Image with mask overlay')\n\n    # set suptitle\n    plt.suptitle('Image with mask')\n    plt.show()\n\n# plot image and a mask\nplot_image_and_mask(dog_img, dog_mask)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`3. Image and bounding box for object detection`:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# sample code from\n# https://stackoverflow.com/questions/37435369/matplotlib-how-to-draw-a-rectangle-on-image\n\n# Create figure and axes\nfig,ax = plt.subplots(1)\n\n# Display the image\nax.imshow(dog_img)\n\n# Create a Rectangle patch\nrect = patches.Rectangle((0,9),980,525,linewidth=1,edgecolor='r',facecolor='none')\n\n# Add the patch to the Axes\nax.add_patch(rect)\nax.axis('off') # disable axis\n\nplt.title('Image with bounding box')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## imgaug package"},{"metadata":{},"cell_type":"markdown","source":"__[imgaug](https://github.com/aleju/imgaug)__ is an extremely powerful package for image augmentation. It contains:\n* Over 40 image augmentors and augmentation techniques;\n* Functionality to augment images with masks, bounding boxes, keypoints and heatmaps. This functionality makes it very easy to augment the dataset containing images for segmentation and object detection problems;\n* Complex augmentation pipelines;\n* Many helper functions for augmentation visualization, convertion, and more.\n\nYou can find full documentation for __imgaug__ [here](https://imgaug.readthedocs.io/en/latest/index.html)."},{"metadata":{},"cell_type":"markdown","source":"`1.` Simple image augmentation examples:"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"! pip install --upgrade imgaug","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# import the library and helpers\nimport imageio\nimport imgaug as ia\nfrom imgaug import augmenters as iaa","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Augmentation of a single image:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# use imageio library to read the image (alternatively you can use OpenCV cv2.imread() function)\nimage = imageio.imread('../input/image1.jpg')\n\n# initialize the augmenters for demo\nrotate = iaa.Affine(rotate=(-25, 25)) # rotate image\ngaussian_noise = iaa.AdditiveGaussianNoise(scale=(10, 60)) # add gaussian noise\ncrop = iaa.Crop(percent=(0, 0.4)) # crop image\nhue = iaa.AddToHueAndSaturation((-60, 60))  # change their color\nelastic_trans = iaa.ElasticTransformation(alpha=90, sigma=9) # water-like effect\ncoarse_drop = iaa.CoarseDropout((0.01, 0.1), size_percent=0.01)# set large image areas to zero","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get augmented images\nimage_rotated = rotate.augment_images([image])\nimage_noise = gaussian_noise.augment_images([image])\nimage_crop = crop.augment_images([image])\nimage_hue = hue.augment_images([image])\nimage_trans = elastic_trans.augment_images([image])\nimage_coarse = coarse_drop.augment_images([image])\n\n# create an array of augmented images for the demo\nimages_aug = [image_rotated[0], image_noise[0], image_crop[0], image_hue[0], image_trans[0], image_coarse[0]]\n\n# plot augmentation examples\nplt.figure(figsize=(15,5))\nplt.axis('off')\nplt.imshow(np.hstack(images_aug))\nplt.title('Sample augmentations')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Augmentation of an image with mask:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# import segmentation maps from imgaug\nfrom imgaug.augmentables.segmaps import SegmentationMapOnImage\n\n# open image with mask and convert to binary map\npil_mask = Image.open('../input/mask1.png')\npil_mask = pil_mask.convert('RGB')\n\nnp_mask = np.array(pil_mask)\nnp_mask = np.clip(np_mask, 0, 1)\n\n# create segmentation map for classes: background, dog\nsegmap = np.zeros(image.shape, dtype=bool)\nsegmap[:] = np_mask\nsegmap = SegmentationMapOnImage(segmap, shape=image.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# initialize augmentations\nseq = iaa.Sequential([\n    iaa.CoarseDropout(0.1, size_percent=0.2),\n    iaa.Affine(rotate=(-30, 30)),\n    iaa.ElasticTransformation(alpha=10, sigma=1)\n])\n\n# apply augmentation for image and mask\nimage_aug, segmap_aug = seq(image=image, segmentation_maps=segmap)\n\n# visualize augmented image and mask\nside_by_side = np.hstack([\n    segmap.draw_on_image(image),\n    segmap_aug.draw_on_image(image_aug),  # show blend of (augmented) image and segmentation map\n    segmap_aug.draw()  # show only the augmented segmentation map\n])\n\nfig, ax = plt.subplots(figsize=(10, 7))\nax.axis('off')\nplt.title('Augmentations for segmentation masks')\nax.imshow(side_by_side)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Augmentation of an image with bounding box:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# import bounding boxes from imgaug\nfrom imgaug.augmentables.bbs import BoundingBox, BoundingBoxesOnImage","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# initialize the bounding box for the original image\n# using helpers from imgaug package\nbbs = BoundingBoxesOnImage([\n    BoundingBox(x1=1, x2=980, y1=9, y2=535)\n], shape=image.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# define a simple augmentations pipeline for the image with bounding box\nseq = iaa.Sequential([\n    iaa.GammaContrast(1.5), # add contrast\n    iaa.Affine(translate_percent={\"x\": 0.1}, scale=0.8), # translate the image\n    iaa.Fliplr(p = 1.0) # apply horizontal flip\n])\n\n# apply augmentations\nimage_aug, bbs_aug = seq(image=image, bounding_boxes=bbs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot the initial and the augmented images with bounding boxes\n# using helpers from imgaug package\nside_by_side = np.hstack([\n    bbs.draw_on_image(image, size=2),\n    bbs_aug.draw_on_image(image_aug, size=2)\n])\n\nfig, ax = plt.subplots(figsize=(10, 7))\nax.axis('off')\nplt.title('Augmentations for bounding boxes')\nax.imshow(side_by_side)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Rotation example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# define a simple augmentations pipeline for the image with bounding box\nseq = iaa.Sequential([\n    iaa.Affine(rotate=(-30, 30)),\n    iaa.GammaContrast(1.5), # add contrast\n    iaa.Affine(translate_percent={\"x\": 0.1}, scale=0.8), # translate the image\n    iaa.Fliplr(p = 1.0) # apply horizontal flip\n])\n\n# apply augmentations\nimage_aug, bbs_aug = seq(image=image, bounding_boxes=bbs)\n\n# plot the initial and the augmented images with bounding boxes\n# using helpers from imgaug package\nside_by_side = np.hstack([\n    bbs.draw_on_image(image, size=2),\n    bbs_aug.draw_on_image(image_aug, size=2)\n])\n\nfig, ax = plt.subplots(figsize=(10, 7))\nax.axis('off')\nplt.title('Augmentations for bounding boxes with rotation')\nax.imshow(side_by_side)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`2.` Example of data augmentation pipeline:"},{"metadata":{},"cell_type":"markdown","source":"With the __imgaug__ library we can create a pipeline of image agumentations. Augmentations from the pipeline will be applied to each image sequentially, but we can set up:\n1. The probability the augmentation will be applied to an image. We can use this to apply, for example, horizontal flip to just 50% of the images.\n2. Apply only a subset of augmenters to an image. For example, apply 0 to 5 of augmenters from the list. This will help to speed up data generation.\n3. Apply augmentations in random order.\n\nLet's see an example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# define an augmentation pipeline\naug_pipeline = iaa.Sequential([\n    iaa.Sometimes(0.5, iaa.GaussianBlur((0, 3.0))), # apply Gaussian blur with a sigma between 0 and 3 to 50% of the images\n    # apply one of the augmentations: Dropout or CoarseDropout\n    iaa.OneOf([\n        iaa.Dropout((0.01, 0.1), per_channel=0.5), # randomly remove up to 10% of the pixels\n        iaa.CoarseDropout((0.03, 0.15), size_percent=(0.02, 0.05), per_channel=0.2),\n    ]),\n    # apply from 0 to 3 of the augmentations from the list\n    iaa.SomeOf((0, 3),[\n        iaa.Sharpen(alpha=(0, 1.0), lightness=(0.75, 1.5)), # sharpen images\n        iaa.Emboss(alpha=(0, 1.0), strength=(0, 2.0)), # emboss images\n        iaa.Fliplr(1.0), # horizontally flip\n        iaa.Sometimes(0.5, iaa.CropAndPad(percent=(-0.25, 0.25))), # crop and pad 50% of the images\n        iaa.Sometimes(0.5, iaa.Affine(rotate=5)) # rotate 50% of the images\n    ])\n],\nrandom_order=True # apply the augmentations in random order\n)\n\n# apply augmentation pipeline to sample image\nimages_aug = np.array([aug_pipeline.augment_image(image) for _ in range(16)])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Helper function to display the images in a grid\n# Source: https://stackoverflow.com/questions/42040747/more-idiomatic-way-to-display-images-in-a-grid-with-numpy\ndef gallery(array, ncols=3):\n    '''\n    Function to arange images into a grid.\n    INPUT:\n        array - numpy array containing images\n        ncols - number of columns in resulting imahe grid\n    OUTPUT:\n        result - reshaped array into a grid with given number of columns\n    '''\n    nindex, height, width, intensity = array.shape\n    nrows = nindex//ncols\n    assert nindex == nrows*ncols\n    result = (array.reshape(nrows, ncols, height, width, intensity)\n              .swapaxes(1,2)\n              .reshape(height*nrows, width*ncols, intensity))\n    return result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# visualize the augmented images\nplt.figure(figsize=(10,10))\nplt.axis('off')\nplt.imshow(gallery(images_aug, ncols = 4))\nplt.title('Augmentation pipeline examples')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`3.` Example of using __imgaug__ with PyTorch:"},{"metadata":{},"cell_type":"markdown","source":"The following code snippet shows how __imgaug__ augmentations can be used in PyTorch dataset for generation of new augmented images:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# define train and test augmentations\nAUG_TRAIN = aug_pipeline # use our pipeline as train augmentations\nAUG_TEST = None # don't use any augmentations for test images\n\n# Define the demo dataset\nclass DogDataset(Dataset):\n    '''\n    Sample dataset for imgaug demonstration.\n    The dataset will consist of just one sample image.\n    '''\n\n    def __init__(self, image, augmentations = None):\n        self.image = image\n        self.augmentations = augmentations # save the augmentations\n\n    def __len__(self):\n        return 1 # return 1 as we have only one image\n\n    def __getitem__(self, idx):\n        # return the augmented image\n        return TF.to_tensor(self.augmentations.augment_image(self.image))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# load the augmented data\n\n# initialize the dataset, pass the augmentation pipeline as an argument to init function\ntrain_ds = DogDataset(image, augmentations = AUG_TRAIN)\n\n# initilize the dataloader for training\ntrainloader = DataLoader(train_ds, batch_size=1, shuffle=True, num_workers=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get the augmented image from trainloader\n# run this cell several times to produce different augmented images\nfor img in  trainloader:\n    plt.axis('off')\n    plt.imshow(TF.to_pil_image(img.reshape(3, 768, 1024))) # convert image to PIL from tensor with to_pil_image PyTorch helper","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Albumentations package\n\n[Albumentations](https://github.com/albu/albumentations) package is based on numpy, OpenCV and imgaug. This is a very pipular package written by Kaggle masters and used widely in Kaggle competitions. Moreover, this package is very efficient. You may find the benchmarking results [here](https://github.com/albu/albumentations#benchmarking-results) and the full documentation for this package [here](https://albumentations.readthedocs.io/en/latest/).\n\nAlbumetations package is capable of:\n* Over 60 pixel level and spatial level transformations;\n* Transforming images with masks bounding boxes and keypoints;\n* Organizing augmentations into pipelines;\n* PyTorch integration."},{"metadata":{},"cell_type":"markdown","source":"`1.` Simple image augmentation examples:"},{"metadata":{},"cell_type":"markdown","source":"Demonstration of single image augmentations:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# import albumentations package\nimport albumentations as A\n\n# initialize augmentations\ngaus_noise = A.GaussNoise() # gaussian noise\nelastic = A.ElasticTransform() # elastic transform\nbright_contrast = A.RandomBrightnessContrast(p=1) # random brightness and contrast\ngamma = A.RandomGamma(p=1) # random gamma\nclahe = A.CLAHE(p=1) # CLAHE (see https://en.wikipedia.org/wiki/Adaptive_histogram_equalization#Contrast_Limited_AHE)\nblur = A.Blur()\n\n# apply augmentations\n# pass image to the augmentation\nimg_gaus = gaus_noise(image = image)\nimg_elastic = elastic(image = image)\nimg_bc = bright_contrast(image = image)\nimg_gamma = gamma(image = image)\nimg_clahe = clahe(image = image)\nimg_blur = blur(image = image)\n\n# access the augmented image by 'image' key\nimg_list = [img_gaus['image'], img_elastic['image'], img_bc['image'], img_gamma['image'], img_clahe['image'], img_blur['image']]\n\n# visualize the augmented images\nplt.figure(figsize=(10,10))\nplt.axis('off')\nplt.imshow(gallery(np.array(img_list), ncols = 3))\nplt.title('Augmentation examples')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Augmenting images with masks:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# compose augmentation pipeline\naug_pipeline = A.Compose([\n    A.ShiftScaleRotate(p = 1),\n    A.RGBShift(),\n    A.Blur(p = 1),\n    A.GaussNoise(p = 1)\n],p=1)\n\n# load the mask\nmask = imageio.imread('../input/mask1.png')\n# apply augmentations to image and a mask\naugmented = aug_pipeline(image = image, mask = mask)\n\n# visualize augmented image and mask\nfig, ax = plt.subplots(1,4, figsize = (15, 10))\n\nax[0].axis('off')\nax[0].imshow(image)\nax[0].set_title('original image')\n\nax[1].axis('off')\nax[1].imshow(augmented['image'])\nax[1].set_title('augmented image')\n\nax[2].axis('off')\nax[2].imshow(augmented['image'])\nax[2].imshow(augmented['mask'].squeeze(), alpha = 0.5, cmap = \"Reds\")\nax[2].set_title('augmented image with mask')\n\nax[3].axis('off')\nax[3].imshow(augmented['mask'].squeeze(), alpha = 1.0, cmap = \"Reds\")\nax[3].set_title('augmented mask')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Augmentation of images with bounding boxes:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# create bounding boxes from mask with cv2\nimport cv2\nmask = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)\nbboxes = cv2.boundingRect(cv2.findNonZero(mask))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# compose augmentation pipeline\naug_pipeline = A.Compose([\n    A.ShiftScaleRotate(rotate_limit=0, p = 1),\n    A.RGBShift(p = 1),\n    A.Blur(p = 1),\n    A.GaussNoise(p = 1)\n],p=1)\n\n# augment image and bounding box\naugmented_boxes = aug_pipeline(image = image, bboxes = [bboxes])\nbox_aug = augmented_boxes['bboxes'][0]\n\n# visualize augmented image and bbox\nfig, ax = plt.subplots(1,2, figsize = (15, 10))\n\nax[0].axis('off')\nax[0].imshow(image)\nrect = patches.Rectangle((bboxes[0],bboxes[1]),bboxes[2],bboxes[3],linewidth=1,edgecolor='r',facecolor='none')\nax[0].add_patch(rect)\nax[0].set_title('original image')\n\nax[1].axis('off')\nax[1].imshow(augmented_boxes['image'])\nrect = patches.Rectangle((box_aug[0],box_aug[1]),box_aug[2],box_aug[3],linewidth=1,edgecolor='r',facecolor='none')\nax[1].add_patch(rect)\nax[1].set_title('augmented image')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`2.` Augmentation pipelines:"},{"metadata":{},"cell_type":"markdown","source":"__Albumentations__ package allows to construct advanced pipelines similar to __imgaug__ pipelines.\n\nLet's see an example (this example was inspired by [this kernel](https://www.kaggle.com/meaninglesslives/unet-plus-plus-with-efficientnet-encoder)):"},{"metadata":{"trusted":true},"cell_type":"code","source":"# compose complex augmentation pipeline\naugmentation_pipeline = A.Compose(\n    [\n        A.HorizontalFlip(p = 0.5), # apply horizontal flip to 50% of images\n        A.OneOf(\n            [\n                # apply one of transforms to 50% of images\n                A.RandomContrast(), # apply random contrast\n                A.RandomGamma(), # apply random gamma\n                A.RandomBrightness(), # apply random brightness\n            ],\n            p = 0.5\n        ),\n        A.OneOf(\n            [\n                # apply one of transforms to 50% images\n                A.ElasticTransform(\n                    alpha = 120,\n                    sigma = 120 * 0.05,\n                    alpha_affine = 120 * 0.03\n                ),\n                A.GridDistortion(),\n                A.OpticalDistortion(\n                    distort_limit = 2,\n                    shift_limit = 0.5\n                ),\n            ],\n            p = 0.5\n        )\n    ],\n    p = 1\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# apply pipeline to sample image\nimages_aug = np.array([augmentation_pipeline(image = image)['image'] for _ in range(16)])\n\n# visualize augmentation results\nplt.figure(figsize=(10,10))\nplt.axis('off')\nplt.imshow(gallery(images_aug, ncols = 4))\nplt.title('Augmentation pipeline examples')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`3.` PyTorch integration example:"},{"metadata":{},"cell_type":"markdown","source":"When using PyTorch you can effortlessly migrate from torchvision to __Albumentations__, because this package provides specialized utilities to be used with PyTorch. Migrating to __Albumentations__ will help to speed up the data generation part and train deep learning models faster. See detailed tutorial on migration from torchvision to __Albumentations__ [here](https://github.com/albu/albumentations/blob/master/notebooks/migrating_from_torchvision_to_albumentations.ipynb)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# import pytorch utilities from albumentations\nfrom albumentations.pytorch import ToTensor\n\n# define the augmentation pipeline\naugmentation_pipeline = A.Compose(\n    [\n        A.HorizontalFlip(p = 0.5), # apply horizontal flip to 50% of images\n        A.OneOf(\n            [\n                # apply one of transforms to 50% of images\n                A.RandomContrast(), # apply random contrast\n                A.RandomGamma(), # apply random gamma\n                A.RandomBrightness(), # apply random brightness\n            ],\n            p = 0.5\n        ),\n        A.OneOf(\n            [\n                # apply one of transforms to 50% images\n                A.ElasticTransform(\n                    alpha = 120,\n                    sigma = 120 * 0.05,\n                    alpha_affine = 120 * 0.03\n                ),\n                A.GridDistortion(),\n                A.OpticalDistortion(\n                    distort_limit = 2,\n                    shift_limit = 0.5\n                ),\n            ],\n            p = 0.5\n        ),\n        \n        # commented the normalization (which you will probably used for torchvision pretrained models)\n        # because images look weird after normalization\n        \n        #A.Normalize(\n        #    mean=[0.485, 0.456, 0.406],\n        #    std=[0.229, 0.224, 0.225]),\n        \n        ToTensor() # convert the image to PyTorch tensor\n    ],\n    p = 1\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# load the augmented data\n\n# Define the demo dataset\nclass DogDataset2(Dataset):\n    '''\n    Sample dataset for Albumentations demonstration.\n    The dataset will consist of just one sample image.\n    '''\n\n    def __init__(self, image, augmentations = None):\n        self.image = image\n        self.augmentations = augmentations # save the augmentations\n\n    def __len__(self):\n        return 1 # return 1 as we have only one image\n\n    def __getitem__(self, idx):\n        # return the augmented image\n        # no need to convert to tensor, because image is converted to tensor already by the pipeline\n        augmented = self.augmentations(image = self.image)\n        return augmented['image']\n\n# initialize the dataset, pass the augmentation pipeline as an argument to init function\ntrain_ds = DogDataset2(image, augmentations = augmentation_pipeline)\n\n# initilize the dataloader for training\ntrainloader = DataLoader(train_ds, batch_size=1, shuffle=True, num_workers=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get the augmented image from trainloader\n# run this cell several times to produce different augmented images\nfor img in trainloader:\n    plt.axis('off')\n    plt.imshow(TF.to_pil_image(img.reshape(3, 768, 1024))) # convert image to PIL from tensor with to_pil_image PyTorch helper","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Augmentor package"},{"metadata":{},"cell_type":"markdown","source":"__[Augmentor](https://github.com/mdbloice/Augmentor)__ package contains less possible augmentations than previous packages, but it has its own outstanding features like [size-preserving rotations](https://github.com/mdbloice/Augmentor#size-preserving-rotations), [size-preseving shearing](https://github.com/mdbloice/Augmentor#size-preserving-shearing) and [cropping](https://github.com/mdbloice/Augmentor#cropping), which is more suitable for machine learning.\n\n__Augmentor__ package also allows to compose augmentation pipelines and use them with PyTorch. You can find the full documentation for __Augmentor__ package [here](https://augmentor.readthedocs.io/en/master/)."},{"metadata":{},"cell_type":"markdown","source":"`1.` Simple image augmentation examples:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# import package\nimport Augmentor\n\n# initialize pipeline\np = Augmentor.DataPipeline([[np.array(image), np.array(mask)]])\n\n# apply augmentations\np.rotate(1, max_left_rotation=3, max_right_rotation=3)\np.shear(1, max_shear_left = 3, max_shear_right = 3)\np.zoom_random(1, percentage_area=0.9)\n\n# sample from augmentation pipeline\nimages_aug = p.sample(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# visualize augmented image\naugmented_image = images_aug[0][0]\naugmented_mask = images_aug[0][1]\n\n# visualize augmented image and mask\nfig, ax = plt.subplots(1,3, figsize = (15, 10))\n\nax[0].axis('off')\nax[0].imshow(image)\nax[0].set_title('original image')\n\nax[1].axis('off')\nax[1].imshow(augmented_image)\nax[1].set_title('augmented image')\n\nax[2].axis('off')\nax[2].imshow(augmented_mask)\nax[2].set_title('augmented mask')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`2.` Example of using __Augmentor__ with PyTorch:"},{"metadata":{},"cell_type":"markdown","source":"There are some [known issues](https://github.com/mdbloice/Augmentor/issues/109) with __Augmentor__ and PyTorch, but we can still use these libraries together.\n\nHere is an example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# load the augmented data\n\n# Define the demo dataset\nclass DogDataset3(Dataset):\n    '''\n    Sample dataset for Augmentor demonstration.\n    The dataset will consist of just one sample image.\n    '''\n\n    def __init__(self, image):\n        self.image = image\n\n    def __len__(self):\n        return 1 # return 1 as we have only one image\n\n    def __getitem__(self, idx):\n        # return the augmented image\n        # no need to convert to tensor, because image is converted to tensor already by the pipeline\n        \n        # initialize the pipeline\n        p = Augmentor.DataPipeline([[np.array(image)]])\n\n        # apply augmentations\n        p.rotate(0.5, max_left_rotation=10, max_right_rotation=10) # rotate the image with 50% probability\n        p.shear(0.5, max_shear_left = 10, max_shear_right = 10) # shear the image with 50% probability\n        p.zoom_random(0.5, percentage_area=0.7) # zoom randomly with 50% probability\n\n        # sample from augmentation pipeline\n        images_aug = p.sample(1)\n        \n        # get augmented image\n        augmented_image = images_aug[0][0]\n        \n        # convert to tensor and return the result\n        return TF.to_tensor(augmented_image)\n\n# initialize the dataset, pass the augmentation pipeline as an argument to init function\ntrain_ds = DogDataset3(image)\n\n# initilize the dataloader for training\ntrainloader = DataLoader(train_ds, batch_size=1, shuffle=True, num_workers=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get the augmented image from the trainloader\n# run this cell several times to produce different augmented images\nfor img in trainloader:\n    plt.axis('off')\n    plt.imshow(TF.to_pil_image(img.reshape(3, 768, 1024))) # convert image to PIL from tensor with to_pil_image PyTorch helper","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Conclusion"},{"metadata":{},"cell_type":"markdown","source":"In this kernel:\n* I described the main features of imgaug, Albumentations and Augmentor packages for image augmentation designed for machine learning;\n* I demonstrated how to use the packages for augmentation of images with masks and bounding boxes;\n* I demonstrated how to use the packages with PyTorch deep learning framework."},{"metadata":{},"cell_type":"markdown","source":"## Additional References and Credits\n1. [Unet Plus Plus with EfficientNet Encoder](https://www.kaggle.com/meaninglesslives/unet-plus-plus-with-efficientnet-encoder) - Kaggle kernel, which uses __Albumentations__ for xray image augmentation with Keras library.\n2. [Example](https://github.com/neptune-ml/open-solution-mapping-challenge/blob/master/src/augmentation.py) of usage of __imgaug__ for [Crowd AI mapping challenge](https://www.crowdai.org/challenges/mapping-challenge).\n3. Very informative Medium [Data Augmentation article](https://medium.com/nanonets/how-to-use-deep-learning-when-you-have-limited-data-part-2-data-augmentation-c26971dc8ced)."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}