{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport pylab as pl # linear algebra + plotting\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cf78719d21f01dc1d1d9c7109a8c9f6064ed4995"},"cell_type":"markdown","source":"It is sometimes a good idea to generate augmented data when the data is not equally distributed among classes, especially with image data. This becomes even more crucial when we don't have any idea about the distribution of classes in test data because there might be fewer samples from more frequent classes and more samples from classes with 1 sample in the train set for example. So, we need to generate modified versions of train images to make sure first, our learner sees enough data and secondly it is invariant to small changes like rotation or camera angles.\n\nkeras has a good utility to generate augmented data from the train set, as @YouHanLee beautifully explained in his kernel:\n\n[https://www.kaggle.com/youhanlee/small-data-many-class-data-augmentation](https://www.kaggle.com/youhanlee/small-data-many-class-data-augmentation)\n\nI prefer to have more control over my augmentation mechanism, so I have made the following code using skimage to generate augmented images from any photo. The same procedures are possible to do with OpenCV and even tensorflow to some extent.\n\nLet's first look at some photos from the same class (from the most frequent class) to see what type of augmentation would be the best:"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\nprint(train.Id.value_counts().head())\nwale_data = {}\nwale_data['w_23a388d'] = train[train.Id=='w_23a388d'].Image.values.tolist()\nwale_data['w_9b5109b'] = train[train.Id=='w_9b5109b'].Image.values.tolist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"be816b3ddc8029dce5682fe1d900193c6584b54b"},"cell_type":"code","source":"for wale_name in wale_data:\n    F = pl.figure(figsize=(15,9))\n    G = pl.GridSpec(3, 4, left=.01, right=.99, bottom=0.05, top=0.9, wspace=.01, hspace=0.01, figure=F)\n    for i in range(12):\n        im = pl.imread('../input/train/' + wale_data[wale_name][i])\n        ax = pl.subplot(G[i])\n        ax.imshow(im)\n        ax.set_axis_off()\n        ax.set_aspect('equal')\n    pl.suptitle(wale_name)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c73cfe21711638de8cb9051de0df9ff2cadc36f7"},"cell_type":"markdown","source":"So, it seems camera angle, zoom, rotations, skewness, translations, not fully out of water tails, white balance and color intensity, and noise induced by the picture quality and water splashes are common sources of differences in images.\n\nand, the important part to distinguish between the wales is their tale pattern. One idea might be to convert the images to grayscale and do some pre-processing, but it is not within the goals of the current kernel. We will focus next on the implementation of the above-mentioned transformations to do the augmentation.\n\nWe will implement the Affine Transform, which is a combination of translation, rotation, and scaling. This transformation preserves parallel lines.\n\nWe can do Projective Transform, which is similar to looking at the object from a different perspective, so it does not preserve parallel lines to remain parallel.\n\nI will also do some random cropping and adding Gaussian noise to the images:"},{"metadata":{"trusted":true,"_uuid":"4a1bde67c291e5b9bf10e62834228f0ba6582d68"},"cell_type":"code","source":"from skimage.transform import warp, AffineTransform, ProjectiveTransform\nfrom skimage.exposure import equalize_adapthist, equalize_hist, rescale_intensity, adjust_gamma, adjust_log, adjust_sigmoid\nfrom skimage.filters import gaussian\nfrom skimage.util import random_noise\nimport random\n\ndef randRange(a, b):\n    '''\n    a utility functio to generate random float values in desired range\n    '''\n    return pl.rand() * (b - a) + a\n\n\ndef randomAffine(im):\n    '''\n    wrapper of Affine transformation with random scale, rotation, shear and translation parameters\n    '''\n    tform = AffineTransform(scale=(randRange(0.75, 1.3), randRange(0.75, 1.3)),\n                            rotation=randRange(-0.25, 0.25),\n                            shear=randRange(-0.2, 0.2),\n                            translation=(randRange(-im.shape[0]//10, im.shape[0]//10), \n                                         randRange(-im.shape[1]//10, im.shape[1]//10)))\n    return warp(im, tform.inverse, mode='reflect')\n\n\ndef randomPerspective(im):\n    '''\n    wrapper of Projective (or perspective) transform, from 4 random points selected from 4 corners of the image within a defined region.\n    '''\n    region = 1/4\n    A = pl.array([[0, 0], [0, im.shape[0]], [im.shape[1], im.shape[0]], [im.shape[1], 0]])\n    B = pl.array([[int(randRange(0, im.shape[1] * region)), int(randRange(0, im.shape[0] * region))], \n                  [int(randRange(0, im.shape[1] * region)), int(randRange(im.shape[0] * (1-region), im.shape[0]))], \n                  [int(randRange(im.shape[1] * (1-region), im.shape[1])), int(randRange(im.shape[0] * (1-region), im.shape[0]))], \n                  [int(randRange(im.shape[1] * (1-region), im.shape[1])), int(randRange(0, im.shape[0] * region))], \n                 ])\n\n    pt = ProjectiveTransform()\n    pt.estimate(A, B)\n    return warp(im, pt, output_shape=im.shape[:2])\n\n\ndef randomCrop(im):\n    '''\n    croping the image in the center from a random margin from the borders\n    '''\n    margin = 1/10\n    start = [int(randRange(0, im.shape[0] * margin)),\n             int(randRange(0, im.shape[1] * margin))]\n    end = [int(randRange(im.shape[0] * (1-margin), im.shape[0])), \n           int(randRange(im.shape[1] * (1-margin), im.shape[1]))]\n    return im[start[0]:end[0], start[1]:end[1]]\n\n\ndef randomIntensity(im):\n    '''\n    rescales the intesity of the image to random interval of image intensity distribution\n    '''\n    return rescale_intensity(im,\n                             in_range=tuple(pl.percentile(im, (randRange(0,10), randRange(90,100)))),\n                             out_range=tuple(pl.percentile(im, (randRange(0,10), randRange(90,100)))))\n\ndef randomGamma(im):\n    '''\n    Gamma filter for contrast adjustment with random gamma value.\n    '''\n    return adjust_gamma(im, gamma=randRange(0.5, 1.5))\n\ndef randomGaussian(im):\n    '''\n    Gaussian filter for bluring the image with random variance.\n    '''\n    return gaussian(im, sigma=randRange(0, 5))\n    \ndef randomFilter(im):\n    '''\n    randomly selects an exposure filter from histogram equalizers, contrast adjustments, and intensity rescaler and applys it on the input image.\n    filters include: equalize_adapthist, equalize_hist, rescale_intensity, adjust_gamma, adjust_log, adjust_sigmoid, gaussian\n    '''\n    Filters = [equalize_adapthist, equalize_hist, adjust_log, adjust_sigmoid, randomGamma, randomGaussian, randomIntensity]\n    filt = random.choice(Filters)\n    return filt(im)\n\n\ndef randomNoise(im):\n    '''\n    random gaussian noise with random variance.\n    '''\n    var = randRange(0.001, 0.01)\n    return random_noise(im, var=var)\n    \n\ndef augment(im, Steps=[randomAffine, randomPerspective, randomFilter, randomNoise, randomCrop]):\n    '''\n    image augmentation by doing a sereis of transfomations on the image.\n    '''\n    for step in Steps:\n        im = step(im)\n    return im","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b49761a4f274648b5f4e4fcd779c1dfbdf22bca0"},"cell_type":"markdown","source":"Here is the step by step result of the transformations:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"7552a442ed4e6d1702c8fcfd4216291982ef5d3a"},"cell_type":"code","source":"im = pl.imread('../input/train/' + train.Image[0])\nF = pl.figure(figsize=(15,9))\nG = pl.GridSpec(2, 3, left=.01, right=.99, bottom=0.05, top=0.9, wspace=.01, hspace=0.05, figure=F)\nax = pl.subplot(G[0])\nax.imshow(im)\nax.set_axis_off()\nax.set_aspect('equal')\nax.set_title('original' + r'$\\rightarrow$')\nfor i, step in enumerate([randomAffine, randomPerspective, randomFilter, randomNoise, randomCrop]):\n    ax = pl.subplot(G[i+1])\n    im = step(im)\n    ax.imshow(im)\n    ax.set_axis_off()\n    ax.set_aspect('equal')\n    ax.set_title(step.__name__ + (r'$\\rightarrow$' if i < 4 else ''))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"22fcfa5d3f670ddccfff5d4cecd0183b25955f99"},"cell_type":"markdown","source":"The filter step is one filter selected randomly from histogram equalizers, contrast adjustments, Gaussian blurring, and intensity rescale:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ab750cc2de591a0b7107a14bfa2c07a9ccfbe0e6"},"cell_type":"code","source":"im = pl.imread('../input/train/' + train.Image[0])\nF = pl.figure(figsize=(15,6))\nG = pl.GridSpec(2, 4, left=.01, right=.99, bottom=0.05, top=0.9, wspace=.01, hspace=0.1, figure=F)\nax = pl.subplot(G[0])\nax.imshow(im)\nax.set_axis_off()\nax.set_aspect('equal')\nax.set_title('original')\nfor i, filt in enumerate([equalize_adapthist, equalize_hist, adjust_log, adjust_sigmoid, randomGamma, randomGaussian, randomIntensity]):\n    ax = pl.subplot(G[i+1])\n    ax.imshow(filt(im))\n    ax.set_axis_off()\n    ax.set_aspect('equal')\n    ax.set_title(filt.__name__ + ' on (original)')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9a9e5248de1703762c305b90d01c8306e1ebdb1b"},"cell_type":"markdown","source":"and, here is the final outcome, 11 augmented images compared to the original image:"},{"metadata":{"trusted":true,"_uuid":"1d388a2bb749816c3a9d6b1c4d350485155cb21e"},"cell_type":"code","source":"im = pl.imread('../input/train/' + train.Image[0])\nAug_im = [augment(im) for i in range(11)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a023300245f2cab29aecd0fc96bf490f944e81f9","_kg_hide-input":true},"cell_type":"code","source":"F = pl.figure(figsize=(15,10))\nG = pl.GridSpec(3, 4, left=.01, right=.99, bottom=0.05, top=0.9, wspace=.01, hspace=0.05, figure=F)\nax = pl.subplot(G[0])\nax.imshow(im)\nax.set_axis_off()\nax.set_aspect('equal')\nax.set_title('original')\nfor i in range(1, 12):\n    ax = pl.subplot(G[i])\n    ax.imshow(Aug_im[i-1])\n    ax.set_axis_off()\n    ax.set_aspect('equal')\n    ax.set_title(f'Augmented image {i}')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3d6ef670d58c0fa87bc210cb89bb7710aa2a83a9"},"cell_type":"markdown","source":"**Further Readings**\n\nI recommend the following links to the docs and some blogs:\n\n* http://scikit-image.org/docs/dev/api/skimage.transform.html\n* http://scikit-image.org/docs/dev/api/skimage.exposure.html\n* http://scikit-image.org/docs/dev/api/skimage.filters.html\n* http://scikit-image.org/docs/dev/auto_examples/xx_applications/plot_geometric.html\n* https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_geometric_transformations/py_geometric_transformations.html\n* https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_filtering/py_filtering.html\n* https://medium.com/nanonets/how-to-use-deep-learning-when-you-have-limited-data-part-2-data-augmentation-c26971dc8ced\n* https://machinelearningmastery.com/image-augmentation-deep-learning-keras/\n\n**Further Steps**\n\nIt is possible to fine-tune the transformations to avoid un-natural final images. \n\nI avoided using horizontal/vertical flip transformations since I think mirrored images of the whale tales might confuse the learning system in this problem.\n\nWe can write the new images on the disk, or we can use this in keras pipelines to augment while reading the data.\n\nI hope it was helpful. Let me know if you have any critics or have a way to improve it."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}