{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Version information\n\n**Version 4** -- The original public version of the notebook.\n\n**Version 5** -- Two major changes were made: \n   1. As was pointed out by [Roman](https://www.kaggle.com/nroman), the hair images were originally designed for the 256x256 size, so they need to be scaled to use with images of different sizes. In this version we introduced a scaling factor for the dimensions of the hair images, so now they should work just fine with any input sizes.\n   2. It was pointed out by [Helgi](https://www.kaggle.com/helgith) that the TensorFlow code from **Version 4** was throwing an error when used in the graph mode. In this new version of the notebook, The TensorFlow code was tweaked to make it workable in the graph mode. An example is added to illustrate how to fetch a training batch and print it to the screen.\n   \n**Version 6** -- fixed some minor typo (thank you [Franko Sikic](https://www.kaggle.com/frankosikic)!). If you want to see how this augmentation can be included in your training pipeline take a look at **Versions 20** of the following public notebook of mine: \n\n[EfficientNet BN+Tabular Features TF CV5 512x512](https://www.kaggle.com/graf10a/efficientnet-bn-tabular-features-tf-cv5-512x512).\n\n**Version 9** -- modified the part illustrating how to incorporate this data augmentation into `tf.data.Dataset` API to show an example of how to deal with both images and tabular data. Please note that since I am running this notebook on CPU, I had to decrease the size of the shuffling buffer from 2048 to 512 in the definition of the `get_training_dataset` function below. \n\nAlso, the images shown in this last part might look a bit strange to you. This is because they were pre-processed with the Shades of Gray algorithm. This is totally optional, I decided to use this images just for fun. You can read more about the Shades of Gray algorithm in this discussion topic: [Shades of Gray prepossessed data](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161719).\n\n**Version 11** -- Updated tfrecords (fixed color artifacts after Shades of Gray pre-processing).\n\n**Version 13** -- Added the batch version of [Chris Deotte's affine augmentation](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96) using the approach from [this great kernel](https://www.kaggle.com/yihdarshieh/make-chris-deotte-s-data-augmentation-faster) by [Yih-Dar SHIEH](https://www.kaggle.com/yihdarshieh). We switched back to the original images (no color constant pre-processing). For more information see the following discussion topic: [Batch form of affine augmentations in Tensor Flow](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/169504).","metadata":{}},{"cell_type":"markdown","source":"### Motivation and acknowledgement\n\nThis notebook is based on the idea suggested by [Roman](https://www.kaggle.com/nroman) in the following discussion topic: [Advanced hair augmentation](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/159176). We first reproduce his result using the OpenCV library and illustrate it with some sample images. After that we re-write the OpenCV code in TensorFlow. The TensorFlow implementation of this technique makes it possible to use this agumentation with the `tf.data` API which is very well suited for tfrecords and TPU.","metadata":{}},{"cell_type":"markdown","source":"### Libraries","metadata":{}},{"cell_type":"code","source":"import os\nimport gc\nimport cv2\nimport math\nimport random\nimport numpy as np\nimport pandas as pd\nfrom glob import glob\nimport tensorflow as tf\nfrom pathlib import Path\nimport matplotlib.pyplot as plt\nimport tensorflow.keras.backend as K\nfrom kaggle_datasets import KaggleDatasets\n\nprint(\"Tensorflow version \" + tf.__version__)\nAUTO = tf.data.experimental.AUTOTUNE","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Image paths","metadata":{}},{"cell_type":"code","source":"n_max=6     # the maximum number of hairs to augment\nim_size=512  # all images are resized to this size\n\nhair_images=glob('/kaggle/input/melanoma-hairs/*.png')\nskin_images = glob(\"/kaggle/input/sample-skin/cancer/*.png\")\n# train_images=glob('/kaggle/input/siim-isic-melanoma-classification/jpeg/train/*.jpg')\n# test_images=glob('/kaggle/input/siim-isic-melanoma-classification/jpeg/test/*.jpg')\n\nlen(hair_images), len(skin_images)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Augmenting hair with OpenCV","metadata":{}},{"cell_type":"code","source":"def hair_aug_ocv(input_img):\n    \n    img=input_img.copy()\n    # Randomly choose the number of hairs to augment (up to n_max)\n    n_hairs = random.randint(0, n_max)\n\n    # If the number of hairs is zero then do nothing\n    if not n_hairs:\n        return img, n_hairs\n\n    # The image height and width (ignore the number of color channels)\n    im_height, im_width, _ = img.shape \n\n    for _ in range(n_hairs):\n\n        # Read a random hair image\n        hair = cv2.imread(random.choice(hair_images)) \n        \n        # Rescale the hair image to the right size (256 -- original size)\n        scale=im_size/256\n        hair = cv2.resize(hair, (int(scale*hair.shape[1]), int(scale*hair.shape[0])), \n                          interpolation=cv2.INTER_AREA)       \n\n        # Flip it\n        # flipcode = 0: flip vertically\n        # flipcode > 0: flip horizontally\n        # flipcode < 0: flip vertically and horizontally    \n        hair = cv2.flip(hair, flipCode=random.choice([-1, 0, 1]))\n\n        # Rotate it\n        hair = cv2.rotate(hair, rotateCode=random.choice([cv2.ROTATE_90_CLOCKWISE,\n                                                          cv2.ROTATE_90_COUNTERCLOCKWISE,\n                                                          cv2.ROTATE_180\n                                                         ])\n                         )\n        \n        \n        # The hair image height and width (ignore the number of color channels)\n        h_height, h_width, _ = hair.shape\n\n        # The top left coord's of the region of interest (roi)  \n        # where the augmentation will be performed\n        roi_h0 = random.randint(0, im_height - h_height)\n        roi_w0 = random.randint(0, im_width - h_width)\n\n        # The region of interest\n        roi = img[roi_h0:(roi_h0 + h_height), roi_w0:(roi_w0 + h_width)]\n\n        # Convert the hair image to grayscale\n        hair2gray = cv2.cvtColor(hair, cv2.COLOR_BGR2GRAY)\n\n        # If the pixel value is smaller than the threshold (10), it is set to 0 (black), \n        # otherwise it is set to a maximum value (255, white).\n        # ret -- the list of thresholds (10 in this case)\n        # mask -- the thresholded image\n        # The original image must be a grayscale image\n        # https://docs.opencv.org/master/d7/d4d/tutorial_py_thresholding.html\n        ret, mask = cv2.threshold(hair2gray, 10, 255, cv2.THRESH_BINARY)\n\n        # Invert the mask\n        mask_inv = cv2.bitwise_not(mask)\n\n        # Bitwise AND won't be performed where mask=0\n        img_bg = cv2.bitwise_and(roi, roi, mask=mask_inv)\n        hair_fg = cv2.bitwise_and(hair, hair, mask=mask)\n        # Fixing colors\n        hair_fg = cv2.cvtColor(hair_fg, cv2.COLOR_BGR2RGB)\n        # Overlapping the image with the hair in the region of interest\n        dst = cv2.add(img_bg, hair_fg)\n        # Inserting the result in the original image\n        img[roi_h0:roi_h0 + h_height, roi_w0:roi_w0 + h_width] = dst\n        \n    return img, n_hairs","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Examples of hair augmentation with OpenCV","metadata":{}},{"cell_type":"code","source":"def aug_examples(paths):\n\n    for img_path in paths:\n        # Read the image\n        img=cv2.imread(img_path)\n        # Fixing colors\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        # Resize to the desired size\n        img = cv2.resize(img , (im_size, im_size), interpolation = cv2.INTER_AREA )\n        # Creating an augmented image\n        img_aug, n_hairs = hair_aug_ocv(img)\n        \n        _, (ax1,ax2) = plt.subplots(1, 2)\n        \n        im_name=img_path.split('/')[-1].split('.')[0]    \n        ax1.set_title(f\"{im_name}\")            \n        ax2.set_title(f\"{im_name} with {n_hairs} {'hair' if n_hairs==1 else 'hairs'}\")\n        \n        ax1.imshow(img)\n        ax2.imshow(img_aug)\n        \n        plt.tight_layout()\n        plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_path = skin_images[0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img=cv2.imread(img_path)\n# Fixing colors\nimg = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"skin_images","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img = cv2.resize(img , (im_size, im_size), interpolation = cv2.INTER_AREA )\n# Creating an augmented image\nimg_aug, n_hairs = hair_aug_ocv(img)\n\n_, (ax1,ax2) = plt.subplots(1, 2)\n\nim_name=img_path.split('/')[-1].split('.')[0]    \nax1.set_title(f\"{im_name}\")            \nax2.set_title(f\"{im_name} with {n_hairs} {'hair' if n_hairs==1 else 'hairs'}\")\n\nax1.imshow(img)\nax2.imshow(img_aug)\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"aug_examples(skin_images)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.preprocessing.image import ImageDataGenerator","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_augmenter=ImageDataGenerator(\n    rescale=1./255, \n    #rotation range and fill mode only\n    #samplewise_center=True, \n    #samplewise_std_normalization=True, \n    #horizontal_flip = True, \n    #vertical_flip = True, \n    #height_shift_range= 0.05, \n    #width_shift_range=0.1, \n    rotation_range=10, \n    #shear_range = 0.1,\n    #fill_mode = 'nearest',\n    #zoom_range=0.10,\n    brightness_range=(0.8,1.2)\n    #preprocessing_function=function_name,\n    )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = train_augmenter.flow_from_directory(\"/kaggle/input/sample-skin/\", )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\ndef plotImages(images_arr):\n    fig, axes = plt.subplots(1, 5, figsize=(20,20))\n    axes = axes.flatten()\n    for img, ax in zip( images_arr, axes):\n        ax.imshow(img)\n    plt.tight_layout()\n    plt.show()\n    \n    \naugmented_images = [data[0][0][1] for i in range(5)]\nplotImages(augmented_images)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[0][0].shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(data[0][0][0])","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}