{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Understanding Clouds EDA\n\n### About the Competition\nThe challenge is to segment satellite images into one of four classes: Sugar, Flower, Fish and Gravel. These clouds look benign compared to big thunderstorms but, in fact, for the Earth’s climate they play a huge role. The reason is that they reflect a lot of sunlight back into space, thereby cooling our planet, while only contributing marginally to the greenhouse effect. This means that it’s really important to figure out how these clouds will change as our planet warms.\n\n"},{"metadata":{},"cell_type":"markdown","source":"## DATA"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# imports\nimport numpy as np\nimport pandas as pd\nimport os\nimport cv2\nfrom glob import glob\nimport imageio\n# visualization\nimport matplotlib.pyplot as plt\nfrom matplotlib import patches as patches\nimport seaborn as sns\n# plotly offline imports\nfrom plotly.offline import download_plotlyjs, init_notebook_mode, iplot\nfrom plotly import subplots\nimport plotly.express as px\nimport plotly.figure_factory as ff\nfrom plotly.graph_objs import *\nfrom plotly.graph_objs.layout import Margin, YAxis, XAxis\ninit_notebook_mode()\n# frequent pattern mining\nfrom mlxtend.frequent_patterns import fpgrowth\nfrom PIL import Image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load Data\n\nalso let's look at the train and test folders along with defining the paths respectively"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# defining data paths\nTRAIN_PATH = '../input/understanding_cloud_organization/train_images/'\nTEST_PATH = '../input/understanding_cloud_organization/test_images/'\n\n# load dataframe with train labels\ntrain_df = pd.read_csv('../input/understanding_cloud_organization/train.csv')\ntrain_fns = sorted(glob(TRAIN_PATH + '*.jpg'))\ntrain_image_path = os.path.join('/kaggle/input/understanding_cloud_organization','train_images')\n\nprint('There are {} images in the train set.'.format(len(train_fns)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# load the filenames for test images\ntest_fns = sorted(glob(TEST_PATH + '*.jpg'))\n\nprint('There are {} images in the test set.'.format(len(test_fns)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plot the pie chart for the train and test datasets:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# plotting a pie chart which demonstrates train and test sets\nlabels = 'Train', 'Test'\nsizes = [len(train_fns), len(test_fns)]\nexplode = (0, 0.1)\n\nfig, ax = plt.subplots(figsize=(6, 6))\nax.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%', shadow=True, startangle=90)\nax.axis('equal')\nax.set_title('Train and Test Sets')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"1st observation:\n\n* for each image in the training set there are 4 lines for each type of cloud\n* \"Image_Label\" contains the image filename along with a cloud type seperated by \"_\"\n* If a certain type of clouds in present on the image, the EncodedPixels column is non-null and contains the segmentation map for the corresponding cloud type.\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# split column\nsplit_df = train_df[\"Image_Label\"].str.split(\"_\", n = 1, expand = True)\n# add new columns to train_df\ntrain_df['Image'] = split_df[0]\ntrain_df['Label'] = split_df[1]\n\n# check the result\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"now that we have our modified csv, we shall explore some more"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Total number of images: %s' % len(train_df['Image'].unique()))\nprint('Images with at least one label: %s' % len(train_df[train_df['EncodedPixels'] != 'NaN']['Image'].unique()))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# different types of clouds we have in our dataset\ntrain_df['Label'].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# count the number of labels of each cloud type\nfish = train_df[train_df['Label'] == 'Fish'].EncodedPixels.count()\nflower = train_df[train_df['Label'] == 'Flower'].EncodedPixels.count()\ngravel = train_df[train_df['Label'] == 'Gravel'].EncodedPixels.count()\nsugar = train_df[train_df['Label'] == 'Sugar'].EncodedPixels.count()\n\nprint('There are {} fish clouds'.format(fish))\nprint('There are {} flower clouds'.format(flower))\nprint('There are {} gravel clouds'.format(gravel))\nprint('There are {} sugar clouds'.format(sugar))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plotting a pie chart\nlabels = 'Fish', 'Flower', 'Gravel', 'Sugar'\nsizes = [fish, flower, gravel, sugar]\n\nfig, ax = plt.subplots(figsize=(6, 6))\nax.pie(sizes, labels=labels, autopct='%1.1f%%', shadow=True, startangle=90)\nax.axis('equal')\nax.set_title('Cloud Types')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we can see that the distribution among the classes are almost balanced which is a great relief!"},{"metadata":{"trusted":true},"cell_type":"code","source":"# explore the number of labels per image\n\nlabels_per_image = train_df.groupby('Image')['EncodedPixels'].count()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('The mean number of labels per image is {}'.format(labels_per_image.mean()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(6,6))\nax.hist(labels_per_image)\nax.set_title('Number of labels per image')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"so we can see that most of the images have 2 labels.\n\nnow let's try to find out the correlation between the different types of clouds"},{"metadata":{"trusted":true},"cell_type":"code","source":"# create dummy columns for each cloud type\ncorr_df = pd.get_dummies(train_df, columns = ['Label'])\n\n#fill null values with -1\ncorr_df = corr_df.fillna(-1)\n\n#define a helper func. to fill dummy columns\ndef get_dummy_val(row, cloud_type):\n    if cloud_type == 'fish':\n        return row['Label_Fish'] * (row['EncodedPixels'] != -1)\n    if cloud_type == 'flower':\n        return row['Label_Flower'] * (row['EncodedPixels'] != -1)\n    if cloud_type == 'gravel':\n        return row['Label_Gravel'] * (row['EncodedPixels'] != -1)\n    if cloud_type == 'sugar':\n        return row['Label_Sugar'] * (row['EncodedPixels'] != -1)\n    \n# fill dummy columns\ncorr_df['Label_Fish'] = corr_df.apply(lambda row: get_dummy_val(row, 'fish'), axis=1)\ncorr_df['Label_Flower'] = corr_df.apply(lambda row: get_dummy_val(row, 'flower'), axis=1)\ncorr_df['Label_Gravel'] = corr_df.apply(lambda row: get_dummy_val(row, 'gravel'), axis=1)\ncorr_df['Label_Sugar'] = corr_df.apply(lambda row: get_dummy_val(row, 'sugar'), axis=1)\n\n# check the result\ncorr_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# group by image\ncorr_df = corr_df.groupby('Image')['Label_Fish', 'Label_Flower', 'Label_Gravel', 'Label_Sugar'].max()\ncorr_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"explore null values"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('There are {} rows with empty segmentation maps.'.format(len(train_df) - train_df.EncodedPixels.count()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plotting a pie chart\nlabels = 'Non-empty', 'Empty'\nsizes = [train_df.EncodedPixels.count(), len(train_df) - train_df.EncodedPixels.count()]\nexplode = (0, 0.1)\n\nfig, ax = plt.subplots(figsize=(6, 6))\nax.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%', shadow=True, startangle=90)\nax.axis('equal')\nax.set_title('Non-empty and Empty Masks')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"now we can explore the correlation between Label_Fish, Label_Flower, Label_Gravel, Label_Sugar columns:\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Find out correlation between columns and plot\ncorrs = np.corrcoef(corr_df.values.T)\nsns.set(font_scale=1)\nsns.set(rc={'figure.figsize':(7,7)})\nhm=sns.heatmap(corrs, cbar = True, annot=True, square = True, fmt = '.2f',\n              yticklabels = ['Fish', 'Flower', 'Gravel', 'Sugar'], \n               xticklabels = ['Fish', 'Flower', 'Gravel', 'Sugar']).set_title('Cloud type correlation heatmap')\n\nfig = hm.get_figure()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, there is **no strong correlation between the types of the clouds** on one image"},{"metadata":{},"cell_type":"markdown","source":"Explore image sizes:\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_img_size(train = True):\n        '''\n    Function to get sizes of images from test and train sets.\n    INPUT:\n        train - indicates whether we are getting sizes of images from train or test set\n    '''\n        if train:\n            path = TRAIN_PATH\n        else:\n            path = TEST_PATH\n            \n        widths = []\n        heights = []\n        \n        imgs = sorted(glob(path + '*.jpg'))\n        \n        max_img = Image.open(imgs[0])\n        min_img = Image.open(imgs[0])\n        \n        for img in range(0, len(imgs)):\n            image = Image.open(imgs[0])\n            width, height = image.size\n            \n            if len(widths) > 0:\n                if width > max(widths):\n                    max_img = image\n                if width < min(widths):\n                    min_img = image\n                    \n            widths.append(width)\n            heights.append(height)\n        \n        return widths, heights, max_img, min_img\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get sizes of images from test and train sets\ntrain_widths, train_heights, max_train, min_train = get_img_size(train = True)\ntest_widths, test_heights, max_test, min_test = get_img_size(train = False)\n\nprint('Maximum width for training set is {}'.format(max(train_widths)))\nprint('Minimum width for training set is {}'.format(min(train_widths)))\nprint('Maximum height for training set is {}'.format(max(train_heights)))\nprint('Minimum height for training set is {}'.format(min(train_heights)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Maximum width for test set is {}'.format(max(test_widths)))\nprint('Minimum width for test set is {}'.format(min(test_widths)))\nprint('Maximum height for test set is {}'.format(max(test_heights)))\nprint('Minimum height for test set is {}'.format(min(test_heights)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"okay so sizes of all the images are same. It does make our life a lot easier :)"},{"metadata":{},"cell_type":"markdown","source":"lets look at some images now!\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# helper function to get a string of labels for the picture\ndef get_labels(image_id):\n    ''' Function to get the labels for the image by name'''\n    im_df = train_df[train_df['Image'] == image_id].fillna('-1')\n    im_df = im_df[im_df['EncodedPixels'] != '-1'].groupby('Label').count()\n    \n    index = im_df.index\n    all_labels = ['Fish', 'Flower', 'Gravel', 'Sugar']\n    \n    labels = ''\n    \n    for label in all_labels:\n        if label in index:\n            labels = labels + ' ' + label\n    \n    return labels\n\n# function to plot a grid of images and their labels\ndef plot_training_images(width = 5, height = 2):\n    \"\"\"\n    Function to plot grid with several examples of cloud images from train set.\n    INPUT:\n        width - number of images per row\n        height - number of rows\n\n    OUTPUT: None\n    \"\"\"\n    \n    # get a list of images from training set\n    images = sorted(glob(TRAIN_PATH + '*.jpg'))\n    \n    fig, axs = plt.subplots(height, width, figsize=(width * 3, height * 3))\n    \n    # create a list of random indices \n    rnd_indices = rnd_indices = [np.random.choice(range(0, len(images))) for i in range(height * width)]\n    \n    for im in range(0, height * width):\n        # open image with a random index\n        image = Image.open(images[rnd_indices[im]])\n        \n        i = im // width\n        j = im % width\n        \n        # plot the image\n        axs[i,j].imshow(image) #plot the data\n        axs[i,j].axis('off')\n        axs[i,j].set_title(get_labels(images[rnd_indices[im]].split('/')[-1]))\n\n    # set suptitle\n    plt.suptitle('Sample images from the train set')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_training_images()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visualize with segmentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"def rle_to_mask(rle_string, width, height):\n    '''\n    convert RLE(run length encoding) string to numpy array\n\n    Parameters: \n    rle_string (str): string of rle encoded mask\n    height (int): height of the mask\n    width (int): width of the mask\n\n    Returns: \n    numpy.array: numpy array of the mask\n    '''\n    \n    rows, cols = height, width\n    \n    if rle_string == -1:\n        return np.zeros((height, width))\n    else:\n        rle_numbers = [int(num_string) for num_string in rle_string.split(' ')]\n        rle_pairs = np.array(rle_numbers).reshape(-1,2)\n        img = np.zeros(rows*cols, dtype=np.uint8)\n        for index, length in rle_pairs:\n            index -= 1\n            img[index:index+length] = 255\n        img = img.reshape(cols,rows)\n        img = img.T\n        return img\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# I will use imgaug library to visualize the segmentation maps. \n# This library has special helpers for visualization and augmentation of images with segmentation maps. \n# You will see how easy it is to work with segmentation maps with imgaug.\n\nfrom __future__ import print_function\nimport numpy as np\n\ndef valid_imshow_data(data):\n    data = np.asarray(data)\n    if data.ndim == 2:\n        return True\n    elif data.ndim == 3:\n        if 3 <= data.shape[2] <= 4:\n            return True\n        else:\n            print('The \"data\" has 3 dimensions but the last dimension '\n                  'must have a length of 3 (RGB) or 4 (RGBA), not \"{}\".'\n                  ''.format(data.shape[2]))\n            return False\n    else:\n        print('To visualize an image the data must be 2 dimensional or '\n              '3 dimensional, not \"{}\".'\n              ''.format(data.ndim))\n        return False\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# import data augmentation\nimport imgaug as ia\nfrom imgaug import augmenters as iaa\n# import segmentation maps from imgaug\nfrom imgaug.augmentables.segmaps import SegmentationMapOnImage\nimport imgaug.imgaug","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_mask(line_id, shape = (2100, 1400)):\n    '''\n    Function to visualize the image and the mask.\n    INPUT:\n        line_id - id of the line to visualize the masks\n        shape - image shape\n    RETURNS:\n        np_mask - numpy segmentation map\n    '''\n    # replace null values with '-1'\n    im_df = train_df.fillna('-1')\n    \n    # convert rle to mask\n    rle = im_df.loc[line_id]['EncodedPixels']\n    if rle != '-1':\n        np_mask = rle_to_mask(rle, shape[0], shape[1])\n        np_mask = np.clip(np_mask, 0, 1)\n    else:\n        # empty mask\n        np_mask = np.zeros((shape[0],shape[1]), dtype=np.uint8)\n        \n    return np_mask\n\n# helper function to get segmentation mask for an image by filename\ndef get_mask_by_image_id(image_id, label):\n    '''\n    Function to visualize several segmentation maps.\n    INPUT:\n        image_id - filename of the image\n    RETURNS:\n        np_mask - numpy segmentation map\n    '''\n    im_df = train_df[train_df['Image'] == image_id.split('/')[-1]].fillna('-1')\n\n    image = np.asarray(Image.open(image_id))\n\n    rle = im_df[im_df['Label'] == label]['EncodedPixels'].values[0]\n    if rle != '-1':\n        np_mask = rle_to_mask(rle, np.asarray(image).shape[1], np.asarray(image).shape[0])\n        np_mask = np.clip(np_mask, 0, 1)\n    else:\n        # empty mask\n        np_mask = np.zeros((np.asarray(image).shape[0], np.asarray(image).shape[1]), dtype=np.uint8)\n        \n    return np_mask\n\ndef visualize_image_with_mask(line_id):\n    '''\n    Function to visualize the image and the mask.\n    INPUT:\n        line_id - id of the line to visualize the masks\n    '''\n    # replace null values with '-1'\n    im_df = train_df.fillna('-1')\n    \n    # get segmentation mask\n    np_mask = get_mask(line_id)\n    \n    # open the image\n    image = Image.open(TRAIN_PATH + im_df.loc[line_id]['Image'])\n\n    # create segmentation map\n    segmap = SegmentationMapOnImage(np_mask, np_mask.shape, nb_classes=2)\n    \n    # visualize the image and map\n    side_by_side = np.hstack([\n        segmap.draw_on_image(np.asarray(image))\n    ]).reshape(np.asarray(image).shape)\n\n    fig, ax = plt.subplots(figsize=(6, 4))\n    ax.axis('off')\n    plt.title(im_df.loc[line_id]['Label'])\n    \n    ax.imshow(side_by_side)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visualize sample masks:"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_image_with_mask(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_image_with_mask(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_training_images_and_masks(n_images = 3):\n    '''\n    Function to plot several random images with segmentation masks.\n    INPUT:\n        n_images - number of images to visualize\n    '''\n    \n    # get a list of images from training set\n    images = sorted(glob(TRAIN_PATH + '*.jpg'))\n    \n    fig, ax = plt.subplots(n_images, 4, figsize=(20, 10))\n    \n    # create a list of random indices \n    rnd_indices = [np.random.choice(range(0, len(images))) for i in range(n_images)]\n    \n    for im in range(0, n_images):\n        # open image with a random index\n        image = Image.open(images[rnd_indices[im]])\n        \n        # get segmentation masks\n        fish = get_mask_by_image_id(images[rnd_indices[im]], 'Fish')\n        flower = get_mask_by_image_id(images[rnd_indices[im]], 'Flower')\n        gravel = get_mask_by_image_id(images[rnd_indices[im]], 'Gravel')\n        sugar = get_mask_by_image_id(images[rnd_indices[im]], 'Sugar')\n        \n        # draw masks on images\n        shape = (np.asarray(image).shape[0], np.asarray(image).shape[1])\n        if np.sum(fish) > 0:\n            segmap_fish = SegmentationMapOnImage(fish, shape=shape, nb_classes=2)\n            im_fish = np.array(segmap_fish.draw_on_image(np.asarray(image))).reshape(np.asarray(image).shape)\n        else:\n            im_fish = np.asarray(image)\n        \n        if np.sum(flower) > 0:\n            segmap_flower = SegmentationMapOnImage(flower, shape=shape, nb_classes=2)\n            im_flower = np.array(segmap_flower.draw_on_image(np.asarray(image))).reshape(np.asarray(image).shape)\n        else:\n            im_flower = np.asarray(image)\n        \n        if np.sum(gravel) > 0:\n            segmap_gravel = SegmentationMapOnImage(gravel, shape=shape, nb_classes=2)\n            im_gravel = np.array(segmap_gravel.draw_on_image(np.asarray(image))).reshape(np.asarray(image).shape)\n        else:\n            im_gravel = np.asarray(image)\n        \n        if np.sum(sugar) > 0:\n            segmap_sugar = SegmentationMapOnImage(sugar, shape=shape, nb_classes=2)\n            im_sugar = np.array(segmap_sugar.draw_on_image(np.asarray(image))).reshape(np.asarray(image).shape)\n        else:\n            im_sugar = np.asarray(image)\n        \n        # plot images and masks\n        ax[im, 0].imshow(im_fish)\n        ax[im, 0].axis('off')\n        ax[im, 0].set_title('Fish')\n        \n        # plot images and masks\n        ax[im, 1].imshow(im_flower)\n        ax[im, 1].axis('off')\n        ax[im, 1].set_title('Flower')\n        \n        # plot images and masks\n        ax[im, 2].imshow(im_gravel)\n        ax[im, 2].axis('off')\n        ax[im, 2].set_title('Gravel')\n        \n        # plot images and masks\n        ax[im, 3].imshow(im_sugar)\n        ax[im, 3].axis('off')\n        ax[im, 3].set_title('Sugar')\n        \n    plt.suptitle('Sample images from the train set')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visualize image grids:"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_training_images_and_masks(n_images = 3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_segmap(image_id):\n    '''\n    Helper function to create a segmentation map for an image by image filename\n    '''\n    # open the image\n    image = np.asarray(Image.open(image_id))\n    \n    # get masks for different classes\n    fish_mask = get_mask_by_image_id(image_id, 'Fish')\n    flower_mask = get_mask_by_image_id(image_id, 'Flower')\n    gravel_mask = get_mask_by_image_id(image_id, 'Gravel')\n    sugar_mask = get_mask_by_image_id(image_id, 'Sugar')\n    \n    # label numpy map with 4 classes\n    segmap = np.zeros((image.shape[0], image.shape[1]), dtype=np.int32)\n    segmap = np.where(fish_mask == 1, 1, segmap)\n    segmap = np.where(flower_mask == 1, 2, segmap)\n    segmap = np.where(gravel_mask == 1, 3, segmap)\n    segmap = np.where(sugar_mask == 1, 4, segmap)\n    \n    # create a segmantation map\n    segmap = SegmentationMapOnImage(segmap, shape=image.shape, nb_classes=5)\n    \n    return segmap\n\ndef draw_labels(image, np_mask, label):\n    '''\n    Function to add labels to the image.\n    '''\n    if np.sum(np_mask) > 0:\n        x,y = 0,0\n        x,y = np.argwhere(np_mask==1)[0]\n                \n        image = imgaug.imgaug.draw_text(image, x, y, label, color=(255, 255, 255), size=50)\n    return image\n\ndef draw_segmentation_maps(image_id):\n    '''\n    Helper function to draw segmantation maps and text.\n    '''\n    # open the image\n    image = np.asarray(Image.open(image_id))\n    \n    # get masks for different classes\n    fish_mask = get_mask_by_image_id(image_id, 'Fish')\n    flower_mask = get_mask_by_image_id(image_id, 'Flower')\n    gravel_mask = get_mask_by_image_id(image_id, 'Gravel')\n    sugar_mask = get_mask_by_image_id(image_id, 'Sugar')\n    \n    # label numpy map with 4 classes\n    segmap = create_segmap(image_id)\n    \n    # draw the map on image\n    image = np.asarray(segmap.draw_on_image(np.asarray(image))).reshape(np.asarray(image).shape)\n    \n    image = draw_labels(image, fish_mask, 'Fish')\n    image = draw_labels(image, flower_mask, 'Flower')\n    image = draw_labels(image, gravel_mask, 'Gravel')\n    image = draw_labels(image, sugar_mask, 'Sugar')\n    \n    return image\n\n# helper function to visualize several segmentation maps on a single image\ndef visualize_several_maps(image_id):\n    '''\n    Function to visualize several segmentation maps.\n    INPUT:\n        image_id - filename of the image\n    '''\n    # open the image\n    image = np.asarray(Image.open(image_id))\n    \n    # draw segmentation maps and labels on image\n    image = draw_segmentation_maps(image_id)\n    \n    # visualize the image and map\n    side_by_side = np.hstack([\n        image\n    ])\n    \n    labels = get_labels(image_id.split('/')[-1])\n\n    fig, ax = plt.subplots(figsize=(15, 7))\n    ax.axis('off')\n    plt.title('Segmentation maps:' + labels)\n    plt.legend()\n    \n    ax.imshow(side_by_side)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create list of all training images filenames\ntrain_fns = sorted(glob(TRAIN_PATH + '*.jpg'))\n\n# generate random index for an image\nnp.random.seed(41)\nrnd_index = np.random.choice(range(len(train_fns)))\n\n# call helper function to visualize the image\nvisualize_several_maps(train_fns[rnd_index])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"create a function to plot sample images with segmentation:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# function to plot a grid of images and their labels and segmantation maps\ndef plot_training_images_and_masks(width = 2, height = 3):\n    \"\"\"\n    Function to plot grid with several examples of cloud images from train set.\n    INPUT:\n        width - number of images per row\n        height - number of rows\n\n    OUTPUT: None\n    \"\"\"\n    \n    # get a list of images from training set\n    images = sorted(glob(TRAIN_PATH + '*.jpg'))\n    \n    fig, axs = plt.subplots(height, width, figsize=(20, 20))\n    \n    # create a list of random indices \n    rnd_indices = rnd_indices = [np.random.choice(range(0, len(images))) for i in range(height * width)]\n    \n    for im in range(0, height * width):\n        # open image with a random index\n        image = Image.open(images[rnd_indices[im]])\n        # draw segmentation maps and labels on image\n        image = draw_segmentation_maps(images[rnd_indices[im]])\n        \n        i = im // width\n        j = im % width\n        \n        # plot the image\n        axs[i,j].imshow(image) #plot the data\n        axs[i,j].axis('off')\n        axs[i,j].set_title(get_labels(images[rnd_indices[im]].split('/')[-1]))\n\n    # set suptitle\n    plt.suptitle('Sample images from the train set')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(40)\nplot_training_images_and_masks()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now comes the modelling part! thankgod ':)"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install -U keras-applications\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Modelling"},{"metadata":{},"cell_type":"markdown","source":"### Imports"},{"metadata":{"trusted":true},"cell_type":"code","source":"import keras_applications\nimport keras\nimport tensorflow as tf\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras_applications.resnext import ResNeXt101","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os, glob\nimport random\nfrom sklearn.model_selection import train_test_split\nimport cv2\nimport numpy as np\nimport pandas as pd\nimport multiprocessing\nfrom copy import deepcopy\nfrom sklearn.metrics import precision_recall_curve, auc\nimport tensorflow.keras as keras\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.callbacks import Callback\nfrom tensorflow.keras.applications.densenet import DenseNet121\nfrom tensorflow.keras.layers import Dense, Flatten\nfrom tensorflow.keras.models import Model, load_model\nfrom tensorflow.keras.utils import Sequence\nfrom albumentations import Compose, VerticalFlip, HorizontalFlip, Rotate, GridDistortion,CenterCrop\nimport matplotlib.pyplot as plt\nfrom IPython.display import Image\nfrom tqdm import tqdm_notebook as tqdm\nfrom numpy.random import seed\nseed(10)\n# from tensorflow import set_random_seed\ntf.random.set_seed(10)\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_imgs_folder = '../input/understanding_cloud_organization/test_images/'\ntrain_imgs_folder = '../input/understanding_cloud_organization/train_images/'\nnum_cores = multiprocessing.cpu_count()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One-hot encoding classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/understanding_cloud_organization/train.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = train_df[~train_df['EncodedPixels'].isnull()]\ntrain_df['Image'] = train_df['Image_Label'].map(lambda x: x.split('_')[0])\ntrain_df['Class'] = train_df['Image_Label'].map(lambda x: x.split('_')[1])\nclasses = train_df['Class'].unique()\ntrain_df = train_df.groupby('Image')['Class'].agg(set).reset_index()\nfor class_name in classes:\n    train_df[class_name] = train_df['Class'].map(lambda x: 1 if class_name in x else 0)\ntrain_df.head()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# dictionary for fast access to ohe vectors\nimg_2_ohe_vector = {img:vec for img, vec in zip(train_df['Image'], train_df.iloc[:, 2:].values)}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Splitting into train/val:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_imgs, val_imgs = train_test_split(train_df['Image'].values, \n                                        test_size=0.1, \n                                        stratify=train_df['Class'].map(lambda x: str(sorted(list(x)))), # sorting present classes in lexicographical order, just to be sure\n                                        random_state=43)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Generator class:"},{"metadata":{"trusted":true},"cell_type":"code","source":"class DataGenenerator(Sequence):\n    def __init__(self, images_list=None, folder_imgs=train_imgs_folder, \n                 batch_size=32, shuffle=True, augmentation=None,\n                 resized_height=224, resized_width=224, num_channels=3):\n        self.batch_size = batch_size\n        self.shuffle = shuffle\n        self.augmentation = augmentation\n        if images_list is None:\n            self.images_list = os.listdir(folder_imgs)\n        else:\n            self.images_list = deepcopy(images_list)\n        self.folder_imgs = folder_imgs\n        self.len = len(self.images_list) // self.batch_size\n        self.resized_height = resized_height\n        self.resized_width = resized_width\n        self.num_channels = num_channels\n        self.num_classes = 4\n        self.is_test = not 'train' in folder_imgs\n        if not shuffle and not self.is_test:\n            self.labels = [img_2_ohe_vector[img] for img in self.images_list[:self.len*self.batch_size]]\n\n    def __len__(self):\n        return self.len\n    \n    def on_epoch_start(self):\n        if self.shuffle:\n            random.shuffle(self.images_list)\n\n    def __getitem__(self, idx):\n        current_batch = self.images_list[idx * self.batch_size: (idx + 1) * self.batch_size]\n        X = np.empty((self.batch_size, self.resized_height, self.resized_width, self.num_channels))\n        y = np.empty((self.batch_size, self.num_classes))\n\n        for i, image_name in enumerate(current_batch):\n            path = os.path.join(self.folder_imgs, image_name)\n            img = cv2.resize(cv2.imread(path), (self.resized_height, self.resized_width)).astype(np.float32)\n            if not self.augmentation is None:\n                augmented = self.augmentation(image=img)\n                img = augmented['image']\n            X[i, :, :, :] = img/255.0\n            if not self.is_test:\n                y[i, :] = img_2_ohe_vector[image_name]\n        return X, y\n\n    def get_labels(self):\n        if self.shuffle:\n            images_current = self.images_list[:self.len*self.batch_size]\n            labels = [img_2_ohe_vector[img] for img in images_current]\n        else:\n            labels = self.labels\n        return np.array(labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"albumentations_train = Compose([\n    VerticalFlip(), HorizontalFlip(), Rotate(limit=20), GridDistortion()\n], p=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"generator instances:"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_generator_train = DataGenenerator(train_imgs, augmentation=albumentations_train)\ndata_generator_train_eval = DataGenenerator(train_imgs, shuffle=False)\ndata_generator_val = DataGenenerator(val_imgs, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"PR-AUC-based Callback\nThe callback would be used:\n\n1. to estimate AUC under precision recall curve for each class,\n2. to early stop after 5 epochs of no improvement in mean PR AUC,\n3. save a model with the best PR AUC in validation,\n4. to reduce learning rate on PR AUC plateau."},{"metadata":{"trusted":true},"cell_type":"code","source":"class PrAucCallback(Callback):\n    def __init__(self, data_generator, num_workers=num_cores, \n                 early_stopping_patience=3, \n                 plateau_patience=3, reduction_rate=0.5,\n                 stage='train', checkpoints_path='checkpoints/'):\n        super(Callback, self).__init__()\n        self.data_generator = data_generator\n        self.num_workers = num_workers\n        self.class_names = ['Fish', 'Flower', 'Sugar', 'Gravel']\n        self.history = [[] for _ in range(len(self.class_names) + 1)] # to store per each class and also mean PR AUC\n        self.early_stopping_patience = early_stopping_patience\n        self.plateau_patience = plateau_patience\n        self.reduction_rate = reduction_rate\n        self.stage = stage\n        self.best_pr_auc = -float('inf')\n        if not os.path.exists(checkpoints_path):\n            os.makedirs(checkpoints_path)\n        self.checkpoints_path = checkpoints_path\n        \n    def compute_pr_auc(self, y_true, y_pred):\n        pr_auc_mean = 0\n        print(f\"\\n{'#'*30}\\n\")\n        for class_i in range(len(self.class_names)):\n            precision, recall, _ = precision_recall_curve(y_true[:, class_i], y_pred[:, class_i])\n            pr_auc = auc(recall, precision)\n            pr_auc_mean += pr_auc/len(self.class_names)\n            print(f\"PR AUC {self.class_names[class_i]}, {self.stage}: {pr_auc:.3f}\\n\")\n            self.history[class_i].append(pr_auc)        \n        print(f\"\\n{'#'*20}\\n PR AUC mean, {self.stage}: {pr_auc_mean:.3f}\\n{'#'*20}\\n\")\n        self.history[-1].append(pr_auc_mean)\n        return pr_auc_mean\n              \n    def is_patience_lost(self, patience):\n        if len(self.history[-1]) > patience:\n            best_performance = max(self.history[-1][-(patience + 1):-1])\n            return best_performance == self.history[-1][-(patience + 1)] and best_performance >= self.history[-1][-1]    \n              \n    def early_stopping_check(self, pr_auc_mean):\n        if self.is_patience_lost(self.early_stopping_patience):\n            self.model.stop_training = True    \n              \n    def model_checkpoint(self, pr_auc_mean, epoch):\n        if pr_auc_mean > self.best_pr_auc:\n            # remove previous checkpoints to save space\n            for checkpoint in glob.glob(os.path.join(self.checkpoints_path, 'classifier_epoch_*')):\n                os.remove(checkpoint)\n        self.best_pr_auc = pr_auc_mean\n        self.model.save(os.path.join(self.checkpoints_path, f'classifier_epoch_{epoch}_val_pr_auc_{pr_auc_mean}.h5'))              \n        print(f\"\\n{'#'*20}\\nSaved new checkpoint\\n{'#'*20}\\n\")\n              \n    def reduce_lr_on_plateau(self):\n        if self.is_patience_lost(self.plateau_patience):\n            new_lr = float(keras.backend.get_value(self.model.optimizer.lr)) * self.reduction_rate\n            keras.backend.set_value(self.model.optimizer.lr, new_lr)\n            print(f\"\\n{'#'*20}\\nReduced learning rate to {new_lr}.\\n{'#'*20}\\n\")\n        \n    def on_epoch_end(self, epoch, logs={}):\n        y_pred = self.model.predict_generator(self.data_generator, workers=self.num_workers)\n        y_true = self.data_generator.get_labels()\n        # estimate AUC under precision recall curve for each class\n        pr_auc_mean = self.compute_pr_auc(y_true, y_pred)\n              \n        if self.stage == 'val':\n            # early stop after early_stopping_patience=4 epochs of no improvement in mean PR AUC\n            self.early_stopping_check(pr_auc_mean)\n\n            # save a model with the best PR AUC in validation\n            self.model_checkpoint(pr_auc_mean, epoch)\n\n            # reduce learning rate on PR AUC plateau\n            self.reduce_lr_on_plateau()            \n        \n    def get_pr_auc_history(self):\n        return self.history","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_metric_callback = PrAucCallback(data_generator_train_eval)\nval_callback = PrAucCallback(data_generator_val, stage='val')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Classifier\n\nDefining a model:"},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.applications.inception_resnet_v2 import InceptionResNetV2\ndef get_model():\n    #base_model = InceptionResNetV2(weights='imagenet', include_top=False, pooling='avg')\n    base_model = model = ResNeXt101(..., backend=tf.keras.backend, layers=tf.keras.layers, weights = 'imagenet', models=tf.keras.models, utils=tf.keras.utils)\n    x = base_model.output\n    y_pred = Dense(4, activation='sigmoid')(x)\n    return Model(inputs=base_model.input, outputs=y_pred)\n\nmodel = get_model()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Initial tuning of the added fully-connected layer"},{"metadata":{"trusted":true},"cell_type":"code","source":"for base_layer in model.layers[:-1]:\n    base_layer.trainable = False\n    \nmodel.compile(optimizer=Adam(lr=0.0001), loss='categorical_crossentropy')\nhistory_0 = model.fit_generator(generator=data_generator_train,\n                              validation_data=data_generator_val,\n                              epochs=1,\n                              callbacks=[train_metric_callback, val_callback],\n                              workers=num_cores,\n                              verbose=1\n                             )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualizing train and val PR AUC"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_with_dots(ax, np_array):\n    ax.scatter(list(range(1, len(np_array) + 1)), np_array, s=50)\n    ax.plot(list(range(1, len(np_array) + 1)), np_array)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"pr_auc_history_train = train_metric_callback.get_pr_auc_history()\npr_auc_history_val = val_callback.get_pr_auc_history()\n\nplt.figure(figsize=(10, 7))\nplot_with_dots(plt, pr_auc_history_train[-1])\nplot_with_dots(plt, pr_auc_history_val[-1])\n\nplt.xlabel('Epoch', fontsize=15)\nplt.ylabel('Mean PR AUC', fontsize=15)\nplt.legend(['Train', 'Val'])\nplt.title('Training and Validation PR AUC', fontsize=20)\nplt.savefig('pr_auc_hist.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Image(\"../input/trained-classifier-epoch-45-resnext101/loss_hist.png\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Image(\"../input/trained-classifier-epoch-45-resnext101/loss_hist_densenet169.png\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Image(\"../input/trained-classifier-epoch-45-resnext101/pr_auc_hist.png\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Image(\"../input/trained-classifier-epoch-45-resnext101/pr_auc_hist_densenet169.png\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Image(\"../input/trained-classifier-epoch-45-resnext101/training_hist_no_aug.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Selecting postprocessing thresholds"},{"metadata":{"trusted":true,"_kg_hide-output":true,"collapsed":true},"cell_type":"code","source":"class_names = ['Fish', 'Flower', 'Sugar', 'Gravel']\ndef get_threshold_for_recall(y_true, y_pred, class_i, recall_threshold=0.94, precision_threshold=0.95, plot=False):\n    precision, recall, thresholds = precision_recall_curve(y_true[:, class_i], y_pred[:, class_i])\n    i = len(thresholds) - 1\n    best_recall_threshold = None\n    while best_recall_threshold is None:\n        next_threshold = thresholds[i]\n        next_recall = recall[i]\n        if next_recall >= recall_threshold:\n            best_recall_threshold = next_threshold\n        i -= 1\n        \n    # consice, even though unnecessary passing through all the values\n    best_precision_threshold = [thres for prec, thres in zip(precision, thresholds) if prec >= precision_threshold][0]\n    \n    if plot:\n        plt.figure(figsize=(10, 7))\n        plt.step(recall, precision, color='r', alpha=0.3, where='post')\n        plt.fill_between(recall, precision, alpha=0.3, color='r')\n        plt.axhline(y=precision[i + 1])\n        recall_for_prec_thres = [rec for rec, thres in zip(recall, thresholds) \n                                 if thres == best_precision_threshold][0]\n        plt.axvline(x=recall_for_prec_thres, color='g')\n        plt.xlabel('Recall')\n        plt.ylabel('Precision')\n        plt.ylim([0.0, 1.05])\n        plt.xlim([0.0, 1.0])\n        plt.legend(['PR curve', \n                    f'Precision {precision[i + 1]: .2f} corresponding to selected recall threshold',\n                    f'Recall {recall_for_prec_thres: .2f} corresponding to selected precision threshold'])\n        plt.title(f'Precision-Recall curve for Class {class_names[class_i]}')\n    return best_recall_threshold, best_precision_threshold\n\ny_pred = model.predict_generator(data_generator_val, workers=num_cores)\ny_true = data_generator_val.get_labels()\nrecall_thresholds = dict()\nprecision_thresholds = dict()\nfor i, class_name in tqdm(enumerate(class_names)):\n    recall_thresholds[class_name], precision_thresholds[class_name] = get_threshold_for_recall(y_true, y_pred, i, plot=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}