{"cells":[{"metadata":{},"cell_type":"markdown","source":"In this kernel,We will go through all the basic steps to build a image segmentation model.For that,we will use many excellent peices of codes written by my fellow kagglers.I personally thank them for sharing their work so that the kaggle community could use it for betterment and gaining knowledge.\n\n**We will go through :**\n- Basic EDA\n- Utility functions\n- Data generators\n- Building our Unet model\n- Evaluation of our model."},{"metadata":{},"cell_type":"markdown","source":"![](https://media.giphy.com/media/UH75adh6PUj16/giphy.gif)"},{"metadata":{},"cell_type":"markdown","source":"#### Also please do an upvote ^ if you like my work.I will be always thankful for your appreciation which motivates me to contribute further to this community."},{"metadata":{},"cell_type":"markdown","source":"## Importing whatever we need..."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom PIL import Image\nimport gc\nimport os\nfrom pathlib import Path\nimport matplotlib.pyplot as plt\nimport keras\nimport cv2\nfrom collections import defaultdict\nfrom skimage.data import imread\nfrom sklearn.model_selection import train_test_split\nfrom keras import backend as K\n\n\npath=Path('../input/understanding_cloud_organization')\nos.listdir(path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### Reading files\ntrain=pd.read_csv(path/'train.csv')\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Basic EDA"},{"metadata":{},"cell_type":"markdown","source":"Before starting with Image segmentation,we will have a basic understanding about the data.This is essential to get good understanding about the problem.\n- We will look at the class distribution.\n- The number of masked and unmasked samples.\n- The sizes of the image and so on.."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train['ImageId']=train['Image_Label'].apply(lambda x : x.split('_')[0])\ntrain['cat']=train['Image_Label'].apply(lambda x : x.split('_')[1])\ntrain[train['EncodedPixels'].notnull()].head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### The number of images of each class"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"cat=train[train['EncodedPixels'].notnull()]['cat'].value_counts()\nplt.bar(cat.index,cat)\nplt.xlabel('category of cloud')\nplt.ylabel('number of masked samples')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- We can see that the classes are not evenly distributed in the training set.The **sugar** type dominates in the patterns observed."},{"metadata":{},"cell_type":"markdown","source":"### Number of masked vs unmasked samples ?"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"x1=train[train['EncodedPixels'].notnull()].shape[0]\nx2=train[train['EncodedPixels'].isnull()].shape[0]\nplt.bar(['has Mask','not Masked'],[x1,x2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Number of missing masks"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['has_mask']= ~pd.isna(train['EncodedPixels'])\ntrain['missing']= pd.isna(train['EncodedPixels'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_nan=train.groupby('ImageId').agg('sum')\ntrain_nan.columns=['No: of Masks','Missing masks']\ntrain_nan['Missing masks'].hist()\n\nmask_count_df=pd.DataFrame(train_nan)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Here you can see that for  most of the images there are two mask which is available.\n- There is no images having all 4 mask as  missing."},{"metadata":{},"cell_type":"markdown","source":"### Number of masks per image"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"mask_count_df = train.groupby('ImageId').agg(np.sum).reset_index()\nmask_count_df.sort_values('has_mask', ascending=False, inplace=True)\nprint(mask_count_df.shape)\nmask_count_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_nan['No: of Masks'].hist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- For mojority of the  images there are 2 masks.\n- There are only less than 250 images having all 4 masks present in it."},{"metadata":{},"cell_type":"markdown","source":"### What are the sizes of images"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"image_size=defaultdict(int)\nimage_file=path/'train_images'\nfor img in image_file.iterdir():\n    img=Image.open(img)\n    image_size[img.size]+=1\n\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_size","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- All images are of the same 2100 * 1400.\n- There are 5546 distinct images in the training set."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"image_size=defaultdict(int)\nimage_file=path/'test_images'\nfor img in image_file.iterdir():\n    img=Image.open(img)\n    image_size[img.size]+=1\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_size","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- All images are of the same 2100 * 1400.\n- There are 3698 distinct images in the test set."},{"metadata":{},"cell_type":"markdown","source":"### What are the number of images with no masks ?"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"no_patterns=0\npatterns=0\n\nfor i in range(0,len(train),4):\n    samples=[x.split('_')[0] for x in train.iloc[i:i+4,0].values]\n    if(samples[0]!=samples[1]!=samples[2]!=samples[3]):\n        raise ValueError\n    labels=train.iloc[i:i+4]['EncodedPixels']\n    if labels.isna().all():\n        no_patterns+=1\n    else:\n        patterns+=1\n        \n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Number of images with patters {} '.format(patterns))\nprint(\"Number of images without patters {} \".format(no_patterns))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = sorted(list(set(train['Image_Label'].apply(lambda x: x.split('_')[1]))))\nprint(labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Utility function"},{"metadata":{"trusted":true},"cell_type":"code","source":"def rle_decode(mask,shape=(1400,2100)):\n    \n    s=mask.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts-=1\n    end=starts+lengths\n    img=np.zeros(shape[0]*shape[1],dtype=np.uint8)\n    for l,m in zip(starts,end):\n        img[l:m]=1\n    return img.reshape(shape[0],shape[1],order='F')\n\n\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_nan[train_nan['No: of Masks']==4].iloc[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"image_name = '00dec6a.jpg'\nimg = imread(str(path)+'/train_images/' + image_name)\n\nfig, ax = plt.subplots(2, 2, figsize=(15, 10))\n\nfor e, label in enumerate(labels):\n    axarr = ax.flat[e]\n    image_label = image_name + '_' + label\n    mask_rle = train.loc[train['Image_Label'] == image_label, 'EncodedPixels'].values[0]\n    try: # label might not be there!\n        mask = rle_decode(mask_rle)\n    except:\n        mask = np.zeros((1400, 2100))\n    axarr.axis('off')\n    axarr.imshow(img)\n    axarr.imshow(mask, alpha=0.5, cmap='gray')\n    axarr.set_title(label, fontsize=24)\nplt.tight_layout(h_pad=0.1, w_pad=0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Source: https://www.kaggle.com/paulorzp/rle-functions-run-lenght-encode-decode\n> Unhide the below code to see the function"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"\n\ndef np_resize(img, input_shape):\n    \"\"\"\n    Reshape a numpy array, which is input_shape=(height, width), \n    as opposed to input_shape=(width, height) for cv2\n    \"\"\"\n    height, width = input_shape\n    return cv2.resize(img, (width, height))\n    \ndef mask2rle(img):\n    '''\n    img: numpy array, 1 - mask, 0 - background\n    Returns run length as string formated\n    '''\n    pixels= img.T.flatten()\n    pixels = np.concatenate([[0], pixels, [0]])\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n    return ' '.join(str(x) for x in runs)\ndef rle2mask(rle, input_shape):\n    width, height = input_shape[:2]\n    \n    mask= np.zeros( width*height ).astype(np.uint8)\n    \n    array = np.asarray([int(x) for x in rle.split()])\n    starts = array[0::2]\n    lengths = array[1::2]\n\n    current_position = 0\n    for index, start in enumerate(starts):\n        mask[int(start):int(start+lengths[index])] = 1\n        current_position += lengths[index]\n        \n    return mask.reshape(height, width).T\n\ndef build_masks(rles, input_shape, reshape=None):\n    depth = len(rles)\n    if reshape is None:\n        masks = np.zeros((*input_shape, depth))\n    else:\n        masks = np.zeros((*reshape, depth))\n    \n    for i, rle in enumerate(rles):\n        if type(rle) is str:\n            if reshape is None:\n                masks[:, :, i] = rle2mask(rle, input_shape)\n            else:\n                mask = rle2mask(rle, input_shape)\n                reshaped_mask = np_resize(mask, reshape)\n                masks[:, :, i] = reshaped_mask\n    \n    return masks\n\ndef build_rles(masks, reshape=None):\n    width, height, depth = masks.shape\n    \n    rles = []\n    \n    for i in range(depth):\n        mask = masks[:, :, i]\n        \n        if reshape:\n            mask = mask.astype(np.float32)\n            mask = np_resize(mask, reshape).astype(np.int64)\n        \n        rle = mask2rle(mask)\n        rles.append(rle)\n        \n    return rles","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Generator"},{"metadata":{},"cell_type":"markdown","source":"In order to train your model, you will ideally need to generate batches of images to feed it. While you do this, you may want to perform common operations across all these images — Operations like rescaling, rotations, crops and shifts, etc. This is called data augmentation. In fact, one very common practice is to resize all images to a one shape, to make the training process uniform.\n\nNote that data augmentation does not change your image — It simply creates another representation of the same image. Imagine if someone took a picture of you, and then rotated that picture by some angle. These are two different pictures, but the object of the picture [you] does not change.\n\nTo achieve this, we use Keras’s ImageDataGenerator.\n\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"We make our Datagenerator class to inherit properties from **keras.utils.sequence** so that we can leverage nice functionalities like multicore processing.This ensures that Datageneration is not an overhead to the system.\n\n\nWe put as arguments relevant information about the data, such as dimension sizes (e.g. a volume of length 32 will have dim=(32,32,32)), number of channels, number of classes, batch size, or decide whether we want to shuffle our data at generation. We also store important information such as labels and the list of IDs that we wish to generate at each pass.\n\nHere, the method **on_epoch_end** is triggered once at the very beginning as well as at the end of each epoch. If the shuffle parameter is set to True, we will get a new order of exploration at each pass (or just keep a linear exploration scheme otherwise)."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class DataGenerator(keras.utils.Sequence):\n    'Generates data for Keras'\n    def __init__(self, list_IDs, df, target_df=None, mode='fit',\n                 base_path='../input/understanding_cloud_organization/train_images',\n                 batch_size=32, dim=(1400, 2100), n_channels=3, reshape=None,\n                 n_classes=4, random_state=2019, shuffle=True):\n        self.dim = dim\n        self.batch_size = batch_size\n        self.df = df\n        self.mode = mode\n        self.base_path = base_path\n        self.target_df = target_df\n        self.list_IDs = list_IDs\n        self.reshape = reshape\n        self.n_channels = n_channels\n        self.n_classes = n_classes\n        self.shuffle = shuffle\n        self.random_state = random_state\n        \n        self.on_epoch_end()\n\n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        return int(np.floor(len(self.list_IDs) / self.batch_size))\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        # Generate indexes of the batch\n        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n\n        # Find list of IDs\n        list_IDs_batch = [self.list_IDs[k] for k in indexes]\n        \n        X = self.__generate_X(list_IDs_batch)\n        \n        if self.mode == 'fit':\n            y = self.__generate_y(list_IDs_batch)\n            return X, y\n        \n        elif self.mode == 'predict':\n            return X\n\n        else:\n            raise AttributeError('The mode parameter should be set to \"fit\" or \"predict\".')\n        \n    def on_epoch_end(self):\n        'Updates indexes after each epoch'\n        self.indexes = np.arange(len(self.list_IDs))\n        if self.shuffle == True:\n            np.random.seed(self.random_state)\n            np.random.shuffle(self.indexes)\n    \n    def __generate_X(self, list_IDs_batch):\n        'Generates data containing batch_size samples'\n        # Initialization\n        if self.reshape is None:\n            X = np.empty((self.batch_size, *self.dim, self.n_channels))\n        else:\n            X = np.empty((self.batch_size, *self.reshape, self.n_channels))\n        \n        # Generate data\n        for i, ID in enumerate(list_IDs_batch):\n            im_name = self.df['ImageId'].iloc[ID]\n            img_path = f\"{self.base_path}/{im_name}\"\n            \n            if self.n_channels == 3:\n                img = self.__load_rgb(img_path)\n            else:\n                img = self.__load_grayscale(img_path)\n            \n            if self.reshape is not None:\n                img = np_resize(img, self.reshape)\n            \n            if len(img.shape) == 2:\n                img = np.expand_dims(img, axis=-1)\n            \n            # Store samples\n            X[i,] = img\n\n        return X\n    \n    def __generate_y(self, list_IDs_batch):\n        if self.reshape is None:\n            y = np.empty((self.batch_size, *self.dim, self.n_classes), dtype=int)\n        else:\n            y = np.empty((self.batch_size, *self.reshape, self.n_classes), dtype=int)\n        \n        for i, ID in enumerate(list_IDs_batch):\n            im_name = self.df['ImageId'].iloc[ID]\n            image_df = self.target_df[self.target_df['ImageId'] == im_name]\n            \n            rles = image_df['EncodedPixels'].values\n            \n            if self.reshape is not None:\n                masks = build_masks(rles, input_shape=self.dim, reshape=self.reshape)\n            else:\n                masks = build_masks(rles, input_shape=self.dim)\n            \n            y[i, ] = masks\n\n        return y\n    \n    def __load_grayscale(self, img_path):\n        img = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)\n        img = img.astype(np.float32) / 255.\n        img = np.expand_dims(img, axis=-1)\n\n        return img\n    \n    def __load_rgb(self, img_path):\n        img = cv2.imread(img_path)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        img = img.astype(np.float32) / 255.\n\n        return img\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Unhide code above to see the DataGenerator class.^"},{"metadata":{},"cell_type":"markdown","source":"## Metrics to Evaluate your Semantic Segmentation Model"},{"metadata":{},"cell_type":"markdown","source":"Dice Coefficient (F1 Score)\n\nSimply put, the Dice Coefficient is 2 * the Area of Overlap divided by the total number of pixels in both images.\n\n![](https://miro.medium.com/max/429/1*yUd5ckecHjWZf6hGrdlwzA.png)"},{"metadata":{},"cell_type":"markdown","source":"If Area of Overlap is 95 \nTotal pixels is 200\n\n(2 * Area of Overlap)/(total pixels combined) = 95*2/200 = 0.95\n\nSource for bce_dice_loss: https://lars76.github.io/neural-networks/object-detection/losses-for-segmentation/"},{"metadata":{"trusted":true},"cell_type":"code","source":"def dice_coef(y_true, y_pred, smooth=1):\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = K.sum(y_true_f * y_pred_f)\n    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we will split our data to train and  validation data,so as to train and validate our model before submitting it to the competition.\nWe will define a BATCH_SIZE of 32,to make the model 32 samples per iteration."},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 32\n\ntrain_idx, val_idx = train_test_split(\n    mask_count_df.index, random_state=2019, test_size=0.15\n)\n\ntrain_generator = DataGenerator(\n    train_idx, \n    df=mask_count_df,\n    target_df=train,\n    batch_size=BATCH_SIZE,\n    reshape=(256, 384),\n    n_channels=3,\n    n_classes=4\n)\n\nval_generator = DataGenerator(\n    val_idx, \n    df=mask_count_df,\n    target_df=train,\n    batch_size=BATCH_SIZE, \n    reshape=(256, 384),\n    n_channels=3,\n    n_classes=4\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Download UNET"},{"metadata":{"_kg_hide-output":true,"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"! pip install segmentation-models","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/qubvel/segmentation_models/master/images/logo.png)"},{"metadata":{},"cell_type":"markdown","source":"Segmentation models is python library with Neural Networks for Image Segmentation based on Keras and Tensorflow Keras frameworks.\n\nThe main features of this library are:\n\n-    High level API (just two lines of code to create model for segmentation)\n-    4 models architectures for binary and multi-class image segmentation (including legendary Unet)\n-    25 available backbones for each architecture\n-    All backbones have pre-trained weights for faster and better convergence\n-    Helpful segmentation losses (Jaccard, Dice, Focal) and metrics (IoU, F-score)\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"from segmentation_models import Unet\nfrom segmentation_models.backbones import get_preprocessing\n\n# LOAD UNET WITH PRETRAINING FROM IMAGENET\npreprocess = get_preprocessing('resnet34') # for resnet, img = (img-110.0)/1.0\nmodel = Unet('resnet34', input_shape=(256, 384, 3), classes=4, activation='sigmoid')\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=[dice_coef])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model Summary"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fit Generator"},{"metadata":{},"cell_type":"markdown","source":"Now we will make use of the fit_generator() method to fit our data.We will only run 5 epochs.You can change it fit your requirements."},{"metadata":{"trusted":true},"cell_type":"code","source":"history=model.fit_generator(train_generator,validation_data=val_generator,epochs=4,verbose=3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluating our model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# PLOT TRAINING\nplt.figure(figsize=(15,5))\nplt.plot(range(history.epoch[-1]+1),history.history['val_dice_coef'],label='val_dice_coef')\nplt.plot(range(history.epoch[-1]+1),history.history['dice_coef'],label='trn_dice_coef')\nplt.title('Training Accuracy'); plt.xlabel('Epoch'); plt.ylabel('Dice_coef');plt.legend(); \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Kernel under construction ! "},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}