{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Understanding Clouds"},{"metadata":{},"cell_type":"markdown","source":"## General Information\n\nIf you like my notes please upvote this Kernel. Furthermore, if you find some mistakes or any problems and any suggestions, please leave a comment in the comments section to help me out. Many thanks in advance!"},{"metadata":{},"cell_type":"markdown","source":"### Change log\n- V34: PR AUC Callback \n- V30:vgg16, 50 epochs, threshold = 0.5\n- V26 : Droped some augmentations, 50 epochs, threshold = 0.5\n- V20: Added new augmentataions, optimize thresholds,\n- PSPNet\n- Gray Images + Unet1 ; 0.48\n- Unet + resnet50 + dice_conf, epochs up to 35, added some albumentions tools \n- Unet + resnet34 + sm.losses.bce_jaccard_loss\n- Version 11: Using segmentation-models: Linknet + vgg16 \n- Version 8: Using segmentation-models: Unet + resnet34\n"},{"metadata":{},"cell_type":"markdown","source":"## References\n\n- Most idea from [Satellite Clouds: U-Net with ResNet Boilerplate](https://www.kaggle.com/xhlulu/satellite-clouds-u-net-with-resnet-boilerplate)\n- Some from [segmentation in pytorch using convenient tools](https://www.kaggle.com/artgor/segmentation-in-pytorch-using-convenient-tools/notebook)"},{"metadata":{},"cell_type":"markdown","source":"## Score Log\n\n- [segmentation-moddels](https://github.com/qubvel/segmentation_models#models-and-backbones), Unet, resnet34, epochs = 30, BATCH_SIZE=32, score = 0.562"},{"metadata":{"_uuid":"30b5bb6f-4e41-4017-b5a1-c9319cb7b5fd","_cell_guid":"ebd38bdb-7ef0-49ce-95c3-5c6ba300d902","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom skimage.data import imread\nimport matplotlib.pyplot as plt\nimport cv2\nimport json\n\nimport multiprocessing\n\nimport albumentations as albu\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import precision_recall_curve, auc\n\nimport keras\nfrom keras import backend as K\nfrom keras.losses import binary_crossentropy\nfrom keras.layers import Input\nfrom keras.layers.convolutional import Conv2D, Conv2DTranspose\nfrom keras.layers.pooling import MaxPooling2D\nfrom keras.layers.merge import concatenate\nfrom keras.losses import binary_crossentropy\nfrom keras.callbacks import Callback, ModelCheckpoint, EarlyStopping, ReduceLROnPlateau\nfrom keras.models import Model\nfrom keras.optimizers import Adam\n\n!pip install segmentation-models --quiet\nimport segmentation_models as sm\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os, glob\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_cpu_cores = multiprocessing.cpu_count()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f7747fb0-b6c2-45e6-a00a-cd881e74c2b6","_cell_guid":"5d35c9dd-074c-4ca7-9895-1755e78f5ba5","trusted":true},"cell_type":"markdown","source":"### Utilities Functions\n\n- `np_resize` : Reize the image to a specified shape\n- `mask2rle` : Convert a mask image array to a Run Length Encoding \n- `rle2mask` : Convert a Run length encoding to a mask image array \n- `build_masks`\n- `build_rles`"},{"metadata":{"_uuid":"35e3efd3-4e6a-4bad-82cb-a619a269a855","_cell_guid":"f4e18b81-e4fc-4bc8-b47f-a164e457e8fe","trusted":true},"cell_type":"code","source":"def np_resize(img, input_shape,graystyle=False):\n    \"\"\"\n    Reshape a numpy array, which is input_shape=(height, width), \n    as opposed to input_shape=(width, height) for cv2\n    \"\"\"\n\n    height, width = input_shape\n    resized_img = cv2.resize(img, (width, height))\n    \n    # keep dimension\n    if graystyle:\n        resized_img = resized_img[..., None]\n        \n    return resized_img\n    \ndef mask2rle(img):\n    '''\n    img: a mask image, numpy array, 1 - mask, 0 - background\n    Returns run length as string formated\n    \n    img.T.flatten()\n    image(width x height x channel), \n    from width -> height -> channel flatten a one dimension array \n    \n    '''\n    pixels= img.T.flatten() \n    pixels = np.concatenate([[0], pixels, [0]])  # add 0 to the beginning and end of array\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n    return ' '.join(str(x) for x in runs)\n\ndef rle2mask(rle, input_shape):\n    width, height = input_shape[:2]\n    \n    mask= np.zeros( width*height ).astype(np.uint8)\n    \n    array = np.asarray([int(x) for x in rle.split()])\n    starts = array[0::2]\n    lengths = array[1::2]\n\n    current_position = 0\n    for index, start in enumerate(starts):\n        mask[int(start):int(start+lengths[index])] = 1\n        current_position += lengths[index]\n        \n    return mask.reshape(height, width).T\n\ndef build_masks(rles, input_shape, reshape=None):\n    depth = len(rles)\n    \n    if reshape is None:\n        masks = np.zeros((*input_shape, depth))\n    else:\n        masks = np.zeros((*reshape, depth))\n    \n    for i, rle in enumerate(rles):\n        if type(rle) is str:\n            if reshape is None:\n                masks[:, :, i] = rle2mask(rle, input_shape)\n            else:\n                mask = rle2mask(rle, input_shape)\n                reshaped_mask = np_resize(mask, reshape)\n                masks[:, :, i] = reshaped_mask\n    \n    return masks\n\ndef build_rles(masks, reshape=None):\n    width, height, depth = masks.shape\n    \n    rles = []\n    \n    for i in range(depth):\n        mask = masks[:, :, i]\n        \n        if reshape:\n            mask = mask.astype(np.float32)\n            mask = np_resize(mask, reshape).astype(np.int64)\n        \n        rle = mask2rle(mask)\n        rles.append(rle)\n        \n    return rles","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Convert a image (width x height x channel) three dimension array to a one dimension array**\n\nExample: \n\nThere is a image (4x2x3 = 24 pixles)\n\n![https://github.com/gsaneryeeb/ARTS/blob/master/data/4x2x3.png?raw=true](https://github.com/gsaneryeeb/ARTS/blob/master/data/4x2x3.png?raw=true)\n\n```python\nimage = np.array([[[1,9,17],\n               [5,13,21]],\n              \n              [[2,10,18],\n               [6,14,22]],\n              \n              [[3,11,19],\n               [7,15,23]],\n              \n              [[4,12,20],\n               [8,16,24]]])\n```\n\n[flatten()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.flatten.html) Return a copy of the array collapsed in to one dimension.\n\n```\nimage.flatten()\n\narray([ 1,  9, 17,  5, 13, 21,  2, 10, 18,  6, 14, 22,  3, 11, 19,  7, 15,\n       23,  4, 12, 20,  8, 16, 24])\n```\n\nIn Run length encoding, We need to expand a image into a one-dimensional array in order of width, height and channel. So we use the function `image.T.flatten()`\n\n```\nimage.T.flatten()\n\narray([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,\n       18, 19, 20, 21, 22, 23, 24])\n```"},{"metadata":{"_uuid":"e4040411-320d-4e3d-893d-326db526bb82","_cell_guid":"26b5eb55-1764-45ca-9e3e-96356535e1d6","trusted":true},"cell_type":"markdown","source":"## Loss function\n$$\n\\mathrm{DC}=\\frac{2 T P}{2 T P+F P+F N}=\\frac{2|X \\cap Y|}{|X|+|Y|}\n$$\n\n$$\n\\begin{array}{c}{\\mathrm{DL}(p, \\hat{p})=1-\\frac{2 p \\hat{p}+1}{p+\\hat{p}+1}} \\\\ {\\text { where } p \\in\\{0,1\\} \\text { and } 0 \\leq \\hat{p} \\leq 1}\\end{array}\n$$\n \nLoss Function: $$B C E+D i c e$$"},{"metadata":{"_uuid":"87120b53-a44b-4075-a233-f42c846c6689","_cell_guid":"bdee9285-1815-41aa-af9a-8eeee31cadef","trusted":true},"cell_type":"code","source":"def dice_coef(y_true, y_pred, smooth=1):\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = K.sum(y_true_f * y_pred_f)\n    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n\ndef dice_loss(y_true, y_pred):\n    smooth = 1.\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = y_true_f * y_pred_f\n    score = (2. * K.sum(intersection) + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n    return 1. - score\n\ndef bce_dice_loss(y_true, y_pred):\n    return binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d9d7e833-2208-4e25-ba00-4eb6fd4cfebb","_cell_guid":"4f7344ff-0507-4935-83f9-c679719f6eae","trusted":true},"cell_type":"markdown","source":"## Preprocessing and EDA"},{"metadata":{"_uuid":"d63bcd45-57c5-4c06-8bd8-54a4188c36bb","_cell_guid":"c4fccdeb-90d9-481f-bff0-d55172942b54","trusted":true},"cell_type":"code","source":"train_images = os.listdir('../input/train_images')\nprint(len(train_images))\n\ntest_images = os.listdir('../input/test_images')\nprint(len(test_images))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Split Image_Label into ImageId and Label\n- Add `hasMask` column"},{"metadata":{"_uuid":"73d456a9-df97-4f9b-8562-7b824b11aaac","_cell_guid":"f0f86633-2014-43bf-a4dd-01fcc85c7eb8","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/train.csv')\ntrain_df.head()\n\n# Split Image_Label into ImageId and Label\ntrain_df['ImageId'] = train_df['Image_Label'].apply(lambda x : x.split('_')[0])\ntrain_df['Label'] = train_df['Image_Label'].apply(lambda x : x.split('_')[1])\ntrain_df['hasMask'] = ~ train_df['EncodedPixels'].isna()\n\nprint(train_df.shape)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"train_df.loc[train_df['EncodedPixels'].isnull() == False, 'Image_Label'].apply(lambda x: x.split('_')[1]).value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Total training images is 5546, every images have four masks.(Fish, Flower, Gravel, Sugar), Total training dataset is 5546 * 4 = 22184"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.loc[train_df['EncodedPixels'].isnull() == False, 'Image_Label'].apply(lambda x: x.split('_')[0]).value_counts().value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Only 266 images have all four masks."},{"metadata":{"_uuid":"0e1821ef-e2e6-48f0-98f6-2669019b3d43","_cell_guid":"aa3d814f-dea7-4a1f-982d-e47bf1f8b7a2","trusted":true},"cell_type":"code","source":"mask_count_df = train_df.groupby('ImageId').agg(np.sum).reset_index()\nmask_count_df.sort_values('hasMask', ascending=False, inplace=True)\nprint(mask_count_df.shape)\nmask_count_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4e111e4c-6a7e-4bd9-a462-5710df9ce0c3","_cell_guid":"96f5489f-2460-49bd-8910-8a3ca264fd19","trusted":true},"cell_type":"code","source":"submission_df = pd.read_csv('../input/sample_submission.csv')\nsubmission_df['ImageId'] = submission_df['Image_Label'].apply(lambda x : x.split('_')[0])\ntest_images = pd.DataFrame(submission_df['ImageId'].unique(), columns=['ImageId'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## One-hot encoding classed"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Mark: `train_ohe_df = train_ohe_df.groupby('ImageId')['Label'].agg(set).reset_index()` \n\n1. Groupby *ImageId*\n2. Set *Label* value"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ohe_df = train_df[~train_df['EncodedPixels'].isnull()]\nclasses = train_ohe_df['Label'].unique()\ntrain_ohe_df = train_ohe_df.groupby('ImageId')['Label'].agg(set).reset_index()\nfor class_name in classes:\n    train_ohe_df[class_name] = train_ohe_df['Label'].map(lambda x: 1 if class_name in x else 0)\nprint(train_ohe_df.shape)\ntrain_ohe_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# dictionary for fast access to ohe vectors\n# key: ImageId\n# value: ohe value\n# {'0011165.jpg': array([1, 1, 0, 0]),\n# '002be4f.jpg': array([1, 1, 1, 0]),...}\nimg_to_ohe_vector = {img: vec for img, vec in zip(train_ohe_df['ImageId'], train_ohe_df.iloc[:, 2:].values)}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Stratified split into train and val"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ohe_df['Label'].map(lambda x: str(sorted(list(x))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_idx, val_idx = train_test_split(mask_count_df.index, \n                                      random_state=42,\n                                      stratify = train_ohe_df['Label'].map(lambda x: str(sorted(list(x)))),# sorting present classes in lexicographical order, just to be sure\n                                      test_size = 0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('train length:',len(train_idx))\nprint('val length:',len(val_idx))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"400dbc3b-68fd-4926-830d-d3c800bc21d9","_cell_guid":"96531f4e-bd67-4d7c-babc-3d4e1ab44286","trusted":true},"cell_type":"markdown","source":"## Data Generator"},{"metadata":{"_uuid":"2c759521-7b53-4a2c-807a-0f144f9ee3b9","_cell_guid":"0210b661-316e-4c10-b137-5b1747989ccf","trusted":true},"cell_type":"code","source":"class DataGenerator(keras.utils.Sequence):\n    'Generates data for Keras'\n    def __init__(self, list_IDs, df, target_df=None, mode='fit',\n                 base_path='../input/train_images',\n                 batch_size=32, dim=(1400, 2100), n_channels=3, reshape=None,\n                 augment=False, n_classes=4, random_state=42, shuffle=True, graystyle=False):\n        self.dim = dim\n        self.batch_size = batch_size\n        self.df = df\n        self.mode = mode\n        self.base_path = base_path\n        self.target_df = target_df\n        self.list_IDs = list_IDs\n        self.reshape = reshape\n        self.n_channels = n_channels\n        self.augment = augment\n        self.n_classes = n_classes\n        self.shuffle = shuffle\n        self.random_state = random_state\n        self.graystyle = graystyle\n        \n        self.on_epoch_end()\n        np.random.seed(self.random_state)\n\n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        return int(np.floor(len(self.list_IDs) / self.batch_size))\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        # Generate indexes of the batch\n        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n\n        # Find list of IDs\n        list_IDs_batch = [self.list_IDs[k] for k in indexes]\n        \n        X = self.__generate_X(list_IDs_batch)\n        \n        if self.mode == 'fit':\n            y = self.__generate_y(list_IDs_batch)\n            \n            if self.augment:\n                X, y = self.__augment_batch(X, y)\n            \n            return X, y\n        \n        elif self.mode == 'predict':\n            return X\n\n        else:\n            raise AttributeError('The mode parameter should be set to \"fit\" or \"predict\".')\n        \n    def on_epoch_end(self):\n        'Updates indexes after each epoch'\n        self.indexes = np.arange(len(self.list_IDs))\n        if self.shuffle == True:\n            np.random.seed(self.random_state)\n            np.random.shuffle(self.indexes)\n    \n    def __generate_X(self, list_IDs_batch):\n        'Generates data containing batch_size samples'\n        # Initialization\n        if self.reshape is None:\n            X = np.empty((self.batch_size, *self.dim, self.n_channels))\n        else:\n            X = np.empty((self.batch_size, *self.reshape, self.n_channels))\n        \n        # Generate data\n        for i, ID in enumerate(list_IDs_batch):\n            im_name = self.df['ImageId'].iloc[ID]\n            img_path = f\"{self.base_path}/{im_name}\"\n            img = cv2.imread(img_path)\n            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n            img = img.astype(np.float32) / 255.\n            \n            if self.reshape is not None:\n                img = np_resize(img, self.reshape)\n            \n            # Store samples\n            X[i,] = img\n\n        return X\n    \n    def __generate_y(self, list_IDs_batch):\n        if self.reshape is None:\n            y = np.empty((self.batch_size, *self.dim, self.n_classes), dtype=int)\n        else:\n            y = np.empty((self.batch_size, *self.reshape, self.n_classes), dtype=int)\n        \n        for i, ID in enumerate(list_IDs_batch):\n            im_name = self.df['ImageId'].iloc[ID]\n            image_df = self.target_df[self.target_df['ImageId'] == im_name]\n            \n            rles = image_df['EncodedPixels'].values\n            \n            if self.reshape is not None:\n                masks = build_masks(rles, input_shape=self.dim, reshape=self.reshape)\n            else:\n                masks = build_masks(rles, input_shape=self.dim)\n            \n            y[i, ] = masks\n\n        return y\n    \n    \n    def __load_rgb(self, img_path):\n        img = cv2.imread(img_path)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        img = img.astype(np.float32) / 255.\n\n        return img\n    \n    def __random_transform(self, img, masks):\n        composition = albu.Compose([\n            albu.HorizontalFlip(p=0.5),\n            albu.VerticalFlip(p=0.5),\n            # albu.RandomRotate90(p=1),\n            # albu.RandomBrightness(),\n            #albu.ElasticTransform(p=1,alpha=120,sigma=120*0.05,alpha_affine=120*0.03),\n            albu.GridDistortion(p=0.5)])\n            #albu.OpticalDistortion(p=1, distort_limit=2, shift_limit=0.5),\n            #albu.ShiftScaleRotate(scale_limit=0.5,rotate_limit=30, shift_limit=0.1, p=1, border_mode=0)])\n        \n        composed = composition(image=img, mask=masks)\n        aug_img = composed['image']\n        aug_masks = composed['mask']\n        \n        return aug_img, aug_masks\n    \n    def __augment_batch(self, img_batch, masks_batch):\n        for i in range(img_batch.shape[0]):\n            img_batch[i, ], masks_batch[i, ] = self.__random_transform(\n                img_batch[i, ], masks_batch[i, ])\n        \n        return img_batch, masks_batch\n    \n    def get_labels(self):\n        if self.shuffle:\n            images_current = self.list_IDs[:self.len * self.batch_size]\n            labels = [img_to_ohe_vector[img] for img in images_current]\n        else:\n            labels = self.labels\n        return np.array(labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Generator instances"},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 32\nBACKBONE = 'resnet34'\nENCODER_WEIGHTS = 'imagenet'\nACTIVATION = 'sigmoid'\nEPOCHS = 30\nLEARNING_RATE = 3e-4\nHEIGHT =320\nWIDTH = 480\nCHANNELS = 3\nES_PATIENCE = 5\nRLROP_PATIENCE = 3\nDECAY_DROP = 0.5\nN_CLASSES = train_df['Label'].nunique()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"678d72ba-3e33-452b-a7c8-d5a956e9d9e3","_cell_guid":"450ab16e-03af-4cf2-ab40-e12e98595ba4","trusted":true},"cell_type":"code","source":"train_generator = DataGenerator(train_idx, \n                                df=mask_count_df, \n                                target_df=train_df, \n                                batch_size=BATCH_SIZE,\n                                reshape=(HEIGHT, WIDTH),\n                                augment=True,\n                                graystyle=False,\n                                shuffle = True,\n                                n_channels=CHANNELS,\n                                n_classes=N_CLASSES)\n\ntrain_eval_generator = DataGenerator(train_idx, \n                              df=mask_count_df, \n                              target_df=train_df, \n                              batch_size=BATCH_SIZE, \n                              reshape=(HEIGHT, WIDTH),\n                              augment=False,\n                              graystyle=False,\n                              shuffle = False,\n                              n_channels=CHANNELS,\n                              n_classes=N_CLASSES)\n\nval_generator = DataGenerator(val_idx, \n                              df=mask_count_df, \n                              target_df=train_df, \n                              batch_size=BATCH_SIZE, \n                              reshape=(HEIGHT, WIDTH),\n                              augment=False,\n                              graystyle=False,\n                              shuffle = False,\n                              n_channels=CHANNELS,\n                              n_classes=N_CLASSES)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"train_generator lengh is \", len(train_generator))\nprint(\"train_eval_generator lengh is \", len(train_eval_generator))\nprint(\"val_generator lengh is \", len(val_generator))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## PR(Pro)-AUC-based Callback\n\nfrom [https://www.kaggle.com/samusram/cloud-classifier-for-post-processing](https://www.kaggle.com/samusram/cloud-classifier-for-post-processing)\n\nUsing Sklearn Precision Recall Curve and AUC\n\n1. to estimate AUC under precision recall curve for each class,\n2. to early stop after 5 epochs of no improvement in mean PR AUC,\n3. save a model with the best PR AUC in validation,\n4. to reduce learning rate on PR AUC plateau."},{"metadata":{"trusted":true},"cell_type":"code","source":"class PrAucCallback(Callback):\n    def __init__(self, data_generator, num_workers=num_cpu_cores, \n                 early_stopping_patience=5, \n                 plateau_patience=3, reduction_rate=0.5,\n                 stage='train', checkpoints_path='checkpoints/'):\n        super(Callback, self).__init__()\n        self.data_generator = data_generator\n        self.num_workers = num_workers\n        self.class_names = ['Fish', 'Flower', 'Sugar', 'Gravel']\n        self.history = [[] for _ in range(len(self.class_names) + 1)] # to store per each class and also mean PR AUC\n        self.early_stopping_patience = early_stopping_patience\n        self.plateau_patience = plateau_patience\n        self.reduction_rate = reduction_rate\n        self.stage = stage\n        self.best_pr_auc = -float('inf')\n        if not os.path.exists(checkpoints_path):\n            os.makedirs(checkpoints_path)\n        self.checkpoints_path = checkpoints_path\n    \n    \n    def compute_pr_auc(self, y_true, y_pred):\n        pr_auc_mean = 0\n        print(f\"\\n{'#'*30}\\n\")\n        for class_i in range(len(self.class_names)):\n            precision, recall, _ = precision_recall_curve(y_true[:, class_i], y_pred[:, class_i])\n            pr_auc = auc(recall, precision)\n            pr_auc_mean += pr_auc/len(self.class_names)\n            print(f\"PR AUC {self.class_names[class_i]}, {self.stage}: {pr_auc:.3f}\\n\")\n            self.history[class_i].append(pr_auc)        \n        print(f\"\\n{'#'*20}\\n PR AUC mean, {self.stage}: {pr_auc_mean:.3f}\\n{'#'*20}\\n\")\n        self.history[-1].append(pr_auc_mean)\n        return pr_auc_mean\n              \n    def is_patience_lost(self, patience):\n        if len(self.history[-1]) > patience:\n            best_performance = max(self.history[-1][-(patience + 1):-1])\n            return best_performance == self.history[-1][-(patience + 1)] and best_performance >= self.history[-1][-1]    \n              \n    def early_stopping_check(self, pr_auc_mean):\n        if self.is_patience_lost(self.early_stopping_patience):\n            self.model.stop_training = True    \n              \n    def model_checkpoint(self, pr_auc_mean, epoch):\n        if pr_auc_mean > self.best_pr_auc:\n            # remove previous checkpoints to save space\n            for checkpoint in glob.glob(os.path.join(self.checkpoints_path, 'classifier_epoch_*')):\n                os.remove(checkpoint)\n        self.best_pr_auc = pr_auc_mean\n        self.model.save(os.path.join(self.checkpoints_path, f'classifier_epoch_{epoch}_val_pr_auc_{pr_auc_mean}.h5'))              \n        print(f\"\\n{'#'*20}\\nSaved new checkpoint\\n{'#'*20}\\n\")\n              \n    def reduce_lr_on_plateau(self):\n        if self.is_patience_lost(self.plateau_patience):\n            new_lr = float(keras.backend.get_value(self.model.optimizer.lr)) * self.reduction_rate\n            keras.backend.set_value(self.model.optimizer.lr, new_lr)\n            print(f\"\\n{'#'*20}\\nReduced learning rate to {new_lr}.\\n{'#'*20}\\n\")\n        \n    def on_epoch_end(self, epoch, logs={}):\n        y_pred = self.model.predict_generator(self.data_generator,\n                                              verbose=1,\n                                              workers=self.num_workers)\n        y_true = self.data_generator.get_labels()\n        # estimate AUC under precision recall curve for each class\n        pr_auc_mean = self.compute_pr_auc(y_true, y_pred)\n              \n        if self.stage == 'val':\n            # early stop after early_stopping_patience=4 epochs of no improvement in mean PR AUC\n            self.early_stopping_check(pr_auc_mean)\n\n            # save a model with the best PR AUC in validation\n            self.model_checkpoint(pr_auc_mean, epoch)\n\n            # reduce learning rate on PR AUC plateau\n            self.reduce_lr_on_plateau()            \n        \n    def get_pr_auc_history(self):\n        return self.history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Instances callback"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_metric_callback = PrAucCallback(train_eval_generator)\nval_callback = PrAucCallback(val_generator, stage='val')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"91e77dbe-9590-4f06-b1fe-e4fcb9a330bd","_cell_guid":"e501e370-3b4d-4bf0-a45c-b660e92f2830","trusted":true},"cell_type":"markdown","source":"## Model Architecture"},{"metadata":{"_uuid":"32f20316-6e77-44e1-b0de-bd125b0a7bdf","_cell_guid":"e13507aa-ff18-4cf8-9c99-f085a01f40a1","trusted":true},"cell_type":"code","source":"def unet1(input_shape):\n    inputs = Input(input_shape)\n\n    c1 = Conv2D(8, (3, 3), activation='elu', padding='same') (inputs)\n    c1 = Conv2D(8, (3, 3), activation='elu', padding='same') (c1)\n    p1 = MaxPooling2D((2, 2), padding='same') (c1)\n\n    c2 = Conv2D(16, (3, 3), activation='elu', padding='same') (p1)\n    c2 = Conv2D(16, (3, 3), activation='elu', padding='same') (c2)\n    p2 = MaxPooling2D((2, 2), padding='same') (c2)\n\n    c3 = Conv2D(32, (3, 3), activation='elu', padding='same') (p2)\n    c3 = Conv2D(32, (3, 3), activation='elu', padding='same') (c3)\n    p3 = MaxPooling2D((2, 2), padding='same') (c3)\n\n    c4 = Conv2D(64, (3, 3), activation='elu', padding='same') (p3)\n    c4 = Conv2D(64, (3, 3), activation='elu', padding='same') (c4)\n    p4 = MaxPooling2D((2, 2), padding='same') (c4)\n\n    c5 = Conv2D(64, (3, 3), activation='elu', padding='same') (p4)\n    c5 = Conv2D(64, (3, 3), activation='elu', padding='same') (c5)\n    p5 = MaxPooling2D((2, 2), padding='same') (c5)\n\n    c55 = Conv2D(128, (3, 3), activation='elu', padding='same') (p5)\n    c55 = Conv2D(128, (3, 3), activation='elu', padding='same') (c55)\n\n    u6 = Conv2DTranspose(64, (2, 2), strides=(2, 2), padding='same') (c55)\n    u6 = concatenate([u6, c5])\n    c6 = Conv2D(64, (3, 3), activation='elu', padding='same') (u6)\n    c6 = Conv2D(64, (3, 3), activation='elu', padding='same') (c6)\n\n    u71 = Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same') (c6)\n    u71 = concatenate([u71, c4])\n    c71 = Conv2D(32, (3, 3), activation='elu', padding='same') (u71)\n    c61 = Conv2D(32, (3, 3), activation='elu', padding='same') (c71)\n\n    u7 = Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same') (c61)\n    u7 = concatenate([u7, c3])\n    c7 = Conv2D(32, (3, 3), activation='elu', padding='same') (u7)\n    c7 = Conv2D(32, (3, 3), activation='elu', padding='same') (c7)\n\n    u8 = Conv2DTranspose(16, (2, 2), strides=(2, 2), padding='same') (c7)\n    u8 = concatenate([u8, c2])\n    c8 = Conv2D(16, (3, 3), activation='elu', padding='same') (u8)\n    c8 = Conv2D(16, (3, 3), activation='elu', padding='same') (c8)\n\n    u9 = Conv2DTranspose(8, (2, 2), strides=(2, 2), padding='same') (c8)\n    u9 = concatenate([u9, c1], axis=3)\n    c9 = Conv2D(8, (3, 3), activation='elu', padding='same') (u9)\n    c9 = Conv2D(8, (3, 3), activation='elu', padding='same') (c9)\n\n    outputs = Conv2D(4, (1, 1), activation='sigmoid') (c9)\n\n    model = Model(inputs=[inputs], outputs=[outputs])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a0973ac4-03c9-423d-a35d-c9d5b3299364","_cell_guid":"bf33a275-d755-43c4-8ee0-8c06a2f2d25a","trusted":true},"cell_type":"markdown","source":"## Training"},{"metadata":{"_uuid":"0110860f-36d8-4eef-b215-6bb4da60d7b1","_cell_guid":"7de48f23-74b1-426e-a4a7-73f2c42d5b73","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"preprocess_input = sm.backbones.get_preprocessing(BACKBONE)\n\nmodel = sm.Unet(encoder_name=BACKBONE,\n                classes=N_CLASSES, \n                input_shape=(HEIGHT,WIDTH,CHANNELS), \n                activation=ACTIVATION)\n\"\"\"\n\nmodel = unet1((320,480,3))\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"earlystopping = EarlyStopping(monitor='val_loss', \n                             mode='min', \n                             patience=ES_PATIENCE,\n                             restore_best_weights=True,\n                             verbose=1)\n\nreduce_lr = ReduceLROnPlateau(monitor='val_loss', \n                              mode='min',\n                              patience=RLROP_PATIENCE,\n                              factor=DECAY_DROP,\n                              min_lr=1e-6,\n                              verbose=1)\n\nmetric_list = [dice_coef]\ncallback_list = [earlystopping, reduce_lr]\noptimizer = Adam(lr = LEARNING_RATE)\n\nmodel.compile(optimizer=optimizer, \n              loss=bce_dice_loss, \n              metrics=metric_list)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"111b088e-b4d0-414e-8ed7-e1aa3461e89d","_cell_guid":"ba34edcb-f127-48c2-ba44-a2e8fedecc9f","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"checkpoint = ModelCheckpoint('model.h5', save_best_only=True)\n\nhistory = model.fit_generator(\n    train_generator,\n    validation_data=val_generator,\n    callbacks=callback_list,\n    epochs=1,\n    verbose=2\n)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6937c4f4-f99d-41cd-bdbf-50e6eff3798c","_cell_guid":"3fc90baa-048b-417a-9ec6-b513feb4f891","trusted":true},"cell_type":"markdown","source":"## Evaluation"},{"metadata":{"_uuid":"d32c85f6-aa0f-4a35-8c38-f2e4bf398509","_cell_guid":"099e6a10-21d0-4ee3-b62f-7447038974ff","trusted":true},"cell_type":"code","source":"with open('history.json', 'w') as f:\n    json.dump(history.history, f)\n\nhistory_df = pd.DataFrame(history.history)\nhistory_df[['loss', 'val_loss']].plot()\nhistory_df[['dice_coef', 'val_dice_coef']].plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualizing train and val PR AUC"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_with_dots(ax, np_array):\n    ax.scatter(list(range(1, len(np_array) + 1)), np_array, s=50)\n    ax.plot(list(range(1, len(np_array) + 1)), np_array)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Training and Validation PR AUC"},{"metadata":{"trusted":true},"cell_type":"code","source":"pr_auc_history_train = train_metric_callback.get_pr_auc_history()\npr_auc_history_val = val_callback.get_pr_auc_history()\n\nplt.figure(figsize=(10, 7))\nplot_with_dots(plt, pr_auc_history_train[-1])\nplot_with_dots(plt, pr_auc_history_val[-1])\n\nplt.xlabel('Epoch', fontsize=15)\nplt.ylabel('Mean PR AUC', fontsize=15)\nplt.legend(['Train', 'Val'])\nplt.title('Training and Validation PR AUC', fontsize=20)\nplt.savefig('pr_auc_hist.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Training and Validation Loss"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10, 7))\nplot_with_dots(plt, history_0.history['loss']+history_1.history['loss'])\nplot_with_dots(plt, history_0.history['val_loss']+history_1.history['val_loss'])\n\nplt.xlabel('Epoch', fontsize=15)\nplt.ylabel('Binary Crossentropy', fontsize=15)\nplt.legend(['Train', 'Val'])\nplt.title('Training and Validation Loss', fontsize=20)\nplt.savefig('loss_hist.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predicting\n\n\n- try threshold = 0.9 size =25000\n\n### Optimize thresholds\n\nUsing predictions on validation dataset and validation label to optimize thresholds"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\"\"\"\nmodel.load_weights('model.h5')\nvalid_masks = []\nencoded_pixels = []\nVALID_BATCH_SIZE = 10\nprobabilities = np.zeros((len(val_idx),350,525)) # Train = 90% train_dataset, Valid = 10% train_dataset.\n\nprint(\"length val_idx =\", len(val_idx))\n#for i in range(0, len(val_idx), VALID_BATCH_SIZE):\nval_gen = DataGenerator(val_idx, \n                        df=mask_count_df, \n                        target_df=train_df, \n                        batch_size=1, \n                        reshape=(320, 480),\n                        augment=False,\n                        graystyle=False,\n                        n_channels=3,\n                        n_classes=4)\n\npred_valid_masks = model.predict_generator(\n    val_gen, \n    workers=1,\n    verbose=1    \n)\n\n\n\n\n    image, masks = batch\n    \n    for mask in masks:\n        if mask.shape != (350, 525):\n            mask = cv2.resize(mask.astype('float32'), dsize=(525, 350), \n                              interpolation=cv2.INTER_LINEAR)\n            valid_masks.append(mask)\n            \n    print(\"valid_masks length:\", len(valid_masks))\n        \n    for j, probability in enumerate(output):\n        print(\"probability shape:\",probability.shape)\n        if probability.shape != (350, 525):\n            probability = cv2.resize(probability, dsize=(525, 350), \n                                     interpolation=cv2.INTER_LINEAR)\n        print(\"after probability shape:\",probability.shape)\n        probabilities[i * 4 + j, :, :] = probability\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"best_threshold = 0.5\nbest_size = 25000","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"def post_process(probability, threshold, min_size):\n    \"\"\"\n    Post processing of each predicted mask, components with lesser number of pixels\n    than `min_size` are ignored\n    \"\"\"\n    \n    mask = cv2.threshold(probability, threshold, 1, cv2.THRESH_BINARY)[1]\n    \n    num_component, component = cv2.connectedComponents(mask.astype(np.uint8))\n    predictions = np.zeros((350, 525), np.float32)\n    num = 0\n    for c in range(1, num_component):\n        p = (component == c)\n        if p.sum() > min_size:\n            predictions[p] = 1\n            num += 1\n    return predictions, num\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sigmoid = lambda x: 1 / (1 + np.exp(-x))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Predict output shape is (320,480,4). 4 is classes(Fish, Flower, Gravel, Surger)\n\nUsing `pred_masks[...,k] ` or `pred_masks[:,:,k]` get one class of mask image."},{"metadata":{"_uuid":"f22b1cf2-1bc1-4a90-9672-8d4b16007034","_cell_guid":"90aba8ed-cbd4-4b8e-9e1c-87a5a68ede19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"model.load_weights('model.h5')\ntest_df = []\nencoded_pixels = []\nTEST_BATCH_SIZE = 500\n\nfor i in range(0, test_images.shape[0], TEST_BATCH_SIZE):\n    batch_idx = list(\n        range(i, min(test_images.shape[0], i + TEST_BATCH_SIZE))\n    )\n\n    test_generator = DataGenerator(\n        batch_idx,\n        df=test_images,\n        shuffle=False,\n        mode='predict',\n        dim=(350, 525),\n        reshape=(320, 480),\n        n_channels=3,\n        graystyle=False,\n        base_path='../input/test_images',\n        target_df=submission_df,\n        batch_size=1,\n        n_classes=4\n    )\n\n    batch_pred_masks = model.predict_generator(\n        test_generator, \n        workers=1,\n        verbose=1\n    ) \n    # Predict out put shape is (320X480X4)\n    # 4  = 4 classes, Fish, Flower, Gravel Surger.\n    \n    for j, idx in enumerate(batch_idx):\n        filename = test_images['ImageId'].iloc[idx]\n        image_df = submission_df[submission_df['ImageId'] == filename].copy()\n        \n        # Batch prediction result set\n        pred_masks = batch_pred_masks[j, ].round().astype(int)\n        pred_rles = build_rles(pred_masks, reshape=(350, 525))\n        \n        image_df['EncodedPixels'] = pred_rles\n        \n        test_df.append(image_df)\n        \n        \n        for k in range(pred_masks.shape[-1]):\n            pred_mask = pred_masks[...,k].astype('float32') \n            \n            if pred_mask.shape != (350, 525):\n                pred_mask = cv2.resize(pred_mask, dsize=(525, 350), interpolation=cv2.INTER_LINEAR)\n                \n            pred_mask, num_predict = post_process(sigmoid(pred_mask), best_threshold, best_size )\n            \n            if num_predict == 0:\n                encoded_pixels.append('')\n            else:\n                r = mask2rle(pred_mask)\n                encoded_pixels.append(r)\n        \"\"\"\n        # pred_rles = build_rles(pred_masks, reshape=(350, 525))\n\n            #image_df['EncodedPixels'] = encoded_pixels\n            #test_df.append(image_df)\n        \"\"\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df['EncodedPixels'] = encoded_pixels\nsubmission_df.to_csv('submission.csv', columns=['Image_Label', 'EncodedPixels'], index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"encoded_pixels"},{"metadata":{"_uuid":"bff54279-8913-4b84-a6c3-daaeb1ef6b28","_cell_guid":"0c927c02-e293-4345-befb-35ac0be85578","trusted":true},"cell_type":"code","source":"#test_df = pd.concat(test_df)\n#test_df.drop(columns='ImageId', inplace=True)\n#test_df.to_csv('all_flower_submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TODO\n\n-  Try threshold = 0.85, size =10000"}],"metadata":{"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}