{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":13333,"databundleVersionId":862146,"sourceType":"competition"}],"dockerImageVersionId":29282,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!pip install segmentation-models --quiet","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-03-12T12:19:37.101957Z","iopub.execute_input":"2024-03-12T12:19:37.102195Z","iopub.status.idle":"2024-03-12T12:19:44.920468Z","shell.execute_reply.started":"2024-03-12T12:19:37.102155Z","shell.execute_reply":"2024-03-12T12:19:44.919594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Understanding Clouds from Satellite Images\n\nUsing satellite images of the clouds, participants are asked to recognize a set of specific patterns in the images. These patterns are: fish, sugar, flower, gravel. The problem can be formulated as four binary image segmentations. In this notebook, I try to provide a solution. Here's an overview of the notebook:\n1. Importing Libraries and Helper Functions\n3. Data Preprocessing (some analysis, and preperation for training, keras data generator)\n5. Setting Model for Training (proposed model and loss function)\n6. Prediction and Evaluation\n7. Conclusion and further suggestions\n\n![](https://i.imgur.com/EOvz5kd.png)\n\n","metadata":{}},{"cell_type":"markdown","source":"# Importing libraries and Helper Functions\n\n","metadata":{}},{"cell_type":"code","source":"import os\nimport json\n\nimport albumentations as albu\nimport cv2\nfrom PIL import Image\nimport tensorflow as tf\n\nimport keras\nfrom keras import backend as K\nfrom keras.models import Model\nfrom keras.layers import Input\nfrom keras.layers.convolutional import Conv2D, Conv2DTranspose\nfrom keras.layers.pooling import MaxPooling2D\nfrom keras.layers.merge import concatenate\nfrom keras.losses import binary_crossentropy\nfrom keras.optimizers import Adam, Nadam\nfrom keras.callbacks import Callback, ModelCheckpoint\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom tqdm import tqdm\nimport datetime\nfrom sklearn.model_selection import train_test_split\nimport segmentation_models as sm\n\n%matplotlib inline","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-03-11T09:17:57.362474Z","iopub.execute_input":"2024-03-11T09:17:57.362823Z","iopub.status.idle":"2024-03-11T09:17:57.395175Z","shell.execute_reply.started":"2024-03-11T09:17:57.362765Z","shell.execute_reply":"2024-03-11T09:17:57.394382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helper functions\n\nHelper functions imported from various references for reading images, decoding RLE masks, and visualization.","metadata":{}},{"cell_type":"code","source":"def get_img(x, folder: str='train_images'):\n    \"\"\"\n    Return image based on image name and folder.\n    \"\"\"\n    data_folder = f\"{path}/{folder}\"\n    image_path = os.path.join(data_folder, x)\n    img = cv2.imread(image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    return img\n\ndef np_resize(img, input_shape):\n    \"\"\"\n    Reshape a numpy array, which is input_shape=(height, width), \n    as opposed to input_shape=(width, height) for cv2\n    \"\"\"\n    height, width = input_shape\n    return cv2.resize(img, (width, height))\n    \ndef mask2rle(img):\n    '''\n    img: numpy array, 1 - mask, 0 - background\n    Returns run length as string formated\n    '''\n    pixels= img.T.flatten()\n    pixels = np.concatenate([[0], pixels, [0]])\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n    return ' '.join(str(x) for x in runs)\n\ndef rle2mask(rle, input_shape):\n    width, height = input_shape[:2]\n    \n    mask= np.zeros( width*height ).astype(np.uint8)\n    \n    array = np.asarray([int(x) for x in rle.split()])\n    starts = array[0::2]\n    lengths = array[1::2]\n\n    current_position = 0\n    for index, start in enumerate(starts):\n        mask[int(start):int(start+lengths[index])] = 1\n        current_position += lengths[index]\n        \n    return mask.reshape(height, width).T\n\ndef build_masks(rles, input_shape, reshape=None):\n    depth = len(rles)\n    if reshape is None:\n        masks = np.zeros((*input_shape, depth))\n    else:\n        masks = np.zeros((*reshape, depth))\n    \n    for i, rle in enumerate(rles):\n        if type(rle) is str:\n            if reshape is None:\n                masks[:, :, i] = rle2mask(rle, input_shape)\n            else:\n                mask = rle2mask(rle, input_shape)\n                reshaped_mask = np_resize(mask, reshape)\n                masks[:, :, i] = reshaped_mask\n    \n    return masks\n\ndef build_rles(masks, reshape=None):\n    width, height, depth = masks.shape\n    \n    rles = []\n    \n    for i in range(depth):\n        mask = masks[:, :, i]\n        \n        if reshape:\n            mask = mask.astype(np.float32)\n            mask = np_resize(mask, reshape).astype(np.int64)\n        \n        rle = mask2rle(mask)\n        rles.append(rle)\n        \n    return rles\n\n\ndef rle_decode(mask_rle: str = '', shape: tuple = (1400, 2100)):\n    '''\n    Decode rle encoded mask.\n    \n    :param mask_rle: run-length as string formatted (start length)\n    :param shape: (height, width) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n    '''\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0] * shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape, order='F')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-03-11T09:18:03.451966Z","iopub.execute_input":"2024-03-11T09:18:03.452245Z","iopub.status.idle":"2024-03-11T09:18:03.479371Z","shell.execute_reply.started":"2024-03-11T09:18:03.452209Z","shell.execute_reply":"2024-03-11T09:18:03.478690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data overview\n\nLet's have a look at the data first.","metadata":{}},{"cell_type":"code","source":"path = '../input/understanding_cloud_organization'\nos.listdir(path)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-11T09:18:11.192013Z","iopub.execute_input":"2024-03-11T09:18:11.192294Z","iopub.status.idle":"2024-03-11T09:18:11.199266Z","shell.execute_reply.started":"2024-03-11T09:18:11.192251Z","shell.execute_reply":"2024-03-11T09:18:11.198563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv(f'{path}/train.csv')\nsub_df = pd.read_csv(f'{path}/sample_submission.csv')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-03-11T09:18:12.437923Z","iopub.execute_input":"2024-03-11T09:18:12.438208Z","iopub.status.idle":"2024-03-11T09:18:16.912990Z","shell.execute_reply.started":"2024-03-11T09:18:12.438166Z","shell.execute_reply":"2024-03-11T09:18:16.912134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_train = len(os.listdir(f'{path}/train_images'))\nn_test = len(os.listdir(f'{path}/test_images'))\nprint(f'There are {n_train} images in train dataset')\nprint(f'There are {n_test} images in test dataset')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-03-11T09:18:16.915062Z","iopub.execute_input":"2024-03-11T09:18:16.915483Z","iopub.status.idle":"2024-03-11T09:18:18.548096Z","shell.execute_reply.started":"2024-03-11T09:18:16.915280Z","shell.execute_reply":"2024-03-11T09:18:18.547244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-11T09:18:18.549352Z","iopub.execute_input":"2024-03-11T09:18:18.549590Z","iopub.status.idle":"2024-03-11T09:18:18.569514Z","shell.execute_reply.started":"2024-03-11T09:18:18.549551Z","shell.execute_reply":"2024-03-11T09:18:18.568767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that there are two columns in the dataset; One represents the image id concatated with the cloud category and the second column is the rle mask for that specific class i.e. the area in which the the cloud pattern is recognized. We can also see that each sample can have multiple cloud types; that brings up some questions:\n\n1. Are there any images with zero cloud types?\n2. On average, how many cloud types each images contain?\n3. Is there a correlation between cloud types? (or what are the probablities of two (or more) cloud types happening together?)\n\nTo be able to explore our data more, let's split the first columns into two columns: \"ImageId\" and \"Label\".","metadata":{}},{"cell_type":"code","source":"train_df['ImageId'] = train_df['Image_Label'].apply(lambda x: x.split('_')[0])\ntrain_df['Label'] = train_df['Image_Label'].apply(lambda x: x.split('_')[1])\n\nprint(train_df.shape)\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-11T09:18:50.856477Z","iopub.execute_input":"2024-03-11T09:18:50.856836Z","iopub.status.idle":"2024-03-11T09:18:50.905769Z","shell.execute_reply.started":"2024-03-11T09:18:50.856774Z","shell.execute_reply":"2024-03-11T09:18:50.905080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Images with at least one label:', len(train_df[train_df['EncodedPixels'] != -1]['ImageId'].unique()))","metadata":{"execution":{"iopub.status.busy":"2024-03-11T09:21:28.771370Z","iopub.execute_input":"2024-03-11T09:21:28.771710Z","iopub.status.idle":"2024-03-11T09:21:28.790411Z","shell.execute_reply.started":"2024-03-11T09:21:28.771666Z","shell.execute_reply":"2024-03-11T09:21:28.789163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that all of the images have at least one label. Now let's explore the distribution of the labels in the dataset. ","metadata":{}},{"cell_type":"code","source":"fish = train_df[train_df['Label'] == 'Fish'].EncodedPixels.count()\nflower = train_df[train_df['Label'] == 'Flower'].EncodedPixels.count()\ngravel = train_df[train_df['Label'] == 'Gravel'].EncodedPixels.count()\nsugar = train_df[train_df['Label'] == 'Sugar'].EncodedPixels.count()","metadata":{"execution":{"iopub.status.busy":"2024-03-11T09:21:31.676695Z","iopub.execute_input":"2024-03-11T09:21:31.676986Z","iopub.status.idle":"2024-03-11T09:21:31.705030Z","shell.execute_reply.started":"2024-03-11T09:21:31.676945Z","shell.execute_reply":"2024-03-11T09:21:31.704172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plotting a pie chart\nlabels = 'Fish', 'Flower', 'Gravel', 'Sugar'\nsizes = [fish, flower, gravel, sugar]\n\nfig, ax = plt.subplots(figsize=(6, 6))\nax.pie(sizes, labels=labels, autopct='%1.1f%%', startangle=90)\nax.axis('equal')\nax.set_title('Cloud Types')\n\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-03-11T09:21:32.444607Z","iopub.execute_input":"2024-03-11T09:21:32.444957Z","iopub.status.idle":"2024-03-11T09:21:32.611077Z","shell.execute_reply.started":"2024-03-11T09:21:32.444902Z","shell.execute_reply":"2024-03-11T09:21:32.609950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Seems like the frequency of the labels are pretty balanced. This is a good property. Now let's check the number of the labels per image.","metadata":{}},{"cell_type":"code","source":"labels_per_image = train_df.groupby('ImageId')['EncodedPixels'].count()\n\nprint(\"The average number of labels per image is {}\".format(labels_per_image.mean()))\n\nfig, ax = plt.subplots(figsize=(6, 6))\nax.hist(labels_per_image)\nax.set_title('Number of Labels per Image')","metadata":{"execution":{"iopub.status.busy":"2024-03-11T09:21:34.198137Z","iopub.execute_input":"2024-03-11T09:21:34.198422Z","iopub.status.idle":"2024-03-11T09:21:34.396133Z","shell.execute_reply.started":"2024-03-11T09:21:34.198381Z","shell.execute_reply":"2024-03-11T09:21:34.395167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So on average, most of the images contain 2 classes. The amount of images that contain all 4 classes are fairly rare. \nLastly, we can check whether there's any correlation between this patterns. To make a correlation table, I created dummy variables for each class.","metadata":{}},{"cell_type":"code","source":"corr_df = pd.get_dummies(train_df, columns=['Label'])\ncorr_df = corr_df.fillna('-1')\n\ndef get_dummy_value(row, cloud_type):\n    ''' Get value for dummy column '''\n    if cloud_type == 'fish':\n        return row['Label_Fish'] * (row['EncodedPixels'] != '-1')\n    if cloud_type == 'flower':\n        return row['Label_Flower'] * (row['EncodedPixels'] != '-1')\n    if cloud_type == 'gravel':\n        return row['Label_Gravel'] * (row['EncodedPixels'] != '-1')\n    if cloud_type == 'sugar':\n        return row['Label_Sugar'] * (row['EncodedPixels'] != '-1')\n    \n# fill dummy columns\ncorr_df['Label_Fish'] = corr_df.apply(lambda row: get_dummy_value(row, 'fish'), axis=1)\ncorr_df['Label_Flower'] = corr_df.apply(lambda row: get_dummy_value(row, 'flower'), axis=1)\ncorr_df['Label_Gravel'] = corr_df.apply(lambda row: get_dummy_value(row, 'gravel'), axis=1)\ncorr_df['Label_Sugar'] = corr_df.apply(lambda row: get_dummy_value(row, 'sugar'), axis=1)\n\ncorr_df = corr_df.groupby('ImageId')[['Label_Fish', 'Label_Flower', 'Label_Gravel', 'Label_Sugar']].max()\ncorr_df.head()\n\ncorrs = np.corrcoef(corr_df.values.T)\n\nsns.set(font_scale=1)\nsns.set(rc={'figure.figsize':(6,6)})\nhm=sns.heatmap(corrs, cbar = True, annot=True, square = True, fmt = '.2f',\n              yticklabels = ['Fish', 'Flower', 'Gravel', 'Sugar'], \n               xticklabels = ['Fish', 'Flower', 'Gravel', 'Sugar']).set_title('Cloud type correlation heatmap')\n\nfig = hm.get_figure()","metadata":{"execution":{"iopub.status.busy":"2024-03-11T09:21:35.542789Z","iopub.execute_input":"2024-03-11T09:21:35.543069Z","iopub.status.idle":"2024-03-11T09:21:38.909054Z","shell.execute_reply.started":"2024-03-11T09:21:35.543027Z","shell.execute_reply":"2024-03-11T09:21:38.908157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's have a look at the images and the masks.","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(25, 16))\nfor j, im_id in enumerate(np.random.choice(train_df['ImageId'].unique(), 4)):\n    for i, (idx, row) in enumerate(train_df.loc[train_df['ImageId'] == im_id].iterrows()):\n        ax = fig.add_subplot(5, 4, j * 4 + i + 1, xticks=[], yticks=[])\n        im = Image.open(f\"{path}/train_images/{row['Image_Label'].split('_')[0]}\")\n        plt.imshow(im)\n        mask_rle = row['EncodedPixels']\n        try: # label might not be there!\n            mask = rle_decode(mask_rle)\n        except:\n            mask = np.zeros((1400, 2100))\n        plt.imshow(mask, alpha=0.5, cmap='gray')\n        ax.set_title(f\"Image: {row['Image_Label'].split('_')[0]}. Label: {row['Label']}\")","metadata":{"execution":{"iopub.status.busy":"2024-03-11T09:21:38.911557Z","iopub.execute_input":"2024-03-11T09:21:38.912277Z","iopub.status.idle":"2024-03-11T09:21:46.700435Z","shell.execute_reply.started":"2024-03-11T09:21:38.912206Z","shell.execute_reply":"2024-03-11T09:21:46.699539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that masks can overlap. Also we can see that clouds are really similar to fish, flower and so on. Another important point: masks are often quite big and can have seemingly empty areas.","metadata":{}},{"cell_type":"markdown","source":"## Preparing Data for Training\n\nAt first, let's create a list of unique image ids and the count of masks for images. This will allow us to make a stratified split based on this count.","metadata":{}},{"cell_type":"code","source":"train_df['hasMask'] = ~ train_df['EncodedPixels'].isna()\n\nmask_count_df = train_df.groupby('ImageId').agg(np.sum).reset_index()\nmask_count_df.sort_values('hasMask', ascending=False, inplace=True)\nprint(mask_count_df.shape)\nmask_count_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-11T09:21:46.701915Z","iopub.execute_input":"2024-03-11T09:21:46.702200Z","iopub.status.idle":"2024-03-11T09:21:46.734767Z","shell.execute_reply.started":"2024-03-11T09:21:46.702150Z","shell.execute_reply":"2024-03-11T09:21:46.734036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Keras Data Generator\n\nThe data generator is a keras-based class which will provide an easy, controlled way to load in the data, along with data augmentation in training. The function outputs the data samples with their IDs given the specific settings that are provided. [See more](https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly)","metadata":{}},{"cell_type":"code","source":"class DataGenerator(keras.utils.Sequence):\n    'Generates data for Keras'\n    def __init__(self, list_IDs, df, target_df=None, mode='fit',\n                 base_path=f'{path}/train_images',\n                 batch_size=32, dim=(1400, 2100), n_channels=3, reshape=None,\n                 augment=False, n_classes=4, random_state=2019, shuffle=True):\n        self.dim = dim\n        self.batch_size = batch_size\n        self.df = df\n        self.mode = mode\n        self.base_path = base_path\n        self.target_df = target_df\n        self.list_IDs = list_IDs\n        self.reshape = reshape\n        self.n_channels = n_channels\n        self.augment = augment\n        self.n_classes = n_classes\n        self.shuffle = shuffle\n        self.random_state = random_state\n        \n        self.on_epoch_end()\n        np.random.seed(self.random_state)\n\n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        return int(np.floor(len(self.list_IDs) / self.batch_size))\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        # Generate indexes of the batch\n        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n\n        # Find list of IDs\n        list_IDs_batch = [self.list_IDs[k] for k in indexes]\n        \n        X = self.__generate_X(list_IDs_batch)\n        \n        if self.mode == 'fit':\n            y = self.__generate_y(list_IDs_batch)\n            \n            if self.augment:\n                X, y = self.__augment_batch(X, y)\n            \n            return X, y\n        \n        elif self.mode == 'predict':\n            return X\n\n        else:\n            raise AttributeError('The mode parameter should be set to \"fit\" or \"predict\".')\n        \n    def on_epoch_end(self):\n        'Updates indexes after each epoch'\n        self.indexes = np.arange(len(self.list_IDs))\n        if self.shuffle == True:\n            np.random.seed(self.random_state)\n            np.random.shuffle(self.indexes)\n    \n    def __generate_X(self, list_IDs_batch):\n        'Generates data containing batch_size samples'\n        # Initialization\n        if self.reshape is None:\n            X = np.empty((self.batch_size, *self.dim, self.n_channels))\n        else:\n            X = np.empty((self.batch_size, *self.reshape, self.n_channels))\n        \n        # Generate data\n        for i, ID in enumerate(list_IDs_batch):\n            im_name = self.df['ImageId'].iloc[ID]\n            img_path = f\"{self.base_path}/{im_name}\"\n            img = self.__load_rgb(img_path)\n            \n            if self.reshape is not None:\n                img = np_resize(img, self.reshape)\n            \n            # Store samples\n            X[i,] = img\n\n        return X\n    \n    def __generate_y(self, list_IDs_batch):\n        if self.reshape is None:\n            y = np.empty((self.batch_size, *self.dim, self.n_classes), dtype=int)\n        else:\n            y = np.empty((self.batch_size, *self.reshape, self.n_classes), dtype=int)\n        \n        for i, ID in enumerate(list_IDs_batch):\n            im_name = self.df['ImageId'].iloc[ID]\n            image_df = self.target_df[self.target_df['ImageId'] == im_name]\n            \n            rles = image_df['EncodedPixels'].values\n            \n            if self.reshape is not None:\n                masks = build_masks(rles, input_shape=self.dim, reshape=self.reshape)\n            else:\n                masks = build_masks(rles, input_shape=self.dim)\n            \n            y[i, ] = masks\n\n        return y\n    \n    def __load_grayscale(self, img_path):\n        img = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)\n        img = img.astype(np.float32) / 255.\n        img = np.expand_dims(img, axis=-1)\n\n        return img\n    \n    def __load_rgb(self, img_path):\n        img = cv2.imread(img_path)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        img = img.astype(np.float32) / 255.\n\n        return img\n    \n    def __random_transform(self, img, masks):\n        composition = albu.Compose([\n            albu.HorizontalFlip(),\n            albu.VerticalFlip(),\n            albu.ShiftScaleRotate(rotate_limit=45, shift_limit=0.15, scale_limit=0.15)\n        ])\n        \n        composed = composition(image=img, mask=masks)\n        aug_img = composed['image']\n        aug_masks = composed['mask']\n        \n        return aug_img, aug_masks\n    \n    def __augment_batch(self, img_batch, masks_batch):\n        for i in range(img_batch.shape[0]):\n            img_batch[i, ], masks_batch[i, ] = self.__random_transform(\n                img_batch[i, ], masks_batch[i, ])\n        \n        return img_batch, masks_batch","metadata":{"execution":{"iopub.status.busy":"2024-03-11T09:21:58.227756Z","iopub.execute_input":"2024-03-11T09:21:58.228048Z","iopub.status.idle":"2024-03-11T09:21:58.411755Z","shell.execute_reply.started":"2024-03-11T09:21:58.228010Z","shell.execute_reply":"2024-03-11T09:21:58.410969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Setting Model for Training\n\nSince this is a segmentation problem, the best option is using a U-net architecture ([see paper](https://arxiv.org/abs/1505.04597)). The name U-net comes from the shape of the network model. The model uses operations like down-sampling convolution and max-pooling to decrease the size of the image to the mid-point (meanwhile increasing the parameters) and then applies up-sampling convolutions to increase the size of the images whilst keeping the high number of parameters. A u-net architecture usually has skip-connections (residual network), which enhances the performance of the network by preventing the problem of vanishing gradient. \n![U-net Diagram](https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/u-net-architecture.png)","metadata":{}},{"cell_type":"markdown","source":"## Loss Function\n\nThe loss function is chosen based on the competition's evaluation metric, dice coefficient. Dice coefficient is a very useful metric for segmentation problems. In the picture below we can see the definition. [(Image Source)](https://cvinvolution.medium.com/dice-loss-in-medical-image-segmentation-d0e476eb486)\n\n![](https://miro.medium.com/v2/resize:fit:1100/format:webp/1*QumkIISngfiVg3sm_gWtCA.png)","metadata":{}},{"cell_type":"code","source":"def dice_coef(y_true, y_pred, smooth=1):\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = K.sum(y_true_f * y_pred_f)\n    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n\ndef dice_loss(y_true, y_pred):\n    smooth = 1.\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = y_true_f * y_pred_f\n    score = (2. * K.sum(intersection) + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n    return 1. - score\n\ndef bce_dice_loss(y_true, y_pred):\n    return binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2024-03-11T09:21:55.507826Z","iopub.execute_input":"2024-03-11T09:21:55.508146Z","iopub.status.idle":"2024-03-11T09:21:55.516921Z","shell.execute_reply.started":"2024-03-11T09:21:55.508090Z","shell.execute_reply":"2024-03-11T09:21:55.516197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"BATCH_SIZE = 32\n\ntrain_idx, val_idx = train_test_split(mask_count_df.index, random_state=222, test_size=0.2)\n\ntrain_generator = DataGenerator(\n    train_idx, \n    df=mask_count_df,\n    target_df=train_df,\n    batch_size=BATCH_SIZE,\n    reshape=(320, 480),\n    augment=True,\n    n_channels=3,\n    n_classes=4\n)\n\nval_generator = DataGenerator(\n    val_idx, \n    df=mask_count_df,\n    target_df=train_df,\n    batch_size=BATCH_SIZE, \n    reshape=(320, 480),\n    augment=False,\n    n_channels=3,\n    n_classes=4\n)","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2024-03-11T09:21:59.941493Z","iopub.execute_input":"2024-03-11T09:21:59.941828Z","iopub.status.idle":"2024-03-11T09:21:59.950571Z","shell.execute_reply.started":"2024-03-11T09:21:59.941770Z","shell.execute_reply":"2024-03-11T09:21:59.949802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I'm using a residual network with 34 convolutional layers. Training images are reshaped to 320*480, because the predictions should be in that size (based on competition description). The activation for the output layer is sigmoid. ","metadata":{}},{"cell_type":"code","source":"model = sm.Unet(\n    'resnet34', \n    classes=4,\n    input_shape=(320, 480, 3),\n    activation='sigmoid'\n)\n\n\nmodel.compile(optimizer=Nadam(lr=0.0002), loss=bce_dice_loss, metrics=[dice_coef])\nmodel.summary()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-11T09:45:18.252214Z","iopub.execute_input":"2024-03-11T09:45:18.252660Z","iopub.status.idle":"2024-03-11T09:45:26.864591Z","shell.execute_reply.started":"2024-03-11T09:45:18.252569Z","shell.execute_reply":"2024-03-11T09:45:26.863527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checkpoint = ModelCheckpoint('model.h5', save_best_only=True)\n\n#log_dir = \"logs/fit/\" + datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n#tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)\n\nhistory = model.fit_generator(\n    train_generator,\n    validation_data=val_generator,\n    callbacks=[checkpoint],\n    epochs=30\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction and  Evaluation","metadata":{}},{"cell_type":"markdown","source":"Now that we have completed the training, we can use the model to do prediction on our test data. ","metadata":{}},{"cell_type":"code","source":"with open('history.json', 'w') as f:\n    json.dump(history.history, f)\n\nhistory_df = pd.DataFrame(history.history)\nhistory_df[['loss', 'val_loss']].plot()\nhistory_df[['dice_coef', 'val_dice_coef']].plot()","metadata":{"execution":{"iopub.status.busy":"2024-03-07T10:17:34.941178Z","iopub.execute_input":"2024-03-07T10:17:34.941468Z","iopub.status.idle":"2024-03-07T10:17:35.367515Z","shell.execute_reply.started":"2024-03-07T10:17:34.941432Z","shell.execute_reply":"2024-03-07T10:17:35.365244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df['ImageId'] = sub_df['Image_Label'].apply(lambda x: x.split('_')[0])\nsub_df['Label'] = sub_df['Image_Label'].apply(lambda x: x.split('_')[1])\n\ntest_imgs = pd.DataFrame(sub_df['ImageId'].unique(), columns=['ImageId'])\n\nsub_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-07T10:26:03.692973Z","iopub.execute_input":"2024-03-07T10:26:03.693351Z","iopub.status.idle":"2024-03-07T10:26:03.732265Z","shell.execute_reply.started":"2024-03-07T10:26:03.693304Z","shell.execute_reply":"2024-03-07T10:26:03.731473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.load_weights('model.h5')\nprediction = []\n\nfor i in range(0, test_imgs.shape[0], 500):\n    batch_idx = list(\n        range(i, min(test_imgs.shape[0], i + 500))\n    )\n\n    test_generator = DataGenerator(\n        batch_idx,\n        df=test_imgs,\n        shuffle=False,\n        mode='predict',\n        dim=(350, 525),\n        reshape=(320, 480),\n        n_channels=3,\n        base_path=f'{path}/test_images',\n        target_df=sub_df,\n        batch_size=1,\n        n_classes=4\n    )\n\n    batch_pred_masks = model.predict_generator(\n        test_generator, \n        workers=1,\n        verbose=1\n    )\n\n    for j, b in enumerate(batch_idx):\n        filename = test_imgs['ImageId'].iloc[b]\n        image_df = sub_df[sub_df['ImageId'] == filename].copy()\n\n        pred_masks = batch_pred_masks[j, ].round().astype(int)\n        pred_rles = build_rles(pred_masks, reshape=(350, 525))\n\n        image_df['EncodedPixels'] = pred_rles\n        prediction.append(image_df)","metadata":{"execution":{"iopub.status.busy":"2024-03-07T10:35:02.616768Z","iopub.execute_input":"2024-03-07T10:35:02.617057Z","iopub.status.idle":"2024-03-07T10:40:08.197142Z","shell.execute_reply.started":"2024-03-07T10:35:02.617018Z","shell.execute_reply":"2024-03-07T10:40:08.196263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred_df = pd.concat(prediction)\n#pred_df.drop(columns='ImageId', inplace=True)\npred_df.to_csv('pred_submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2024-03-07T10:41:07.847572Z","iopub.execute_input":"2024-03-07T10:41:07.848020Z","iopub.status.idle":"2024-03-07T10:41:10.270702Z","shell.execute_reply.started":"2024-03-07T10:41:07.847954Z","shell.execute_reply":"2024-03-07T10:41:10.269812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure(figsize=(25, 16))\nfor j, im_id in enumerate(np.random.choice(pred_df['ImageId'].unique(), 4)):\n    for i, (idx, row) in enumerate(pred_df.loc[pred_df['ImageId'] == im_id].iterrows()):\n        ax = fig.add_subplot(5, 4, j * 4 + i + 1, xticks=[], yticks=[])\n        im = Image.open(f\"{path}/test_images/{row['ImageId']}\")\n        plt.imshow(im)\n        mask_rle = row['EncodedPixels']\n        try: # label might not be there!\n            mask = rle_decode(mask_rle)\n        except:\n            mask = np.zeros((1400, 2100))\n        plt.imshow(mask, alpha=0.5, cmap='gray')\n        ax.set_title(f\"Image: {row['Image_Label'].split('_')[0]}. Label: {row['Label']}\")","metadata":{"execution":{"iopub.status.busy":"2024-03-07T10:42:47.809082Z","iopub.execute_input":"2024-03-07T10:42:47.809428Z","iopub.status.idle":"2024-03-07T10:42:54.528802Z","shell.execute_reply.started":"2024-03-07T10:42:47.809379Z","shell.execute_reply":"2024-03-07T10:42:54.528065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As expected, we can see that the model is not able to recognize the patterns in test images. Unfortunately, I didn't put much time to improve my model due to the limited time that I had. However, here are some observations and some ideas on improve the model:\n\n**1. Checking correctness:** I will go over the model and dataset, making sure everything is working correctly. These can be things like checking that values are normalized correctly, or if the loss function is implemented in the correct way. Once we are sure that the model is not faulty, we can move on to the next step.\n\n**2. Increasing complexity:** The training accuracy (65%) is pretty low. First I'd try to increase that by increasing \"complexity\" like: running model on more epochs, adding layer, adding more parameters, etc. \n\n**3. Regularization:** I might reach a point where I'm overfitting (signifacnt gap between train and test/validation performance.) At this stage I can make use of regularization techniques, dropout, weight decay, layer or batch normalization, etc. \n\n**4. Model Comparison:** Hopefully at this point I'm having some decent results and I can compare them to other similar models, and try to improve my model. ","metadata":{}}]}