{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":13333,"databundleVersionId":862146,"sourceType":"competition"}],"dockerImageVersionId":30588,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# About this kernel\n\n* **Preprocessing**: Expand the train dataframe to include image ID. Also create `mask_count_df` which will be useful for later.\n* **Utility Functions**: Mostly copied from Paul's kernel and SIIM starter code (see references). You won't need to modify those.\n* **Sample Test**: Simply visualizing a sample image and its masks,\n* **Data Generator**: Very long and possibly complex. If you can, skip this part of the code. **If you absolute need to modify the data generation process, please take a look `__generate_X` and `__generate_y`**; in theory everything else should be left as is. This code is different from my previous Keras U-Net boilerplate since it lets you reshape the input image as well as the mask, and lets you \n* **Model Architecture**: The architecture is slightly different from the other kernels, since **it learns to predict all of the four masks at the same time**, instead of predicting a single mask and duplicating it. It also takes as input grayscale images.\n* **Training**: Running only for 25 epochs.\n* **Evaluation & Submission**: The submission code is pretty messy. Essentially, I'm splitting the test dataframe into multiple chunks, then run the model and `mask2rle` converter on the results. I'm doing this in order to not run out of RAM as we try to convert all the masks from array to RLE.\n\n## Changelog\n* **V27**: Replace ResNet-18 with ResNet-34.\n* **V25**: Replace RAdam with NAdam.\n* **V22**: Replace Adam with RAdam.\n* **V18**: Changed the vanilla U-Net by using ResNet encoder. This is easily done using the incredible [*segmentation-models*](https://github.com/qubvel/segmentation_models) library made by qubvel.\n* **V15**: Fixed the test image size output, which should have been 350x525 instead of 1600x2100 (which is valid for training data only). Also removed the \"Micro-EDA\" and \"Show Sample\", since [artgor's ](https://www.kaggle.com/artgor/segmentation-in-pytorch-using-convenient-tools) kernel already covers `albumentations` very well.\n\n## References\n* `segmentation-models`: https://github.com/qubvel/segmentation_models\n* EDA of albumentations: https://www.kaggle.com/artgor/segmentation-in-pytorch-using-convenient-tools\n* Data generator: https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly\n* RLE encoding and decoding: https://www.kaggle.com/paulorzp/rle-functions-run-lenght-encode-decode\n* Architecture: https://www.kaggle.com/jesperdramsch/intro-chest-xray-dicom-viz-u-nets-full-data\n* Mask encoding: https://www.kaggle.com/c/siim-acr-pneumothorax-segmentation/data\n* My original Kernel U-Net: https://www.kaggle.com/xhlulu/severstal-simple-keras-u-net-boilerplate","metadata":{}},{"cell_type":"code","source":"%env SM_FRAMEWORK=tf.keras\n!pip install segmentation-models --quiet\n!pip install keras\n!pip install tensorflow","metadata":{"execution":{"iopub.status.busy":"2023-12-06T06:12:59.10749Z","iopub.execute_input":"2023-12-06T06:12:59.107885Z","iopub.status.idle":"2023-12-06T06:13:35.307528Z","shell.execute_reply.started":"2023-12-06T06:12:59.107853Z","shell.execute_reply":"2023-12-06T06:13:35.30653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport json\n\nimport albumentations as albu\nimport cv2\nimport keras\nimport tensorflow as tf\ntf.config.run_functions_eagerly(True)\nfrom keras import backend as K\nfrom keras.models import Model\nfrom keras.layers import Input\nfrom keras.layers import Conv2D, Conv2DTranspose\nfrom keras.layers import MaxPooling2D\nfrom keras.layers import concatenate\nfrom keras.losses import binary_crossentropy\nfrom keras.optimizers import Adam, Nadam\nfrom keras.callbacks import Callback, ModelCheckpoint\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nfrom sklearn.model_selection import train_test_split\nimport segmentation_models as sm","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.status.busy":"2023-12-06T06:13:35.309755Z","iopub.execute_input":"2023-12-06T06:13:35.310088Z","iopub.status.idle":"2023-12-06T06:13:48.399701Z","shell.execute_reply.started":"2023-12-06T06:13:35.310061Z","shell.execute_reply":"2023-12-06T06:13:48.398803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocessing","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/understanding_cloud_organization/train.csv')\ntrain_df['ImageId'] = train_df['Image_Label'].apply(lambda x: x.split('_')[0])\ntrain_df['ClassId'] = train_df['Image_Label'].apply(lambda x: x.split('_')[1])\ntrain_df['hasMask'] = ~ train_df['EncodedPixels'].isna()\n\nprint(train_df.shape)\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-12-06T06:13:48.401034Z","iopub.execute_input":"2023-12-06T06:13:48.401557Z","iopub.status.idle":"2023-12-06T06:13:52.699125Z","shell.execute_reply.started":"2023-12-06T06:13:48.40153Z","shell.execute_reply":"2023-12-06T06:13:52.698278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mask_count_df = train_df.groupby('ImageId').agg(np.sum).reset_index()\nmask_count_df.sort_values('hasMask', ascending=False, inplace=True)\nprint(mask_count_df.shape)\nmask_count_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-12-06T06:13:52.701347Z","iopub.execute_input":"2023-12-06T06:13:52.701621Z","iopub.status.idle":"2023-12-06T06:13:52.903112Z","shell.execute_reply.started":"2023-12-06T06:13:52.701597Z","shell.execute_reply":"2023-12-06T06:13:52.902226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df = pd.read_csv('/kaggle/input/understanding_cloud_organization/train.csv')\nsub_df['ImageId'] = sub_df['Image_Label'].apply(lambda x: x.split('_')[0])\ntest_imgs = pd.DataFrame(sub_df['ImageId'].unique(), columns=['ImageId'])","metadata":{"execution":{"iopub.status.busy":"2023-12-06T06:13:52.90426Z","iopub.execute_input":"2023-12-06T06:13:52.904544Z","iopub.status.idle":"2023-12-06T06:13:55.010397Z","shell.execute_reply.started":"2023-12-06T06:13:52.904521Z","shell.execute_reply":"2023-12-06T06:13:55.009577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Utility Functions\n\nSource: https://www.kaggle.com/paulorzp/rle-functions-run-lenght-encode-decode\n\nUnhide below for the definition of `np_resize`, `build_masks`, `build_rles`.","metadata":{}},{"cell_type":"code","source":"def np_resize(img, input_shape):\n    \"\"\"\n    Reshape a numpy array, which is input_shape=(height, width), \n    as opposed to input_shape=(width, height) for cv2\n    \"\"\"\n    height, width = input_shape\n    return cv2.resize(img, (width, height))\n    \ndef mask2rle(img):\n    '''\n    img: numpy array, 1 - mask, 0 - background\n    Returns run length as string formated\n    '''\n    pixels= img.T.flatten()\n    pixels = np.concatenate([[0], pixels, [0]])\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n    return ' '.join(str(x) for x in runs)\n\ndef rle2mask(rle, input_shape):\n    width, height = input_shape[:2]\n    \n    mask= np.zeros( width*height ).astype(np.uint8)\n    \n    array = np.asarray([int(x) for x in rle.split()])\n    starts = array[0::2]\n    lengths = array[1::2]\n\n    current_position = 0\n    for index, start in enumerate(starts):\n        mask[int(start):int(start+lengths[index])] = 1\n        current_position += lengths[index]\n        \n    return mask.reshape(height, width).T\n\ndef build_masks(rles, input_shape, reshape=None):\n    depth = len(rles)\n    if reshape is None:\n        masks = np.zeros((*input_shape, depth))\n    else:\n        masks = np.zeros((*reshape, depth))\n    \n    for i, rle in enumerate(rles):\n        if type(rle) is str:\n            if reshape is None:\n                masks[:, :, i] = rle2mask(rle, input_shape)\n            else:\n                mask = rle2mask(rle, input_shape)\n                reshaped_mask = np_resize(mask, reshape)\n                masks[:, :, i] = reshaped_mask\n    \n    return masks\n\ndef build_rles(masks, reshape=None):\n    width, height, depth = masks.shape\n    \n    rles = []\n    \n    for i in range(depth):\n        mask = masks[:, :, i]\n        \n        if reshape:\n            mask = mask.astype(np.float32)\n            mask = np_resize(mask, reshape).astype(np.int64)\n        \n        rle = mask2rle(mask)\n        rles.append(rle)\n        \n    return rles","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"execution":{"iopub.status.busy":"2023-12-06T06:13:55.011911Z","iopub.execute_input":"2023-12-06T06:13:55.012205Z","iopub.status.idle":"2023-12-06T06:13:55.026729Z","shell.execute_reply.started":"2023-12-06T06:13:55.012179Z","shell.execute_reply":"2023-12-06T06:13:55.025753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## RAdam\n\nUnhide below to see definition of `RAdam`:","metadata":{}},{"cell_type":"code","source":"class RAdam(keras.optimizers.Optimizer):\n    \"\"\"RAdam optimizer.\n    # Arguments\n        lr: float >= 0. Learning rate.\n        beta_1: float, 0 < beta < 1. Generally close to 1.\n        beta_2: float, 0 < beta < 1. Generally close to 1.\n        epsilon: float >= 0. Fuzz factor. If `None`, defaults to `K.epsilon()`.\n        decay: float >= 0. Learning rate decay over each update.\n        weight_decay: float >= 0. Weight decay for each param.\n        amsgrad: boolean. Whether to apply the AMSGrad variant of this\n            algorithm from the paper \"On the Convergence of Adam and\n            Beyond\".\n        total_steps: int >= 0. Total number of training steps. Enable warmup by setting a positive value.\n        warmup_proportion: 0 < warmup_proportion < 1. The proportion of increasing steps.\n        min_lr: float >= 0. Minimum learning rate after warmup.\n    # References\n        - [Adam - A Method for Stochastic Optimization](https://arxiv.org/abs/1412.6980v8)\n        - [On the Convergence of Adam and Beyond](https://openreview.net/forum?id=ryQu7f-RZ)\n        - [On The Variance Of The Adaptive Learning Rate And Beyond](https://arxiv.org/pdf/1908.03265v1.pdf)\n    \"\"\"\n\n    def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999,\n                 epsilon=None, decay=0., weight_decay=0., amsgrad=False,\n                 total_steps=0, warmup_proportion=0.1, min_lr=0., **kwargs):\n        super(RAdam, self).__init__(**kwargs)\n        with K.name_scope(self.__class__.__name__):\n            self.iterations = K.variable(0, dtype='int64', name='iterations')\n            self.lr = K.variable(lr, name='lr')\n            self.beta_1 = K.variable(beta_1, name='beta_1')\n            self.beta_2 = K.variable(beta_2, name='beta_2')\n            self.decay = K.variable(decay, name='decay')\n            self.weight_decay = K.variable(weight_decay, name='weight_decay')\n            self.total_steps = K.variable(total_steps, name='total_steps')\n            self.warmup_proportion = K.variable(warmup_proportion, name='warmup_proportion')\n            self.min_lr = K.variable(min_lr, name='min_lr')\n        if epsilon is None:\n            epsilon = K.epsilon()\n        self.epsilon = epsilon\n        self.initial_decay = decay\n        self.initial_weight_decay = weight_decay\n        self.initial_total_steps = total_steps\n        self.amsgrad = amsgrad\n\n    def get_updates(self, loss, params):\n        grads = self.get_gradients(loss, params)\n        self.updates = [K.update_add(self.iterations, 1)]\n\n        lr = self.lr\n\n        if self.initial_decay > 0:\n            lr = lr * (1. / (1. + self.decay * K.cast(self.iterations, K.dtype(self.decay))))\n\n        t = K.cast(self.iterations, K.floatx()) + 1\n\n        if self.initial_total_steps > 0:\n            warmup_steps = self.total_steps * self.warmup_proportion\n            lr = K.switch(\n                t <= warmup_steps,\n                lr * (t / warmup_steps),\n                self.min_lr + (lr - self.min_lr) * (1.0 - K.minimum(t, self.total_steps) / self.total_steps),\n            )\n\n        ms = [K.zeros(K.int_shape(p), dtype=K.dtype(p), name='m_' + str(i)) for (i, p) in enumerate(params)]\n        vs = [K.zeros(K.int_shape(p), dtype=K.dtype(p), name='v_' + str(i)) for (i, p) in enumerate(params)]\n\n        if self.amsgrad:\n            vhats = [K.zeros(K.int_shape(p), dtype=K.dtype(p), name='vhat_' + str(i)) for (i, p) in enumerate(params)]\n        else:\n            vhats = [K.zeros(1, name='vhat_' + str(i)) for i in range(len(params))]\n\n        self.weights = [self.iterations] + ms + vs + vhats\n\n        beta_1_t = K.pow(self.beta_1, t)\n        beta_2_t = K.pow(self.beta_2, t)\n\n        sma_inf = 2.0 / (1.0 - self.beta_2) - 1.0\n        sma_t = sma_inf - 2.0 * t * beta_2_t / (1.0 - beta_2_t)\n\n        for p, g, m, v, vhat in zip(params, grads, ms, vs, vhats):\n            m_t = (self.beta_1 * m) + (1. - self.beta_1) * g\n            v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(g)\n\n            m_corr_t = m_t / (1.0 - beta_1_t)\n            if self.amsgrad:\n                vhat_t = K.maximum(vhat, v_t)\n                v_corr_t = K.sqrt(vhat_t / (1.0 - beta_2_t) + self.epsilon)\n                self.updates.append(K.update(vhat, vhat_t))\n            else:\n                v_corr_t = K.sqrt(v_t / (1.0 - beta_2_t) + self.epsilon)\n\n            r_t = K.sqrt((sma_t - 4.0) / (sma_inf - 4.0) *\n                         (sma_t - 2.0) / (sma_inf - 2.0) *\n                         sma_inf / sma_t)\n\n            p_t = K.switch(sma_t >= 5, r_t * m_corr_t / v_corr_t, m_corr_t)\n\n            if self.initial_weight_decay > 0:\n                p_t += self.weight_decay * p\n\n            p_t = p - lr * p_t\n\n            self.updates.append(K.update(m, m_t))\n            self.updates.append(K.update(v, v_t))\n            new_p = p_t\n\n            # Apply constraints.\n            if getattr(p, 'constraint', None) is not None:\n                new_p = p.constraint(new_p)\n\n            self.updates.append(K.update(p, new_p))\n        return self.updates\n\n    def get_config(self):\n        config = {\n            'lr': float(K.get_value(self.lr)),\n            'beta_1': float(K.get_value(self.beta_1)),\n            'beta_2': float(K.get_value(self.beta_2)),\n            'decay': float(K.get_value(self.decay)),\n            'weight_decay': float(K.get_value(self.weight_decay)),\n            'epsilon': self.epsilon,\n            'amsgrad': self.amsgrad,\n            'total_steps': float(K.get_value(self.total_steps)),\n            'warmup_proportion': float(K.get_value(self.warmup_proportion)),\n            'min_lr': float(K.get_value(self.min_lr)),\n        }\n        base_config = super(RAdam, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-12-06T06:13:55.028407Z","iopub.execute_input":"2023-12-06T06:13:55.02879Z","iopub.status.idle":"2023-12-06T06:13:55.056269Z","shell.execute_reply.started":"2023-12-06T06:13:55.02875Z","shell.execute_reply":"2023-12-06T06:13:55.055538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loss function\n\nSource for `bce_dice_loss`: https://lars76.github.io/neural-networks/object-detection/losses-for-segmentation/","metadata":{}},{"cell_type":"code","source":"def dice_coef(y_true, y_pred, smooth=1):\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    y_true_f = K.cast(y_true_f, 'float32')\n    y_pred_f = K.cast(y_pred_f, 'float32')\n    intersection = K.sum(y_true_f * y_pred_f)\n    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n\ndef dice_loss(y_true, y_pred):\n    smooth = 1.\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    y_true_f = K.cast(y_true_f, 'float32')\n    y_pred_f = K.cast(y_pred_f, 'float32')\n    intersection = y_true_f * y_pred_f\n    score = (2. * K.sum(intersection) + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n    return 1. - score\n\ndef bce_dice_loss(y_true, y_pred):\n    return binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2023-12-06T06:13:55.057282Z","iopub.execute_input":"2023-12-06T06:13:55.057554Z","iopub.status.idle":"2023-12-06T06:13:55.070594Z","shell.execute_reply.started":"2023-12-06T06:13:55.057531Z","shell.execute_reply":"2023-12-06T06:13:55.069807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Generator","metadata":{}},{"cell_type":"markdown","source":"Unhide below for the definition of `DataGenerator`:","metadata":{}},{"cell_type":"code","source":"class DataGenerator(keras.utils.Sequence):\n    'Generates data for Keras'\n    def __init__(self, list_IDs, df, target_df=None, mode='fit',\n                 base_path='/kaggle/input/understanding_cloud_organization/train_images',\n                 batch_size=32, dim=(1400, 2100), n_channels=3, reshape=None,\n                 augment=False, n_classes=4, random_state=2019, shuffle=True):\n        self.dim = dim\n        self.batch_size = batch_size\n        self.df = df\n        self.mode = mode\n        self.base_path = base_path\n        self.target_df = target_df\n        self.list_IDs = list_IDs\n        self.reshape = reshape\n        self.n_channels = n_channels\n        self.augment = augment\n        self.n_classes = n_classes\n        self.shuffle = shuffle\n        self.random_state = random_state\n        \n        self.on_epoch_end()\n        np.random.seed(self.random_state)\n\n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        return int(np.floor(len(self.list_IDs) / self.batch_size))\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        # Generate indexes of the batch\n        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n\n        # Find list of IDs\n        list_IDs_batch = [self.list_IDs[k] for k in indexes]\n        \n        X = self.__generate_X(list_IDs_batch)\n        \n        if self.mode == 'fit':\n            y = self.__generate_y(list_IDs_batch)\n            \n            if self.augment:\n                X, y = self.__augment_batch(X, y)\n            \n            return X, y\n        \n        elif self.mode == 'predict':\n            return X\n\n        else:\n            raise AttributeError('The mode parameter should be set to \"fit\" or \"predict\".')\n        \n    def on_epoch_end(self):\n        'Updates indexes after each epoch'\n        self.indexes = np.arange(len(self.list_IDs))\n        if self.shuffle == True:\n            np.random.seed(self.random_state)\n            np.random.shuffle(self.indexes)\n    \n    def __generate_X(self, list_IDs_batch):\n        'Generates data containing batch_size samples'\n        # Initialization\n        if self.reshape is None:\n            X = np.empty((self.batch_size, *self.dim, self.n_channels))\n        else:\n            X = np.empty((self.batch_size, *self.reshape, self.n_channels))\n        \n        # Generate data\n        for i, ID in enumerate(list_IDs_batch):\n            im_name = self.df['ImageId'].iloc[ID]\n            img_path = f\"{self.base_path}/{im_name}\"\n            img = self.__load_rgb(img_path)\n            \n            if self.reshape is not None:\n                img = np_resize(img, self.reshape)\n            \n            # Store samples\n            X[i,] = img\n\n        return X\n    \n    def __generate_y(self, list_IDs_batch):\n        if self.reshape is None:\n            y = np.empty((self.batch_size, *self.dim, self.n_classes), dtype=int)\n        else:\n            y = np.empty((self.batch_size, *self.reshape, self.n_classes), dtype=int)\n        \n        for i, ID in enumerate(list_IDs_batch):\n            im_name = self.df['ImageId'].iloc[ID]\n            image_df = self.target_df[self.target_df['ImageId'] == im_name]\n            \n            rles = image_df['EncodedPixels'].values\n            \n            if self.reshape is not None:\n                masks = build_masks(rles, input_shape=self.dim, reshape=self.reshape)\n            else:\n                masks = build_masks(rles, input_shape=self.dim)\n            \n            y[i, ] = masks\n\n        return y\n    \n    def __load_grayscale(self, img_path):\n        img = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)\n        img = img.astype(np.float32) / 255.\n        img = np.expand_dims(img, axis=-1)\n\n        return img\n    \n    def __load_rgb(self, img_path):\n        img = cv2.imread(img_path)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        img = img.astype(np.float32) / 255.\n\n        return img\n    \n    def __random_transform(self, img, masks):\n        composition = albu.Compose([\n            albu.HorizontalFlip(),\n            albu.VerticalFlip(),\n            albu.ShiftScaleRotate(rotate_limit=45, shift_limit=0.15, scale_limit=0.15)\n        ])\n        \n        composed = composition(image=img, mask=masks)\n        aug_img = composed['image']\n        aug_masks = composed['mask']\n        \n        return aug_img, aug_masks\n    \n    def __augment_batch(self, img_batch, masks_batch):\n        for i in range(img_batch.shape[0]):\n            img_batch[i, ], masks_batch[i, ] = self.__random_transform(\n                img_batch[i, ], masks_batch[i, ])\n        \n        return img_batch, masks_batch","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-12-06T06:13:55.071806Z","iopub.execute_input":"2023-12-06T06:13:55.072085Z","iopub.status.idle":"2023-12-06T06:13:55.094734Z","shell.execute_reply.started":"2023-12-06T06:13:55.072062Z","shell.execute_reply":"2023-12-06T06:13:55.093976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Architecture","metadata":{}},{"cell_type":"markdown","source":"Unhide below for the old architecture:","metadata":{}},{"cell_type":"code","source":"def vanilla_unet(input_shape):\n    \"\"\"\n    This is the old model. Best LB is ~0.5\n    \"\"\"\n    inputs = Input(input_shape)\n\n    c1 = Conv2D(8, (3, 3), activation='elu', padding='same') (inputs)\n    c1 = Conv2D(8, (3, 3), activation='elu', padding='same') (c1)\n    p1 = MaxPooling2D((2, 2), padding='same') (c1)\n\n    c2 = Conv2D(16, (3, 3), activation='elu', padding='same') (p1)\n    c2 = Conv2D(16, (3, 3), activation='elu', padding='same') (c2)\n    p2 = MaxPooling2D((2, 2), padding='same') (c2)\n\n    c3 = Conv2D(32, (3, 3), activation='elu', padding='same') (p2)\n    c3 = Conv2D(32, (3, 3), activation='elu', padding='same') (c3)\n    p3 = MaxPooling2D((2, 2), padding='same') (c3)\n\n    c4 = Conv2D(64, (3, 3), activation='elu', padding='same') (p3)\n    c4 = Conv2D(64, (3, 3), activation='elu', padding='same') (c4)\n    p4 = MaxPooling2D((2, 2), padding='same') (c4)\n\n    c5 = Conv2D(64, (3, 3), activation='elu', padding='same') (p4)\n    c5 = Conv2D(64, (3, 3), activation='elu', padding='same') (c5)\n    p5 = MaxPooling2D((2, 2), padding='same') (c5)\n\n    c55 = Conv2D(128, (3, 3), activation='elu', padding='same') (p5)\n    c55 = Conv2D(128, (3, 3), activation='elu', padding='same') (c55)\n\n    u6 = Conv2DTranspose(64, (2, 2), strides=(2, 2), padding='same') (c55)\n    u6 = concatenate([u6, c5])\n    c6 = Conv2D(64, (3, 3), activation='elu', padding='same') (u6)\n    c6 = Conv2D(64, (3, 3), activation='elu', padding='same') (c6)\n\n    u71 = Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same') (c6)\n    u71 = concatenate([u71, c4])\n    c71 = Conv2D(32, (3, 3), activation='elu', padding='same') (u71)\n    c61 = Conv2D(32, (3, 3), activation='elu', padding='same') (c71)\n\n    u7 = Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same') (c61)\n    u7 = concatenate([u7, c3])\n    c7 = Conv2D(32, (3, 3), activation='elu', padding='same') (u7)\n    c7 = Conv2D(32, (3, 3), activation='elu', padding='same') (c7)\n\n    u8 = Conv2DTranspose(16, (2, 2), strides=(2, 2), padding='same') (c7)\n    u8 = concatenate([u8, c2])\n    c8 = Conv2D(16, (3, 3), activation='elu', padding='same') (u8)\n    c8 = Conv2D(16, (3, 3), activation='elu', padding='same') (c8)\n\n    u9 = Conv2DTranspose(8, (2, 2), strides=(2, 2), padding='same') (c8)\n    u9 = concatenate([u9, c1], axis=3)\n    c9 = Conv2D(8, (3, 3), activation='elu', padding='same') (u9)\n    c9 = Conv2D(8, (3, 3), activation='elu', padding='same') (c9)\n\n    outputs = Conv2D(4, (1, 1), activation='sigmoid') (c9)\n\n    model = Model(inputs=[inputs], outputs=[outputs])\n    \n    return model","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-12-06T06:13:55.098323Z","iopub.execute_input":"2023-12-06T06:13:55.098687Z","iopub.status.idle":"2023-12-06T06:13:55.118786Z","shell.execute_reply.started":"2023-12-06T06:13:55.098654Z","shell.execute_reply":"2023-12-06T06:13:55.117996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{}},{"cell_type":"code","source":"BATCH_SIZE = 32\n\ntrain_idx, val_idx = train_test_split(\n    mask_count_df.index, random_state=2019, test_size=0.2\n)\n\ntrain_generator = DataGenerator(\n    train_idx, \n    df=mask_count_df,\n    target_df=train_df,\n    batch_size=BATCH_SIZE,\n    reshape=(320, 480),\n    augment=True,\n    n_channels=3,\n    n_classes=4\n)\n\nval_generator = DataGenerator(\n    val_idx, \n    df=mask_count_df,\n    target_df=train_df,\n    batch_size=BATCH_SIZE, \n    reshape=(320, 480),\n    augment=False,\n    n_channels=3,\n    n_classes=4\n)","metadata":{"execution":{"iopub.status.busy":"2023-12-06T06:13:55.119879Z","iopub.execute_input":"2023-12-06T06:13:55.120235Z","iopub.status.idle":"2023-12-06T06:13:55.132069Z","shell.execute_reply.started":"2023-12-06T06:13:55.120199Z","shell.execute_reply":"2023-12-06T06:13:55.131257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Unhide below for summary of model architecture.","metadata":{}},{"cell_type":"code","source":"model = sm.Unet(\n    'resnet34', \n    classes=4,\n    input_shape=(320, 480, 3),\n    activation='sigmoid'\n)\nmodel.compile(optimizer=Nadam(lr=0.0002), loss=bce_dice_loss, metrics=[dice_coef])\nmodel.summary()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-12-06T06:13:55.133192Z","iopub.execute_input":"2023-12-06T06:13:55.133498Z","iopub.status.idle":"2023-12-06T06:14:00.778761Z","shell.execute_reply.started":"2023-12-06T06:13:55.133474Z","shell.execute_reply":"2023-12-06T06:14:00.777869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Unhide below for training history.","metadata":{}},{"cell_type":"code","source":"checkpoint = ModelCheckpoint('model.h5', save_best_only=True)\n\nhistory = model.fit_generator(\n    train_generator,\n    validation_data=val_generator,\n    callbacks=[checkpoint],\n    epochs=15\n)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-12-06T06:14:00.780019Z","iopub.execute_input":"2023-12-06T06:14:00.780322Z","iopub.status.idle":"2023-12-06T08:21:13.31094Z","shell.execute_reply.started":"2023-12-06T06:14:00.780295Z","shell.execute_reply":"2023-12-06T08:21:13.309947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluation & Submission","metadata":{}},{"cell_type":"code","source":"with open('history.json', 'w') as f:\n    json.dump(history.history, f)\n\nhistory_df = pd.DataFrame(history.history)\nhistory_df[['loss', 'val_loss']].plot()\nhistory_df[['dice_coef', 'val_dice_coef']].plot()","metadata":{"execution":{"iopub.status.busy":"2023-12-06T08:21:13.312371Z","iopub.execute_input":"2023-12-06T08:21:13.312669Z","iopub.status.idle":"2023-12-06T08:21:13.892762Z","shell.execute_reply.started":"2023-12-06T08:21:13.312642Z","shell.execute_reply":"2023-12-06T08:21:13.891893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.load_weights('model.h5')\ntest_df = []\n\nfor i in range(0, test_imgs.shape[0], 500):\n    batch_idx = list(\n        range(i, min(test_imgs.shape[0], i + 500))\n    )\n\n    test_generator = DataGenerator(\n        batch_idx,\n        df=test_imgs,\n        shuffle=False,\n        mode='predict',\n        dim=(350, 525),\n        reshape=(320, 480),\n        n_channels=3,\n        base_path='/kaggle/input/understanding_cloud_organization/test_images',\n        target_df=sub_df,\n        batch_size=1,\n        n_classes=4\n    )\n\n    batch_pred_masks = model.predict_generator(\n        test_generator, \n        workers=1,\n        verbose=1\n    )\n\n    for j, b in enumerate(batch_idx):\n        filename = test_imgs['ImageId'].iloc[b]\n        image_df = sub_df[sub_df['ImageId'] == filename].copy()\n\n        pred_masks = batch_pred_masks[j, ].round().astype(int)\n        pred_rles = build_rles(pred_masks, reshape=(350, 525))\n\n        image_df['EncodedPixels'] = pred_rles\n        test_df.append(image_df)","metadata":{"execution":{"iopub.status.busy":"2023-12-06T08:21:13.893963Z","iopub.execute_input":"2023-12-06T08:21:13.894243Z","iopub.status.idle":"2023-12-06T08:21:15.2878Z","shell.execute_reply.started":"2023-12-06T08:21:13.894218Z","shell.execute_reply":"2023-12-06T08:21:15.285889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"test_df = pd.concat(test_df)\ntest_df.drop(columns='ImageId', inplace=True)\ntest_df.to_csv('flower_submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-12-06T08:21:15.288761Z","iopub.status.idle":"2023-12-06T08:21:15.289179Z","shell.execute_reply.started":"2023-12-06T08:21:15.289011Z","shell.execute_reply":"2023-12-06T08:21:15.289028Z"},"trusted":true},"execution_count":null,"outputs":[]}]}