{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## General information\n\nIn this kernel I work with the data from Understanding Clouds from Satellite Images competition.\n```\nShallow clouds play a huge role in determining the Earth's climate. They’re also difficult to understand and to represent in climate models. By classifying different types of cloud organization, researchers at Max Planck hope to improve our physical understanding of these clouds, which in turn will help us build better climate models.\n```\n\nSo in this competition we are tasked with multiclass segmentation task: finding 4 different cloud patterns in the images. On the other hand, we make predictions for each pair of image and label separately, so this could be treated as 4 binary segmentation tasks.\nIt is important to notice that images (and masks) are `1400 x 2100`, but predicted masks should be `350 x 525`.\n\nIn this kernel I'll use (or will use in next versions) the following notable libraries:\n- [albumentations](https://github.com/albu/albumentations): this is a great library for image augmentation which makes it easier and more convenient\n- [catalyst](https://github.com/catalyst-team/catalyst): this is a great library which makes using PyTorch easier, helps with reprodicibility and contains a lot of useful utils\n- [segmentation_models_pytorch](https://github.com/qubvel/segmentation_models.pytorch): this is a great library with convenient wrappers for models, losses and other useful things\n- [pytorch-toolbelt](https://github.com/BloodAxe/pytorch-toolbelt): this is a great library with many useful shortcuts for building pytorch models\n\n\nUPD: Version 35 - changed calculation of optimal threshold and min size\n![](https://i.imgur.com/EOvz5kd.png)","metadata":{}},{"cell_type":"code","source":"!pip install catalyst==19.9.3\n!pip install git+https://github.com/qubvel/segmentation_models.pytorch\n!pip install pretrainedmodels\n!pip install pytorch_toolbelt==0.1.4\n!pip install torchvision==0.4","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-13T02:36:50.080868Z","iopub.execute_input":"2021-11-13T02:36:50.081201Z","iopub.status.idle":"2021-11-13T02:39:16.390008Z","shell.execute_reply.started":"2021-11-13T02:36:50.081136Z","shell.execute_reply":"2021-11-13T02:39:16.388824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Importing libraries","metadata":{}},{"cell_type":"code","source":"import os\nimport cv2\nimport collections\nimport time \nimport tqdm\nfrom PIL import Image\nfrom functools import partial\ntrain_on_gpu = True\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score\n\nimport torchvision\nimport torchvision.transforms as transforms\nimport torch\nfrom torch.utils.data import TensorDataset, DataLoader,Dataset\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim import lr_scheduler\nfrom torch.utils.data.sampler import SubsetRandomSampler\nfrom torch.optim.lr_scheduler import StepLR, ReduceLROnPlateau, CosineAnnealingLR\n\nimport albumentations as albu\nfrom albumentations import torch as AT\n\nfrom catalyst.data import Augmentor\nfrom catalyst.dl import utils\nfrom catalyst.data.reader import ImageReader, ScalarReader, ReaderCompose, LambdaReader\nfrom catalyst.dl.runner import SupervisedRunner\nfrom catalyst.contrib.models.segmentation import Unet\nfrom catalyst.dl.callbacks import DiceCallback, EarlyStoppingCallback, InferCallback, CheckpointCallback\n\nimport segmentation_models_pytorch as smp","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-13T02:40:33.434331Z","iopub.execute_input":"2021-11-13T02:40:33.434678Z","iopub.status.idle":"2021-11-13T02:40:33.525282Z","shell.execute_reply.started":"2021-11-13T02:40:33.434609Z","shell.execute_reply":"2021-11-13T02:40:33.523556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helper functions and classes","metadata":{}},{"cell_type":"code","source":"def get_img(x, folder: str='train_images'):\n    \"\"\"\n    Return image based on image name and folder.\n    \"\"\"\n    data_folder = f\"{path}/{folder}\"\n    image_path = os.path.join(data_folder, x)\n    img = cv2.imread(image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    return img\n\n\ndef rle_decode(mask_rle: str = '', shape: tuple = (1400, 2100)):\n    '''\n    Decode rle encoded mask.\n    \n    :param mask_rle: run-length as string formatted (start length)\n    :param shape: (height, width) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n    '''\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0] * shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape, order='F')\n\n\ndef make_mask(df: pd.DataFrame, image_name: str='img.jpg', shape: tuple = (1400, 2100)):\n    \"\"\"\n    Create mask based on df, image name and shape.\n    \"\"\"\n    encoded_masks = df.loc[df['im_id'] == image_name, 'EncodedPixels']\n    masks = np.zeros((shape[0], shape[1], 4), dtype=np.float32)\n\n    for idx, label in enumerate(encoded_masks.values):\n        if label is not np.nan:\n            mask = rle_decode(label)\n            masks[:, :, idx] = mask\n            \n    return masks\n\n\ndef to_tensor(x, **kwargs):\n    \"\"\"\n    Convert image or mask.\n    \"\"\"\n    return x.transpose(2, 0, 1).astype('float32')\n\n\ndef mask2rle(img):\n    '''\n    Convert mask to rle.\n    img: numpy array, 1 - mask, 0 - background\n    Returns run length as string formated\n    '''\n    pixels= img.T.flatten()\n    pixels = np.concatenate([[0], pixels, [0]])\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n    return ' '.join(str(x) for x in runs)\n\n\ndef visualize(image, mask, original_image=None, original_mask=None):\n    \"\"\"\n    Plot image and masks.\n    If two pairs of images and masks are passes, show both.\n    \"\"\"\n    fontsize = 14\n    class_dict = {0: 'Fish', 1: 'Flower', 2: 'Gravel', 3: 'Sugar'}\n    \n    if original_image is None and original_mask is None:\n        f, ax = plt.subplots(1, 5, figsize=(24, 24))\n\n        ax[0].imshow(image)\n        for i in range(4):\n            ax[i + 1].imshow(mask[:, :, i])\n            ax[i + 1].set_title(f'Mask {class_dict[i]}', fontsize=fontsize)\n    else:\n        f, ax = plt.subplots(2, 5, figsize=(24, 12))\n\n        ax[0, 0].imshow(original_image)\n        ax[0, 0].set_title('Original image', fontsize=fontsize)\n                \n        for i in range(4):\n            ax[0, i + 1].imshow(original_mask[:, :, i])\n            ax[0, i + 1].set_title(f'Original mask {class_dict[i]}', fontsize=fontsize)\n        \n        ax[1, 0].imshow(image)\n        ax[1, 0].set_title('Transformed image', fontsize=fontsize)\n        \n        \n        for i in range(4):\n            ax[1, i + 1].imshow(mask[:, :, i])\n            ax[1, i + 1].set_title(f'Transformed mask {class_dict[i]}', fontsize=fontsize)\n            \n            \ndef visualize_with_raw(image, mask, original_image=None, original_mask=None, raw_image=None, raw_mask=None):\n    \"\"\"\n    Plot image and masks.\n    If two pairs of images and masks are passes, show both.\n    \"\"\"\n    fontsize = 14\n    class_dict = {0: 'Fish', 1: 'Flower', 2: 'Gravel', 3: 'Sugar'}\n\n    f, ax = plt.subplots(3, 5, figsize=(24, 12))\n\n    ax[0, 0].imshow(original_image)\n    ax[0, 0].set_title('Original image', fontsize=fontsize)\n\n    for i in range(4):\n        ax[0, i + 1].imshow(original_mask[:, :, i])\n        ax[0, i + 1].set_title(f'Original mask {class_dict[i]}', fontsize=fontsize)\n\n\n    ax[1, 0].imshow(raw_image)\n    ax[1, 0].set_title('Original image', fontsize=fontsize)\n\n    for i in range(4):\n        ax[1, i + 1].imshow(raw_mask[:, :, i])\n        ax[1, i + 1].set_title(f'Raw predicted mask {class_dict[i]}', fontsize=fontsize)\n        \n    ax[2, 0].imshow(image)\n    ax[2, 0].set_title('Transformed image', fontsize=fontsize)\n\n\n    for i in range(4):\n        ax[2, i + 1].imshow(mask[:, :, i])\n        ax[2, i + 1].set_title(f'Predicted mask with processing {class_dict[i]}', fontsize=fontsize)\n            \n            \ndef plot_with_augmentation(image, mask, augment):\n    \"\"\"\n    Wrapper for `visualize` function.\n    \"\"\"\n    augmented = augment(image=image, mask=mask)\n    image_flipped = augmented['image']\n    mask_flipped = augmented['mask']\n    visualize(image_flipped, mask_flipped, original_image=image, original_mask=mask)\n    \n    \nsigmoid = lambda x: 1 / (1 + np.exp(-x))\n\n\ndef post_process(probability, threshold, min_size):\n    \"\"\"\n    Post processing of each predicted mask, components with lesser number of pixels\n    than `min_size` are ignored\n    \"\"\"\n    # don't remember where I saw it\n    mask = cv2.threshold(probability, threshold, 1, cv2.THRESH_BINARY)[1]\n    num_component, component = cv2.connectedComponents(mask.astype(np.uint8))\n    predictions = np.zeros((350, 525), np.float32)\n    num = 0\n    for c in range(1, num_component):\n        p = (component == c)\n        if p.sum() > min_size:\n            predictions[p] = 1\n            num += 1\n    return predictions, num\n\n\ndef get_training_augmentation():\n    train_transform = [\n\n        albu.HorizontalFlip(p=0.5),\n        albu.ShiftScaleRotate(scale_limit=0.5, rotate_limit=0, shift_limit=0.1, p=0.5, border_mode=0),\n        albu.GridDistortion(p=0.5),\n        albu.OpticalDistortion(p=0.5, distort_limit=2, shift_limit=0.5),\n        albu.Resize(320, 640)\n    ]\n    return albu.Compose(train_transform)\n\n\ndef get_validation_augmentation():\n    \"\"\"Add paddings to make image shape divisible by 32\"\"\"\n    test_transform = [\n        albu.Resize(320, 640)\n    ]\n    return albu.Compose(test_transform)\n\n\ndef get_preprocessing(preprocessing_fn):\n    \"\"\"Construct preprocessing transform\n    \n    Args:\n        preprocessing_fn (callbale): data normalization function \n            (can be specific for each pretrained neural network)\n    Return:\n        transform: albumentations.Compose\n    \n    \"\"\"\n    \n    _transform = [\n        albu.Lambda(image=preprocessing_fn),\n        albu.Lambda(image=to_tensor, mask=to_tensor),\n    ]\n    return albu.Compose(_transform)\n\n\ndef dice(img1, img2):\n    img1 = np.asarray(img1).astype(np.bool)\n    img2 = np.asarray(img2).astype(np.bool)\n\n    intersection = np.logical_and(img1, img2)\n\n    return 2. * intersection.sum() / (img1.sum() + img2.sum())","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data overview\n\nLet's have a look at the data first.","metadata":{}},{"cell_type":"code","source":"path = '/input/understanding_cloud_organization'\nos.listdir(path)","metadata":{"execution":{"iopub.status.busy":"2021-11-13T02:24:01.387751Z","iopub.execute_input":"2021-11-13T02:24:01.388174Z","iopub.status.idle":"2021-11-13T02:24:01.447867Z","shell.execute_reply.started":"2021-11-13T02:24:01.388113Z","shell.execute_reply":"2021-11-13T02:24:01.442454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have folders with train and test images, file with train image ids and masks and sample submission.","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(f'{path}/train.csv')\nsub = pd.read_csv(f'{path}/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2021-11-12T05:35:12.337109Z","iopub.execute_input":"2021-11-12T05:35:12.33745Z","iopub.status.idle":"2021-11-12T05:35:15.610761Z","shell.execute_reply.started":"2021-11-12T05:35:12.337399Z","shell.execute_reply":"2021-11-12T05:35:15.609383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-12T05:46:37.767199Z","iopub.execute_input":"2021-11-12T05:46:37.767514Z","iopub.status.idle":"2021-11-12T05:46:37.777517Z","shell.execute_reply.started":"2021-11-12T05:46:37.767461Z","shell.execute_reply":"2021-11-12T05:46:37.776448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-12T05:46:43.868753Z","iopub.execute_input":"2021-11-12T05:46:43.869048Z","iopub.status.idle":"2021-11-12T05:46:43.882353Z","shell.execute_reply.started":"2021-11-12T05:46:43.868996Z","shell.execute_reply":"2021-11-12T05:46:43.881673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_train = len(os.listdir(f'{path}/train_images'))\nn_test = len(os.listdir(f'{path}/test_images'))\nprint(f'There are {n_train} images in train dataset')\nprint(f'There are {n_test} images in test dataset')","metadata":{"execution":{"iopub.status.busy":"2021-11-12T05:36:04.554097Z","iopub.execute_input":"2021-11-12T05:36:04.55441Z","iopub.status.idle":"2021-11-12T05:36:05.092688Z","shell.execute_reply.started":"2021-11-12T05:36:04.55436Z","shell.execute_reply":"2021-11-12T05:36:05.092029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['Image_Label'].apply(lambda x: x.split('_')[1]).value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-11-12T05:39:30.937532Z","iopub.execute_input":"2021-11-12T05:39:30.937837Z","iopub.status.idle":"2021-11-12T05:39:30.96381Z","shell.execute_reply.started":"2021-11-12T05:39:30.937786Z","shell.execute_reply":"2021-11-12T05:39:30.963087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So we have ~5.5k images in train dataset and they can have up to 4 masks: Fish, Flower, Gravel and Sugar.","metadata":{}},{"cell_type":"code","source":"train.loc[train['EncodedPixels'].isnull() == False, 'Image_Label'].apply(lambda x: x.split('_')[1]).value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-11-12T05:41:33.543592Z","iopub.execute_input":"2021-11-12T05:41:33.543891Z","iopub.status.idle":"2021-11-12T05:41:33.568083Z","shell.execute_reply.started":"2021-11-12T05:41:33.543837Z","shell.execute_reply":"2021-11-12T05:41:33.566989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.loc[train['EncodedPixels'].isnull() == False, 'Image_Label'].apply(lambda x: x.split('_')[0]).value_counts().value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-11-12T05:43:00.974979Z","iopub.execute_input":"2021-11-12T05:43:00.975307Z","iopub.status.idle":"2021-11-12T05:43:01.004741Z","shell.execute_reply.started":"2021-11-12T05:43:00.975252Z","shell.execute_reply":"2021-11-12T05:43:01.003888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"But there are a lot of empty masks. In fact only 266 images have all four masks. It is important to remember this.","metadata":{}},{"cell_type":"code","source":"train['label'] = train['Image_Label'].apply(lambda x: x.split('_')[1])\ntrain['im_id'] = train['Image_Label'].apply(lambda x: x.split('_')[0])\n\n\nsub['label'] = sub['Image_Label'].apply(lambda x: x.split('_')[1])\nsub['im_id'] = sub['Image_Label'].apply(lambda x: x.split('_')[0])","metadata":{"execution":{"iopub.status.busy":"2021-11-12T05:47:56.931631Z","iopub.execute_input":"2021-11-12T05:47:56.931934Z","iopub.status.idle":"2021-11-12T05:47:56.983267Z","shell.execute_reply.started":"2021-11-12T05:47:56.931881Z","shell.execute_reply":"2021-11-12T05:47:56.982604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-11-12T05:48:05.187237Z","iopub.execute_input":"2021-11-12T05:48:05.187524Z","iopub.status.idle":"2021-11-12T05:48:05.200593Z","shell.execute_reply.started":"2021-11-12T05:48:05.187477Z","shell.execute_reply":"2021-11-12T05:48:05.199759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-11-12T05:48:15.976764Z","iopub.execute_input":"2021-11-12T05:48:15.977141Z","iopub.status.idle":"2021-11-12T05:48:15.99282Z","shell.execute_reply.started":"2021-11-12T05:48:15.977074Z","shell.execute_reply":"2021-11-12T05:48:15.991939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's have a look at the images and the masks.","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(25, 16))\nfor j, im_id in enumerate(np.random.choice(train['im_id'].unique(), 4)):\n    for i, (idx, row) in enumerate(train.loc[train['im_id'] == im_id].iterrows()):\n        ax = fig.add_subplot(5, 4, j * 4 + i + 1, xticks=[], yticks=[])\n        im = Image.open(f\"{path}/train_images/{row['Image_Label'].split('_')[0]}\")\n        plt.imshow(im)\n        mask_rle = row['EncodedPixels']\n        try: # label might not be there!\n            mask = rle_decode(mask_rle)\n        except:\n            mask = np.zeros((1400, 2100))\n        plt.imshow(mask, alpha=0.5, cmap='gray')\n        ax.set_title(f\"Image: {row['Image_Label'].split('_')[0]}. Label: {row['label']}\")","metadata":{"execution":{"iopub.status.busy":"2021-11-12T05:48:36.171381Z","iopub.execute_input":"2021-11-12T05:48:36.17167Z","iopub.status.idle":"2021-11-12T05:48:44.465408Z","shell.execute_reply.started":"2021-11-12T05:48:36.171623Z","shell.execute_reply":"2021-11-12T05:48:44.4631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that masks can overlap. Also we can see that clouds are really similar to fish, flower and so on. Another important point: masks are often quite big and can have seemingly empty areas.","metadata":{}},{"cell_type":"markdown","source":"## Preparing data for modelling\n\nAt first, let's create a list of unique image ids and the count of masks for images. This will allow us to make a stratified split based on this count.","metadata":{}},{"cell_type":"code","source":"id_mask_count = train.loc[train['EncodedPixels'].isnull() == False, 'Image_Label'].apply(lambda x: x.split('_')[0]).value_counts().\\\nreset_index().rename(columns={'index': 'img_id', 'Image_Label': 'count'})\ntrain_ids, valid_ids = train_test_split(id_mask_count['img_id'].values, random_state=42, stratify=id_mask_count['count'], test_size=0.1)\ntest_ids = sub['Image_Label'].apply(lambda x: x.split('_')[0]).drop_duplicates().values","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploring augmentations with albumentations\n\nOne of important things while working with images is choosing good augmentations. There are a lot of them, let's have a look at augmentations from albumentations!","metadata":{}},{"cell_type":"code","source":"image_name = '8242ba0.jpg'\nimage = get_img(image_name)\nmask = make_mask(train, image_name)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize(image, mask)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is how original image and its masks look like. Let's try adding some augmentations","metadata":{}},{"cell_type":"code","source":"plot_with_augmentation(image, mask, albu.HorizontalFlip(p=1))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_with_augmentation(image, mask, albu.VerticalFlip(p=1))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_with_augmentation(image, mask, albu.RandomRotate90(p=1))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_with_augmentation(image, mask, albu.ElasticTransform(p=1, alpha=120, sigma=120 * 0.05, alpha_affine=120 * 0.03))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_with_augmentation(image, mask, albu.GridDistortion(p=1))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_with_augmentation(image, mask, albu.OpticalDistortion(p=1, distort_limit=2, shift_limit=0.5))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Setting up data for training in Catalyst","metadata":{}},{"cell_type":"code","source":"class CloudDataset(Dataset):\n    def __init__(self, df: pd.DataFrame = None, datatype: str = 'train', img_ids: np.array = None,\n                 transforms = albu.Compose([albu.HorizontalFlip(),AT.ToTensor()]),\n                preprocessing=None):\n        self.df = df\n        if datatype != 'test':\n            self.data_folder = f\"{path}/train_images\"\n        else:\n            self.data_folder = f\"{path}/test_images\"\n        self.img_ids = img_ids\n        self.transforms = transforms\n        self.preprocessing = preprocessing\n\n    def __getitem__(self, idx):\n        image_name = self.img_ids[idx]\n        mask = make_mask(self.df, image_name)\n        image_path = os.path.join(self.data_folder, image_name)\n        img = cv2.imread(image_path)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        augmented = self.transforms(image=img, mask=mask)\n        img = augmented['image']\n        mask = augmented['mask']\n        if self.preprocessing:\n            preprocessed = self.preprocessing(image=img, mask=mask)\n            img = preprocessed['image']\n            mask = preprocessed['mask']\n        return img, mask\n\n    def __len__(self):\n        return len(self.img_ids)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we define model and training parameters","metadata":{}},{"cell_type":"code","source":"ENCODER = 'resnet50'\nENCODER_WEIGHTS = 'imagenet'\nDEVICE = 'cuda'\n\nACTIVATION = None\nmodel = smp.Unet(\n    encoder_name=ENCODER, \n    encoder_weights=ENCODER_WEIGHTS, \n    classes=4, \n    activation=ACTIVATION,\n)\npreprocessing_fn = smp.encoders.get_preprocessing_fn(ENCODER, ENCODER_WEIGHTS)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_workers = 0\nbs = 16\ntrain_dataset = CloudDataset(df=train, datatype='train', img_ids=train_ids, transforms = get_training_augmentation(), preprocessing=get_preprocessing(preprocessing_fn))\nvalid_dataset = CloudDataset(df=train, datatype='valid', img_ids=valid_ids, transforms = get_validation_augmentation(), preprocessing=get_preprocessing(preprocessing_fn))\n\ntrain_loader = DataLoader(train_dataset, batch_size=bs, shuffle=True, num_workers=num_workers)\nvalid_loader = DataLoader(valid_dataset, batch_size=bs, shuffle=False, num_workers=num_workers)\n\nloaders = {\n    \"train\": train_loader,\n    \"valid\": valid_loader\n}","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_epochs = 19\nlogdir = \"./logs/segmentation\"\n\n# model, criterion, optimizer\noptimizer = torch.optim.Adam([\n    {'params': model.decoder.parameters(), 'lr': 1e-2}, \n    {'params': model.encoder.parameters(), 'lr': 1e-3},  \n])\nscheduler = ReduceLROnPlateau(optimizer, factor=0.15, patience=2)\ncriterion = smp.utils.losses.BCEDiceLoss(eps=1.)\nrunner = SupervisedRunner()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model training","metadata":{}},{"cell_type":"code","source":"runner.train(\n    model=model,\n    criterion=criterion,\n    optimizer=optimizer,\n    scheduler=scheduler,\n    loaders=loaders,\n    callbacks=[DiceCallback(), EarlyStoppingCallback(patience=5, min_delta=0.001)],\n    logdir=logdir,\n    num_epochs=num_epochs,\n    verbose=True\n)","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"utils.plot_metrics(\n    logdir=logdir, \n    # specify which metrics we want to plot\n    metrics=[\"loss\", \"dice\", 'lr', '_base/lr']\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploring predictions\nLet's make predictions on validation dataset.\n\nAt first we need to optimize thresholds ","metadata":{}},{"cell_type":"code","source":"encoded_pixels = []\nloaders = {\"infer\": valid_loader}\nrunner.infer(\n    model=model,\n    loaders=loaders,\n    callbacks=[\n        CheckpointCallback(\n            resume=f\"{logdir}/checkpoints/best.pth\"),\n        InferCallback()\n    ],\n)\nvalid_masks = []\nprobabilities = np.zeros((2220, 350, 525))\nfor i, (batch, output) in enumerate(tqdm.tqdm(zip(\n        valid_dataset, runner.callbacks[0].predictions[\"logits\"]))):\n    image, mask = batch\n    for m in mask:\n        if m.shape != (350, 525):\n            m = cv2.resize(m, dsize=(525, 350), interpolation=cv2.INTER_LINEAR)\n        valid_masks.append(m)\n\n    for j, probability in enumerate(output):\n        if probability.shape != (350, 525):\n            probability = cv2.resize(probability, dsize=(525, 350), interpolation=cv2.INTER_LINEAR)\n        probabilities[i * 4 + j, :, :] = probability","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Find optimal values\n\nFirst of all, my thanks to @samusram for finding a mistake in my validation\nhttps://www.kaggle.com/c/understanding_cloud_organization/discussion/107711#622412\n\nAnd now I find optimal values separately for each class.","metadata":{}},{"cell_type":"code","source":"class_params = {}\nfor class_id in range(4):\n    print(class_id)\n    attempts = []\n    for t in range(0, 100, 5):\n        t /= 100\n        for ms in [0, 100, 1200, 5000, 10000]:\n            masks = []\n            for i in range(class_id, len(probabilities), 4):\n                probability = probabilities[i]\n                predict, num_predict = post_process(sigmoid(probability), t, ms)\n                masks.append(predict)\n\n            d = []\n            for i, j in zip(masks, valid_masks[class_id::4]):\n                if (i.sum() == 0) & (j.sum() == 0):\n                    d.append(1)\n                else:\n                    d.append(dice(i, j))\n\n            attempts.append((t, ms, np.mean(d)))\n\n    attempts_df = pd.DataFrame(attempts, columns=['threshold', 'size', 'dice'])\n\n\n    attempts_df = attempts_df.sort_values('dice', ascending=False)\n    print(attempts_df.head())\n    best_threshold = attempts_df['threshold'].values[0]\n    best_size = attempts_df['size'].values[0]\n    \n    class_params[class_id] = (best_threshold, best_size)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(class_params)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.lineplot(x='threshold', y='dice', hue='size', data=attempts_df);\nplt.title('Threshold and min size vs dice for one of the classes');","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's have a look at our masks.","metadata":{}},{"cell_type":"code","source":"for i, (input, output) in enumerate(zip(\n        valid_dataset, runner.callbacks[0].predictions[\"logits\"])):\n    image, mask = input\n        \n    image_vis = image.transpose(1, 2, 0)\n    mask = mask.astype('uint8').transpose(1, 2, 0)\n    pr_mask = np.zeros((350, 525, 4))\n    for j in range(4):\n        probability = cv2.resize(output.transpose(1, 2, 0)[:, :, j], dsize=(525, 350), interpolation=cv2.INTER_LINEAR)\n        pr_mask[:, :, j], _ = post_process(sigmoid(probability), class_params[j][0], class_params[j][1])\n    #pr_mask = (sigmoid(output) > best_threshold).astype('uint8').transpose(1, 2, 0)\n    \n        \n    visualize_with_raw(image=image_vis, mask=pr_mask, original_image=image_vis, original_mask=mask, raw_image=image_vis, raw_mask=output.transpose(1, 2, 0))\n    \n    if i >= 2:\n        break","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Predicting","metadata":{}},{"cell_type":"code","source":"import gc\ntorch.cuda.empty_cache()\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_dataset = CloudDataset(df=sub, datatype='test', img_ids=test_ids, transforms = get_validation_augmentation(), preprocessing=get_preprocessing(preprocessing_fn))\ntest_loader = DataLoader(test_dataset, batch_size=8, shuffle=False, num_workers=0)\n\nloaders = {\"test\": test_loader}","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"encoded_pixels = []\nimage_id = 0\nfor i, test_batch in enumerate(tqdm.tqdm(loaders['test'])):\n    runner_out = runner.predict_batch({\"features\": test_batch[0].cuda()})['logits']\n    for i, batch in enumerate(runner_out):\n        for probability in batch:\n            \n            probability = probability.cpu().detach().numpy()\n            if probability.shape != (350, 525):\n                probability = cv2.resize(probability, dsize=(525, 350), interpolation=cv2.INTER_LINEAR)\n            predict, num_predict = post_process(sigmoid(probability), class_params[image_id % 4][0], class_params[image_id % 4][1])\n            if num_predict == 0:\n                encoded_pixels.append('')\n            else:\n                r = mask2rle(predict)\n                encoded_pixels.append(r)\n            image_id += 1","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub['EncodedPixels'] = encoded_pixels\nsub.to_csv('submission.csv', columns=['Image_Label', 'EncodedPixels'], index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}