{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Image Segmentation From scratch using Pytorch\n"},{"metadata":{},"cell_type":"markdown","source":"This is my first hands on with image segmentation and I tried to learn from existing pytorch notebooks.\nOne thing I imediately noticed is Using High level frameworks like catalyst is very convinient but For\nlearner like me who came here to learn it is difficult to know what is happening under the hood and\nthat's what we are here for,that is what we want to learn. so I wrote this kernel.\n### It does all things from scartch so we can see what's happening "},{"metadata":{},"cell_type":"markdown","source":"## Features of this kernel \n* Using vanila Unet Architecture\n* Deterministic behaviour for reproducability\n* K-fold cross validation is already Implemented (i.e. data spliting is done)\n* loss function is also implmented for clearity\n* Training loop is open to see exactly what is happening\n* Processing output by removing mask that occur on black part of input image\n* Drawing convex hull before optimizing thresholds\n..."},{"metadata":{},"cell_type":"markdown","source":"## Unet architecture\n![Unet](https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/u-net-architecture.png)"},{"metadata":{},"cell_type":"markdown","source":"Finally Huge thanks to **artgor, ryches, ratthachat, [repo1](https://github.com/qubvel/segmentation_models.pytorch/blob/master/segmentation_models_pytorch/utils/losses.py) ,[repo2](https://github.com/milesial/Pytorch-UNet) ** for their code\n\n### **references**\n1. https://www.kaggle.com/artgor/segmentation-in-pytorch-using-convenient-tools\n2. https://www.kaggle.com/ryches/turbo-charging-andrew-s-pytorch\n3. https://github.com/qubvel/segmentation_models.pytorch/blob/master/segmentation_models_pytorch/utils/losses.py\n4. https://github.com/milesial/Pytorch-UNet\n5. https://www.kaggle.com/ratthachat/cloud-convexhull-polygon-postprocessing-no-gpu"},{"metadata":{},"cell_type":"markdown","source":"# Imports"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport cv2\nimport time\nimport tqdm\nimport random\nimport collections\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom PIL import Image\nfrom functools import partial\nimport matplotlib.pyplot as plt\nfrom tqdm.auto import tqdm as tq\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import roc_auc_score\n\n\nimport torch\nimport torchvision\nimport torch.nn as nn\nimport torch.optim as optim\nimport torch.nn.functional as F\nfrom torch.optim import lr_scheduler\nimport torchvision.transforms as transforms\nfrom torch.utils.data.sampler import SubsetRandomSampler\nfrom torch.utils.data import TensorDataset, DataLoader, Dataset\nfrom torch.optim.lr_scheduler import StepLR, ReduceLROnPlateau\n\n# ablumentations for easy image augmentation for input as well as output\nimport albumentations as albu\n# from albumentations import torch as AT\nplt.style.use('bmh')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# seeding function for reproducibility\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n\n\ndef resize_it(x):\n    if x.shape != (350, 525):\n        x = cv2.resize(x, dsize=(525, 350), interpolation=cv2.INTER_LINEAR)\n    return x\n\n\n# Dataset class\nclass CloudDataset(Dataset):\n    def __init__(\n        self,\n        df: pd.DataFrame = None,\n        datatype: str = \"train\",\n        img_ids: np.array = None,\n        transforms=albu.Compose([albu.HorizontalFlip()]), #, AT.ToTensor()\n    ):\n        self.df = df\n        if datatype != \"test\":\n            self.data_folder = f\"{img_paths}/train_images_525/train_images_525\"\n        else:\n            self.data_folder = f\"{img_paths}/test_images_525/test_images_525\"\n        self.img_ids = img_ids\n        self.transforms = transforms\n\n    def __getitem__(self, idx):\n        image_name = self.img_ids[idx]\n        mask = make_mask(self.df, image_name)\n        image_path = os.path.join(self.data_folder, image_name)\n        img = cv2.imread(image_path)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        augmented = self.transforms(image=img, mask=mask)\n        img = np.transpose(augmented[\"image\"], [2, 0, 1])\n        mask = np.transpose(augmented[\"mask\"], [2, 0, 1])\n        return img, mask\n\n    def __len__(self):\n        return len(self.img_ids)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Helper functions"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# helper functions\nclass_names = ['Fish', 'Flower', 'Sugar', 'Gravel']\n\ndef draw_convex_hull(mask, mode='convex'):\n    \n    img = np.zeros(mask.shape)\n    contours, hier = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)\n\n    for c in contours:\n        if mode=='rect': # simple rectangle\n            x, y, w, h = cv2.boundingRect(c)\n            cv2.rectangle(img, (x, y), (x+w, y+h), (255, 255, 255), -1)\n        if mode=='convex': # minimum convex hull\n            hull = cv2.convexHull(c)\n            cv2.drawContours(img, [hull], 0, (255, 255, 255),-1)\n        else: # minimum area rectangle\n            rect = cv2.minAreaRect(c)\n            box = cv2.boxPoints(rect)\n            box = np.int0(box)\n            cv2.drawContours(img, [box], 0, (255, 255, 255),-1)\n    return img/255.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"def get_img(x, folder: str = \"train_images_525/train_images_525\"):\n    \"\"\"\n    Return image based on image name and folder.\n    \"\"\"\n    data_folder = f\"{img_paths}/{folder}\"\n    image_path = os.path.join(data_folder, x)\n    img = cv2.imread(image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    return img\n\n\n# def rle_decode(mask_rle: str = \"\", shape: tuple = (1400, 2100)):\n#     \"\"\"\n#     Decode rle encoded mask.\n\n#     :param mask_rle: run-length as string formatted (start length)\n#     :param shape: (height, width) of array to return\n#     Returns numpy array, 1 - mask, 0 - background\n#     \"\"\"\n#     s = mask_rle.split()\n#     starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n#     starts -= 1\n#     ends = starts + lengths\n#     img = np.zeros(shape[0] * shape[1], dtype=np.uint8)\n#     for lo, hi in zip(starts, ends):\n#         img[lo:hi] = 1\n#     return img.reshape(shape, order=\"F\")\n\ndef make_mask(df,img_name):\n    masks=np.zeros((350,525,4),dtype=np.float32)\n    for i,j in enumerate([\"Fish\", \"Flower\", \"Gravel\", \"Sugar\"]):\n        val=cv2.imread('../input/understanding-clouds-resized/train_masks_525/train_masks_525/'+j+img_name)\n        if val is not None:\n            val=cv2.resize(val,(525, 350))\n            masks[:,:,i]=val[:,:,0]\n#             masks.append()\n#     masks=np.asarray(masks).reshape(350,525,4)\n    masks=masks/255\n    return masks\n# def make_mask(df: pd.DataFrame, image_name: str = \"img.jpg\", shape: tuple = (350, 525)):\n#     \"\"\"\n#     Create mask based on df, image name and shape.\n#     \"\"\"\n#     masks = np.zeros((shape[0], shape[1], 4), dtype=np.float32)\n#     df = df[df[\"im_id\"] == image_name]\n#     for idx, im_name in enumerate(df[\"im_id\"].values):\n#         for classidx, classid in enumerate([\"Fish\", \"Flower\", \"Gravel\", \"Sugar\"]):\n#             mask = cv2.imread(\n#                 \"../input/understanding-clouds-resized/train_masks_525/train_masks_525/\"\n#                 + classid\n#                 + im_name\n#             )\n# #             assert im_name == image_name\n# #             print(im_name,image_name)\n#             if mask is None:\n#                 continue\n#             if mask[:, :, 0].shape != (350, 525):\n#                 mask = cv2.resize(mask, (525, 350))\n#             masks[:, :, classidx] = mask[:, :, 0]\n#     masks = masks / 255\n#     return masks\n\n\ndef to_tensor(x, **kwargs):\n    \"\"\"\n    Convert image or mask.\n    \"\"\"\n    return x.transpose(2, 0, 1).astype(\"float32\")\n\n\ndef mask2rle(img):\n    \"\"\"\n    Convert mask to rle.\n    img: numpy array, 1 - mask, 0 - background\n    Returns run length as string formated\n    \"\"\"\n    pixels = img.T.flatten()\n    pixels = np.concatenate([[0], pixels, [0]])\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n    return \" \".join(str(x) for x in runs)\n\n\ndef visualize(image, mask, original_image=None, original_mask=None):\n    \"\"\"\n    Plot image and masks.\n    If two pairs of images and masks are passes, show both.\n    \"\"\"\n    fontsize = 14\n    class_dict = {0: \"Fish\", 1: \"Flower\", 2: \"Gravel\", 3: \"Sugar\"}\n\n    if original_image is None and original_mask is None:\n        f, ax = plt.subplots(1, 5, figsize=(24, 24))\n\n        ax[0].imshow(image)\n        for i in range(4):\n            ax[i + 1].imshow(mask[:, :, i])\n            ax[i + 1].set_title(f\"Mask {class_dict[i]}\", fontsize=fontsize)\n    else:\n        f, ax = plt.subplots(2, 5, figsize=(24, 12))\n\n        ax[0, 0].imshow(original_image)\n        ax[0, 0].set_title(\"Original image\", fontsize=fontsize)\n\n        for i in range(4):\n            ax[0, i + 1].imshow(original_mask[:, :, i])\n            ax[0, i + 1].set_title(f\"Original mask {class_dict[i]}\", fontsize=fontsize)\n\n        ax[1, 0].imshow(image)\n        ax[1, 0].set_title(\"Transformed image\", fontsize=fontsize)\n\n        for i in range(4):\n            ax[1, i + 1].imshow(mask[:, :, i])\n            ax[1, i + 1].set_title(\n                f\"Transformed mask {class_dict[i]}\", fontsize=fontsize\n            )\n\n\ndef visualize_with_raw(\n    image, mask, original_image=None, original_mask=None, raw_image=None, raw_mask=None\n):\n    \"\"\"\n    Plot image and masks.\n    If two pairs of images and masks are passes, show both.\n    \"\"\"\n    fontsize = 14\n    class_dict = {0: \"Fish\", 1: \"Flower\", 2: \"Gravel\", 3: \"Sugar\"}\n\n    f, ax = plt.subplots(3, 5, figsize=(24, 12))\n\n    ax[0, 0].imshow(original_image)\n    ax[0, 0].set_title(\"Original image\", fontsize=fontsize)\n\n    for i in range(4):\n        ax[0, i + 1].imshow(original_mask[:, :, i])\n        ax[0, i + 1].set_title(f\"Original mask {class_dict[i]}\", fontsize=fontsize)\n\n    ax[1, 0].imshow(raw_image)\n    ax[1, 0].set_title(\"Original image\", fontsize=fontsize)\n\n    for i in range(4):\n        ax[1, i + 1].imshow(raw_mask[:, :, i])\n        ax[1, i + 1].set_title(f\"Raw predicted mask {class_dict[i]}\", fontsize=fontsize)\n\n    ax[2, 0].imshow(image)\n    ax[2, 0].set_title(\"Transformed image\", fontsize=fontsize)\n\n    for i in range(4):\n        ax[2, i + 1].imshow(mask[:, :, i])\n        ax[2, i + 1].set_title(\n            f\"Predicted mask with processing {class_dict[i]}\", fontsize=fontsize\n        )\n\n\ndef plot_with_augmentation(image, mask, augment):\n    \"\"\"\n    Wrapper for `visualize` function.\n    \"\"\"\n    augmented = augment(image=image, mask=mask)\n    image_flipped = augmented[\"image\"]\n    mask_flipped = augmented[\"mask\"]\n    visualize(image_flipped, mask_flipped, original_image=image, original_mask=mask)\n\n\n# sigmoid = lambda x: 1 / (1 + np.exp(-x))\ndef sigmoid(x):\n    return 1 / (1 + np.exp(-x))\n\n\n\ndef post_process(probability, threshold, min_size):\n    \"\"\"\n    This is slightly different from other kernels as we draw convex hull here itself.\n    Post processing of each predicted mask, components with lesser number of pixels\n    than `min_size` are ignored\n    \"\"\"\n    # don't remember where I saw it\n    mask = (cv2.threshold(probability, threshold, 1, cv2.THRESH_BINARY)[1])\n    mask = draw_convex_hull(mask.astype(np.uint8))\n    num_component, component = cv2.connectedComponents(mask.astype(np.uint8))\n    predictions = np.zeros((350, 525), np.float32)\n    num = 0\n    for c in range(1, num_component):\n        p = component == c\n        if p.sum() > min_size:\n            predictions[p] = 1\n            num += 1\n    return predictions, num\n\ndef get_training_augmentation():\n#     train_transform = [\n#         albu.HorizontalFlip(p=0.5),\n#         albu.ShiftScaleRotate(\n#             scale_limit=0.5,\n#             rotate_limit=0,\n#             shift_limit=0.1,\n#             p=0.5,\n#             border_mode=0\n#         ),\n#         albu.GridDistortion(p=0.5),\n#         albu.Resize(320, 640),\n#         albu.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),\n#     ]\n    train_transform = [\n        albu.ShiftScaleRotate(scale_limit=0.5, rotate_limit=0, shift_limit=0.1, p=0.6, border_mode=0),\n        albu.OneOf([\n            albu.ElasticTransform(p=0.5, alpha=50, sigma=120 * 0.02, alpha_affine=120 * 0.02),\n            albu.GridDistortion(p=0.5),\n            albu.OpticalDistortion(p=0.5, distort_limit=0.4, shift_limit=0.5)\n        ], p=0.8),\n        albu.RandomRotate90(p=0.5),\n        albu.Resize(320, 640),\n        albu.VerticalFlip(p=0.5),\n        albu.HorizontalFlip(p=0.5),\n        albu.OneOf([\n            albu.IAASharpen(alpha=(0.1, 0.3), p=0.5),\n            albu.CLAHE(p=0.8),\n            albu.GaussNoise(var_limit=(10.0, 50.0), p=0.5),\n            #GaussianBlur(blur_limit=3, p=0.5),\n            albu.ISONoise(color_shift=(0.01, 0.05), intensity=(0.1, 0.5), p=0.3),\n        ], p=0.8),\n        albu.RandomBrightnessContrast(p=0.8),\n        albu.RandomGamma(p=0.8),\n        albu.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))\n    ]\n    return albu.Compose(train_transform)\n\n\ndef get_validation_augmentation():\n    \"\"\"Add paddings to make image shape divisible by 32\"\"\"\n    test_transform = [\n        albu.Resize(320, 640),\n        albu.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),\n    ]\n    return albu.Compose(test_transform)\n\n\ndef get_preprocessing(preprocessing_fn):\n    \"\"\"Construct preprocessing transform\n\n    Args:\n        preprocessing_fn (callbale): data normalization function\n            (can be specific for each pretrained neural network)\n    Return:\n        transform: albumentations.Compose\n\n    \"\"\"\n\n    _transform = [\n        albu.Lambda(image=preprocessing_fn),\n        albu.Lambda(image=to_tensor, mask=to_tensor),\n    ]\n    return albu.Compose(_transform)\n\n\ndef dice(img1, img2):\n#     print(\"Pre\",img1)\n    img1 = np.asarray(img1).astype(np.bool)\n#     print(\"Post\",img1)\n    img2 = np.asarray(img2).astype(np.bool)\n\n    intersection = np.logical_and(img1, img2)\n\n    return 2.0 * intersection.sum() / (img1.sum() + img2.sum())\n\ndef dice_no_threshold(\n    outputs: torch.Tensor,\n    targets: torch.Tensor,\n    eps: float = 1e-7,\n    threshold: float = None,\n):\n    \"\"\"\n    Reference:\n    https://catalyst-team.github.io/catalyst/_modules/catalyst/dl/utils/criterion/dice.html\n    \"\"\"\n    if threshold is not None:\n        outputs = (outputs > threshold).float()\n\n    intersection = torch.sum(targets * outputs)\n    union = torch.sum(targets) + torch.sum(outputs)\n    dice = 2 * intersection / (union + eps)\n\n    return dice\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = \"../input/understanding_cloud_organization\"\nimg_paths = \"../input/understanding-clouds-resized\"\ntrain_on_gpu = torch.cuda.is_available()\nSEED = 42\nMODEL_NO = 0 # in K-fold\nN_FOLDS = 10 # in K-fold\nseed_everything(SEED)\nos.listdir(path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Make split in train test validation"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(f\"{path}/train.csv\")\ntrain[\"label\"] = train[\"Image_Label\"].apply(lambda x: x.split(\"_\")[1])\ntrain[\"im_id\"] = train[\"Image_Label\"].apply(lambda x: x.split(\"_\")[0])\n\nsub = pd.read_csv(f\"{path}/sample_submission.csv\")\nsub[\"label\"] = sub[\"Image_Label\"].apply(lambda x: x.split(\"_\")[1])\nsub[\"im_id\"] = sub[\"Image_Label\"].apply(lambda x: x.split(\"_\")[0])\n\n# split data\nid_mask_count = (\n    train.loc[train[\"EncodedPixels\"].isnull() == False, \"Image_Label\"]\n    .apply(lambda x: x.split(\"_\")[0])\n    .value_counts()\n    .sort_index()\n    .reset_index()\n    .rename(columns={\"index\": \"img_id\", \"Image_Label\": \"count\"})\n)\nids = id_mask_count[\"img_id\"].values\nli = [\n    [train_index, test_index]\n    for train_index, test_index in StratifiedKFold(\n        n_splits=N_FOLDS, random_state=SEED\n    ).split(ids, id_mask_count[\"count\"])\n]\ntrain_ids, valid_ids = ids[li[MODEL_NO][0]], ids[li[MODEL_NO][1]]\ntest_ids = sub[\"Image_Label\"].apply(lambda x: x.split(\"_\")[0]).drop_duplicates().values\n\nprint(f\"training set   {train_ids[:5]}.. with length {len(train_ids)}\")\nprint(f\"validation set {valid_ids[:5]}.. with length {len(valid_ids)}\")\nprint(f\"testing set    {test_ids[:5]}.. with length {len(test_ids)}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# define dataset and dataloader\nnum_workers = 2\nbs = 8\ntrain_dataset = CloudDataset(\n    df=train,\n    datatype=\"train\",\n    img_ids=train_ids,\n    transforms=get_training_augmentation(),\n)\nvalid_dataset = CloudDataset(\n    df=train,\n    datatype=\"valid\",\n    img_ids=valid_ids,\n    transforms=get_validation_augmentation(),\n)\n\ntrain_loader = DataLoader(\n    train_dataset, batch_size=bs, shuffle=True, num_workers=num_workers\n)\nvalid_loader = DataLoader(\n    valid_dataset, batch_size=bs, shuffle=False, num_workers=num_workers\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model Definition"},{"metadata":{"trusted":true},"cell_type":"code","source":"class double_conv(nn.Module):\n    \"\"\"(conv => BN => ReLU) * 2\"\"\"\n\n    def __init__(self, in_ch, out_ch):\n        super(double_conv, self).__init__()\n        self.conv = nn.Sequential(\n            nn.Conv2d(in_ch, out_ch, 3, padding=1),\n            nn.BatchNorm2d(out_ch),\n            nn.ReLU(inplace=True),\n            nn.Conv2d(out_ch, out_ch, 3, padding=1),\n            nn.BatchNorm2d(out_ch),\n            nn.ReLU(inplace=True),\n        )\n\n    def forward(self, x):\n        x = self.conv(x)\n        return x\n\n\nclass inconv(nn.Module):\n    def __init__(self, in_ch, out_ch):\n        super(inconv, self).__init__()\n        self.conv = double_conv(in_ch, out_ch)\n\n    def forward(self, x):\n        x = self.conv(x)\n        return x\n\n\nclass down(nn.Module):\n    def __init__(self, in_ch, out_ch):\n        super(down, self).__init__()\n        self.mpconv = nn.Sequential(nn.MaxPool2d(2), double_conv(in_ch, out_ch))\n\n    def forward(self, x):\n        x = self.mpconv(x)\n        return x\n\n\nclass up(nn.Module):\n    def __init__(self, in_ch, out_ch, bilinear=True):\n        super(up, self).__init__()\n\n        if bilinear:\n            self.up = nn.Upsample(scale_factor=2, mode=\"bilinear\", align_corners=True)\n        else:\n            self.up = nn.ConvTranspose2d(in_ch // 2, in_ch // 2, 2, stride=2)\n\n        self.conv = double_conv(in_ch, out_ch)\n\n    def forward(self, x1, x2):\n        x1 = self.up(x1)\n\n        # input is CHW\n        diffY = x2.size()[2] - x1.size()[2]\n        diffX = x2.size()[3] - x1.size()[3]\n\n        x1 = F.pad(x1, (diffX // 2, diffX - diffX // 2, diffY // 2, diffY - diffY // 2))\n        \n        x = torch.cat([x2, x1], dim=1)\n        return self.conv(x)\n\n\nclass outconv(nn.Module):\n    def __init__(self, in_ch, out_ch):\n        super(outconv, self).__init__()\n        self.conv = nn.Conv2d(in_ch, out_ch, 1)\n\n    def forward(self, x):\n        x = self.conv(x)\n        return x\n\n\nclass UNet(nn.Module):\n    def __init__(self, n_channels, n_classes):\n        super(UNet, self).__init__()\n        self.inc = inconv(n_channels, 64)\n        self.down1 = down(64, 128)\n        self.down2 = down(128, 256)\n        self.down3 = down(256, 512)\n        self.down4 = down(512, 512)\n        self.up1 = up(1024, 256, False)\n        self.up2 = up(512, 128, False)\n        self.up3 = up(256, 64, False)\n        self.up4 = up(128, 64, False)\n        self.outc = outconv(64, n_classes)\n\n    def forward(self, x):\n        x1 = self.inc(x)\n        x2 = self.down1(x1)\n        x3 = self.down2(x2)\n        x4 = self.down3(x3)\n        x5 = self.down4(x4)\n        x = self.up1(x5, x4)\n        x = self.up2(x, x3)\n        x = self.up3(x, x2)\n        x = self.up4(x, x1)\n        x = self.outc(x)\n        return torch.sigmoid(x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install segmentation-models-pytorch\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import segmentation_models_pytorch as smp\nmodel = smp.Unet('densenet169', encoder_weights='imagenet', classes=4, activation='sigmoid').float()\n# model = UNet(n_channels=3, n_classes=4).float()\nif train_on_gpu:\n    model.cuda()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# model # print Model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loss function definition"},{"metadata":{"trusted":true},"cell_type":"code","source":"def f_score(pr, gt, beta=1, eps=1e-7, threshold=None, activation='sigmoid'):\n    \"\"\"\n    Args:\n        pr (torch.Tensor): A list of predicted elements\n        gt (torch.Tensor):  A list of elements that are to be predicted\n        eps (float): epsilon to avoid zero division\n        threshold: threshold for outputs binarization\n    Returns:\n        float: IoU (Jaccard) score\n    \"\"\"\n\n    if activation is None or activation == \"none\":\n        activation_fn = lambda x: x\n    elif activation == \"sigmoid\":\n        activation_fn = torch.nn.Sigmoid()\n    elif activation == \"softmax2d\":\n        activation_fn = torch.nn.Softmax2d()\n    else:\n        raise NotImplementedError(\n            \"Activation implemented for sigmoid and softmax2d\"\n        )\n\n    pr = activation_fn(pr)\n\n    if threshold is not None:\n        pr = (pr > threshold).float()\n\n\n    tp = torch.sum(gt * pr)\n    fp = torch.sum(pr) - tp\n    fn = torch.sum(gt) - tp\n\n    score = ((1 + beta ** 2) * tp + eps) \\\n            / ((1 + beta ** 2) * tp + beta ** 2 * fn + fp + eps)\n\n    return score\n\n\nclass DiceLoss(nn.Module):\n    __name__ = 'dice_loss'\n\n    def __init__(self, eps=1e-7, activation='sigmoid'):\n        super().__init__()\n        self.activation = activation\n        self.eps = eps\n\n    def forward(self, y_pr, y_gt):\n        return 1 - f_score(y_pr, y_gt, beta=1., \n                           eps=self.eps, threshold=None, \n                           activation=self.activation)\n\n\nclass BCEDiceLoss(DiceLoss):\n    __name__ = 'bce_dice_loss'\n\n    def __init__(self, eps=1e-7, activation='sigmoid', lambda_dice=1.0, lambda_bce=1.0):\n        super().__init__(eps, activation)\n        if activation == None:\n            self.bce = nn.BCELoss(reduction='mean')\n        else:\n            self.bce = nn.BCEWithLogitsLoss(reduction='mean')\n        self.lambda_dice=lambda_dice\n        self.lambda_bce=lambda_bce\n\n    def forward(self, y_pr, y_gt):\n        dice = super().forward(y_pr, y_gt)\n        bce = self.bce(y_pr, y_gt)\n        return (self.lambda_dice*dice) + (self.lambda_bce* bce)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## RAdam Optimizer"},{"metadata":{"trusted":true},"cell_type":"code","source":"import math\nimport torch\nfrom torch.optim.optimizer import Optimizer, required\n\nclass RAdam(Optimizer):\n\n    def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0):\n        if not 0.0 <= lr:\n            raise ValueError(\"Invalid learning rate: {}\".format(lr))\n        if not 0.0 <= eps:\n            raise ValueError(\"Invalid epsilon value: {}\".format(eps))\n        if not 0.0 <= betas[0] < 1.0:\n            raise ValueError(\"Invalid beta parameter at index 0: {}\".format(betas[0]))\n        if not 0.0 <= betas[1] < 1.0:\n            raise ValueError(\"Invalid beta parameter at index 1: {}\".format(betas[1]))\n            \n        defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay)\n        self.buffer = [[None, None, None] for ind in range(10)]\n        super(RAdam, self).__init__(params, defaults)\n\n    def __setstate__(self, state):\n        super(RAdam, self).__setstate__(state)\n\n    def step(self, closure=None):\n\n        loss = None\n        if closure is not None:\n            loss = closure()\n\n        for group in self.param_groups:\n\n            for p in group['params']:\n                if p.grad is None:\n                    continue\n                grad = p.grad.data.float()\n                if grad.is_sparse:\n                    raise RuntimeError('RAdam does not support sparse gradients')\n\n                p_data_fp32 = p.data.float()\n\n                state = self.state[p]\n\n                if len(state) == 0:\n                    state['step'] = 0\n                    state['exp_avg'] = torch.zeros_like(p_data_fp32)\n                    state['exp_avg_sq'] = torch.zeros_like(p_data_fp32)\n                else:\n                    state['exp_avg'] = state['exp_avg'].type_as(p_data_fp32)\n                    state['exp_avg_sq'] = state['exp_avg_sq'].type_as(p_data_fp32)\n\n                exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']\n                beta1, beta2 = group['betas']\n\n                exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)\n                exp_avg.mul_(beta1).add_(1 - beta1, grad)\n\n                state['step'] += 1\n                buffered = self.buffer[int(state['step'] % 10)]\n                if state['step'] == buffered[0]:\n                    N_sma, step_size = buffered[1], buffered[2]\n                else:\n                    buffered[0] = state['step']\n                    beta2_t = beta2 ** state['step']\n                    N_sma_max = 2 / (1 - beta2) - 1\n                    N_sma = N_sma_max - 2 * state['step'] * beta2_t / (1 - beta2_t)\n                    buffered[1] = N_sma\n\n                    # more conservative since it's an approximated value\n                    if N_sma >= 5:\n                        step_size = math.sqrt((1 - beta2_t) * (N_sma - 4) / (N_sma_max - 4) * (N_sma - 2) / N_sma * N_sma_max / (N_sma_max - 2)) / (1 - beta1 ** state['step'])\n                    else:\n                        step_size = 1.0 / (1 - beta1 ** state['step'])\n                    buffered[2] = step_size\n\n                if group['weight_decay'] != 0:\n                    p_data_fp32.add_(-group['weight_decay'] * group['lr'], p_data_fp32)\n\n                # more conservative since it's an approximated value\n                if N_sma >= 5:            \n                    denom = exp_avg_sq.sqrt().add_(group['eps'])\n                    p_data_fp32.addcdiv_(-step_size * group['lr'], exp_avg, denom)\n                else:\n                    p_data_fp32.add_(-step_size * group['lr'], exp_avg)\n\n                p.data.copy_(p_data_fp32)\n\n        return loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class BCEDiceLoss(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.dice = smp.utils.losses.DiceLoss()\n        self.bce = nn.BCELoss(reduction='mean')\n\n    def forward(self, y_pr, y_gt):\n        dice = self.dice(y_pr, y_gt)\n        bce = self.bce(y_pr, y_gt)\n        return (1.0*dice)+(1.0*bce)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# criterion = BCEDiceLoss(eps=1.0, activation=None)\ncriterion = BCEDiceLoss()\n# optimizer = RAdam(model.parameters(), lr = 0.005)\noptimizer = torch.optim.Adam(model.parameters(), lr=0.005)\ncurrent_lr = [param_group['lr'] for param_group in optimizer.param_groups][0]\nscheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, factor=0.2, patience=2, cooldown=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training loop"},{"metadata":{"trusted":true},"cell_type":"code","source":"# number of epochs to train the model\nn_epochs = 12\ntrain_loss_list = []\nvalid_loss_list = []\ndice_score_list = []\nlr_rate_list = []\nvalid_loss_min = np.Inf # track change in validation loss\nfor epoch in range(1, n_epochs+1):\n\n    # keep track of training and validation loss\n    train_loss = 0.0\n    valid_loss = 0.0\n    dice_score = 0.0\n    ###################\n    # train the model #\n    ###################\n    model.train()\n    bar = tq(train_loader, postfix={\"train_loss\":0.0})\n    for data, target in bar:\n        # move tensors to GPU if CUDA is available\n        if train_on_gpu:\n            data, target = data.cuda(), target.cuda()\n        # clear the gradients of all optimized variables\n        optimizer.zero_grad()\n        # forward pass: compute predicted outputs by passing inputs to the model\n        output = model(data)\n        # calculate the batch loss\n        loss = criterion(torch.sigmoid(output), target)\n        #print(loss)\n        # backward pass: compute gradient of the loss with respect to model parameters\n        loss.backward()\n        # perform a single optimization step (parameter update)\n        optimizer.step()\n        # update training loss\n        train_loss += loss.item()*data.size(0)\n        bar.set_postfix(ordered_dict={\"train_loss\":loss.item()})\n    ######################    \n    # validate the model #\n    ######################\n    model.eval()\n    del data, target\n    with torch.no_grad():\n        bar = tq(valid_loader, postfix={\"valid_loss\":0.0, \"dice_score\":0.0})\n        for data, target in bar:\n            # move tensors to GPU if CUDA is available\n            if train_on_gpu:\n                data, target = data.cuda(), target.cuda()\n            # forward pass: compute predicted outputs by passing inputs to the model\n            output = model(data)\n            # calculate the batch loss\n            loss = criterion(torch.sigmoid(output), target)\n            # update average validation loss \n            valid_loss += loss.item()*data.size(0)\n            dice_cof = dice_no_threshold(torch.sigmoid(output).cpu(), target.cpu()).item()\n            dice_score +=  dice_cof * data.size(0)\n            bar.set_postfix(ordered_dict={\"valid_loss\":loss.item(), \"dice_score\":dice_cof})\n    \n    # calculate average losses\n    train_loss = train_loss/len(train_loader.dataset)\n    valid_loss = valid_loss/len(valid_loader.dataset)\n    dice_score = dice_score/len(valid_loader.dataset)\n    train_loss_list.append(train_loss)\n    valid_loss_list.append(valid_loss)\n    dice_score_list.append(dice_score)\n    lr_rate_list.append([param_group['lr'] for param_group in optimizer.param_groups])\n    \n    # print training/validation statistics \n    print('Epoch: {}  Training Loss: {:.6f}  Validation Loss: {:.6f} Dice Score: {:.6f}'.format(\n        epoch, train_loss, valid_loss, dice_score))\n    \n    # save model if validation loss has decreased\n    if valid_loss <= valid_loss_min:\n        print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(\n        valid_loss_min,\n        valid_loss))\n        torch.save(model.state_dict(), 'model_cifar.pt')\n        valid_loss_min = valid_loss\n    \n    scheduler.step(valid_loss)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Ploting Metrics"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10))\nplt.plot([i[0] for i in lr_rate_list])\nplt.ylabel('learing rate during training', fontsize=22)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10))\nplt.plot(train_loss_list,  marker='o', label=\"Training Loss\")\nplt.plot(valid_loss_list,  marker='o', label=\"Validation Loss\")\nplt.ylabel('loss', fontsize=22)\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10))\nplt.plot(dice_score_list)\nplt.ylabel('Dice score')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# load best model\nmodel.load_state_dict(torch.load('model_cifar.pt'))\nmodel.eval();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_masks = []\ncount = 0\ntr = min(len(valid_ids)*4, 2000)\nprobabilities = np.zeros((tr, 350, 525), dtype = np.float32)\nfor data, target in tq(valid_loader):\n    if train_on_gpu:\n        data = data.cuda()\n    target = target.cpu().detach().numpy()\n    outpu = model(data).cpu().detach().numpy()\n    for p in range(data.shape[0]):\n        output, mask = outpu[p], target[p]\n        for m in mask:\n            valid_masks.append(resize_it(m))\n        for probability in output:\n            probabilities[count, :, :] = resize_it(probability)\n            count += 1\n        if count >= tr - 1:\n            break\n    if count >= tr - 1:\n        break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Grid Search for best Threshold"},{"metadata":{"trusted":true},"cell_type":"code","source":"class_params = {}\nfor class_id in range(4):\n    print(class_id)\n    attempts = []\n    for t in range(0, 100, 5):\n        t /= 100\n        for ms in [0, 100, 1200, 5000, 10000, 30000]:\n            masks, d = [], []\n            for i in range(class_id, len(probabilities), 4):\n                probability = probabilities[i]\n                predict, num_predict = post_process(probability, t, ms)\n                masks.append(predict)\n            for i, j in zip(masks, valid_masks[class_id::4]):\n                if (i.sum() == 0) & (j.sum() == 0):\n                    d.append(1)\n                else:\n                    d.append(dice(i, j))\n            attempts.append((t, ms, np.mean(d)))\n\n    attempts_df = pd.DataFrame(attempts, columns=['threshold', 'size', 'dice'])\n    attempts_df = attempts_df.sort_values('dice', ascending=False)\n    print(attempts_df.head())\n    best_threshold = attempts_df['threshold'].values[0]\n    best_size = attempts_df['size'].values[0]\n    class_params[class_id] = (best_threshold, best_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del masks\ndel valid_masks\ndel probabilities\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"attempts_df = pd.DataFrame(attempts, columns=['threshold', 'size', 'dice'])\nprint(class_params)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"attempts_df.groupby(['threshold'])['dice'].max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"attempts_df.groupby(['size'])['dice'].max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"attempts_df = attempts_df.sort_values('dice', ascending=False)\nattempts_df.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lineplot(x='threshold', y='dice', hue='size', data=attempts_df);\nplt.title('Threshold and min size vs dice');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"best_threshold = attempts_df['threshold'].values[0]\nbest_size = attempts_df['size'].values[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i, (data, target) in enumerate(valid_loader):\n    if train_on_gpu:\n        data = data.cuda()\n    output = ((model(data))[0]).cpu().detach().numpy()\n    image  = data[0].cpu().detach().numpy()\n    mask   = target[0].cpu().detach().numpy()\n    output = output.transpose(1 ,2, 0)\n    image_vis = image.transpose(1, 2, 0)\n    mask = mask.astype('uint8').transpose(1, 2, 0)\n    pr_mask = np.zeros((350, 525, 4))\n    for j in range(4):\n        probability = resize_it(output[:, :, j])\n        pr_mask[:, :, j], _ = post_process(probability,\n                                           class_params[j][0],\n                                           class_params[j][1])\n    visualize_with_raw(image=image_vis, mask=pr_mask,\n                      original_image=image_vis, original_mask=mask,\n                      raw_image=image_vis, raw_mask=output)\n    if i >= 6:\n        break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.cuda.empty_cache()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dataset = CloudDataset(df=sub,\n                            datatype='test', \n                            img_ids=test_ids,\n                            transforms=get_validation_augmentation())\ntest_loader = DataLoader(test_dataset, batch_size=4,\n                         shuffle=False, num_workers=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train_dataset, train_loader\ndel valid_dataset, valid_loader\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"subm = pd.read_csv(\"../input/understanding_cloud_organization/sample_submission.csv\")\npathlist = [\"../input/understanding_cloud_organization/test_images/\" + i.split(\"_\")[0] for i in subm['Image_Label']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_black_mask(image_path):\n    img = cv2.imread(image_path)\n    img = cv2.resize(img, (525,350))\n    hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)\n    lower = np.array([0, 0, 0], np.uint8)\n    upper = np.array([180, 255, 10], np.uint8)\n    return (~ (cv2.inRange(hsv, lower, upper) > 250)).astype(int)\n\nplt.imshow(get_black_mask(pathlist[120]))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"encoded_pixels = []\nimage_id = 0\ncou = 0\nnp_saved = 0\nfor data, target in tq(test_loader):\n    if train_on_gpu:\n        data = data.cuda()\n    output = model(data)\n    del data\n    for i, batch in enumerate(output):\n        for probability in batch:\n            probability = resize_it(probability.cpu().detach().numpy())\n            predict, num_predict = post_process(probability,\n                                                class_params[image_id % 4][0],\n                                                class_params[image_id % 4][1])\n            if num_predict == 0:\n                encoded_pixels.append('')\n            else:\n                black_mask = get_black_mask(pathlist[cou])\n                np_saved += np.sum(predict)\n                predict = np.multiply(predict, black_mask)\n                np_saved -= np.sum(predict)\n                r = mask2rle(predict)\n                encoded_pixels.append(r)\n            cou += 1\n            image_id += 1\n\nprint(f\"number of pixel saved {np_saved}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['EncodedPixels'] = encoded_pixels\nsub.to_csv('submission.csv', columns=['Image_Label', 'EncodedPixels'], index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Thank you for reading this do upvote if you like it"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}