{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# What's the difference?\n\nOffline training Efficientnet in the [@lofass](https://www.kaggle.com/iafoss)'s [kernel](https://www.kaggle.com/iafoss/hubmap-pytorch-fast-ai-starter).","metadata":{}},{"cell_type":"markdown","source":"# Description\nWelcome to Human BioMolecular Atlas Program (HuBMAP) competition. The objective of this challenge is segmentation of regions with glomeruli in human kidney tissue images across different tissue preparation pipelines. Glomeruli are a type of functional tissue units (FTUs): three-dimensional blocks of cells centered around a capillary, such that each cell in this block is within diffusion distance from any other cell in the same block. The provided data consists of 11 fresh frozen and 9 Formalin Fixed Paraffin Embedded (FFPE) PAS kidney images: 8 - train, 5+7 - test. Each of them has ~50k pixel size and is saved as a high-resolution tiff image. To make such large images to be suitable for training of a neural network, they must be [cut into tiles](https://www.kaggle.com/iafoss/256x256-images). Based on the size of the detected features, I'd expect that the appropriate tile size for this data should be 1024x1024. However, it would be an overshoot for a starter code and the initial model development. Therefore, I use tiles of 4 times lower resolution, 256x256, and one could run larger resolution tiles on the finalized setup (in my preliminary tests it gave 0.002-0.003 CV boost).\n\nIn this kernel I provide a fast.ai starter code based on a U-shape network (UneXt50) I used in Severstal and Understanding Clouds competitions and share several tricks from the previous segmentation competitions. I hope the knowledge about these tricks in the beginning of the competition will further motivate the model development, and we will not see something like posting of all tricks in a week before the end of the competition, like in the recent RSNA challenge.","metadata":{}},{"cell_type":"code","source":"%reload_ext autoreload\n%autoreload 2\n%matplotlib inline\n\nfrom fastai.vision.all import *\nfrom torch.utils.data import Dataset, DataLoader\nimport pandas as pd\nimport numpy as np\nimport os\nimport cv2\nimport gc\nimport random\nfrom albumentations import *\nfrom sklearn.model_selection import KFold\nimport matplotlib.pyplot as plt\nfrom lovasz import lovasz_hinge\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-11T10:52:12.245380Z","iopub.execute_input":"2022-07-11T10:52:12.245738Z","iopub.status.idle":"2022-07-11T10:52:12.328547Z","shell.execute_reply.started":"2022-07-11T10:52:12.245706Z","shell.execute_reply":"2022-07-11T10:52:12.327822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bs = 32\nnfolds = 4\nfold = 0\nSEED = 2020\n# TRAIN = '../input/hubmap-256x256/train/'\n# MASKS = '../input/hubmap-256x256/masks/'\n# LABELS = '../input/hubmap-kidney-segmentation/train.csv'\nTRAIN = '../input/hubmap-2022-256x256/train/'\nMASKS = '../input/hubmap-2022-256x256/masks/'\nLABELS = '../input/hubmap-organ-segmentation/train.csv'\nNUM_WORKERS = 4","metadata":{"execution":{"iopub.status.busy":"2022-07-11T10:52:12.330113Z","iopub.execute_input":"2022-07-11T10:52:12.330371Z","iopub.status.idle":"2022-07-11T10:52:12.391296Z","shell.execute_reply.started":"2022-07-11T10:52:12.330346Z","shell.execute_reply":"2022-07-11T10:52:12.390502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    #the following line gives ~10% speedup\n    #but may lead to some stochasticity in the results \n    torch.backends.cudnn.benchmark = True\n    \nseed_everything(SEED)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-11T10:52:12.392926Z","iopub.execute_input":"2022-07-11T10:52:12.393427Z","iopub.status.idle":"2022-07-11T10:52:12.456920Z","shell.execute_reply.started":"2022-07-11T10:52:12.393380Z","shell.execute_reply":"2022-07-11T10:52:12.456024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data\nOne important thing here is the train/val split. To avoid possible leaks resulted by a similarity of tiles from the same images, it is better to keep tiles from each image together in train or in test.","metadata":{}},{"cell_type":"code","source":"# # https://www.kaggle.com/iafoss/256x256-images\n# mean = np.array([0.65459856,0.48386562,0.69428385])\n# std = np.array([0.15167958,0.23584107,0.13146145])\n\n\n# https://www.kaggle.com/datasets/thedevastator/hubmap-2022-256x256\nmean = np.array([0.7720342, 0.74582646, 0.76392896])\nstd = np.array([0.24745085, 0.26182273, 0.25782376])\n\ndef img2tensor(img,dtype:np.dtype=np.float32):\n    if img.ndim==2 : img = np.expand_dims(img,2)\n    img = np.transpose(img,(2,0,1))\n    return torch.from_numpy(img.astype(dtype, copy=False))\n\n# class HuBMAPDataset(Dataset):\n#     def __init__(self, fold=fold, train=True, tfms=None):\n#         ids = pd.read_csv(LABELS).id.values\n#         kf = KFold(n_splits=nfolds,random_state=SEED,shuffle=True)\n#         ids = set(ids[list(kf.split(ids))[fold][0 if train else 1]])\n#         self.fnames = [fname for fname in os.listdir(TRAIN) if fname.split('_')[0] in ids]\n#         self.train = train\n#         self.tfms = tfms\n        \n#     def __len__(self):\n#         return len(self.fnames)\n    \n#     def __getitem__(self, idx):\n#         fname = self.fnames[idx]\n#         img = cv2.cvtColor(cv2.imread(os.path.join(TRAIN,fname)), cv2.COLOR_BGR2RGB)\n#         mask = cv2.imread(os.path.join(MASKS,fname),cv2.IMREAD_GRAYSCALE)\n#         if self.tfms is not None:\n#             augmented = self.tfms(image=img,mask=mask)\n#             img,mask = augmented['image'],augmented['mask']\n#         return img2tensor((img/255.0 - mean)/std),img2tensor(mask)\n\nclass HuBMAPDataset(Dataset):\n    def __init__(self, fold=fold, train=True, tfms=None):\n        ids = pd.read_csv(LABELS).id.astype(str).values\n        kf = KFold(n_splits=nfolds,random_state=SEED,shuffle=True)\n        ids = set(ids[list(kf.split(ids))[fold][0 if train else 1]])\n        self.fnames = [fname for fname in os.listdir(TRAIN) if fname.split('_')[0] in ids]\n        self.train = train\n        self.tfms = tfms\n        \n    def __len__(self):\n        return len(self.fnames)\n    \n    def __getitem__(self, idx):\n        fname = self.fnames[idx]\n        img = cv2.cvtColor(cv2.imread(os.path.join(TRAIN,fname)), cv2.COLOR_BGR2RGB)\n        mask = cv2.imread(os.path.join(MASKS,fname),cv2.IMREAD_GRAYSCALE)\n        if self.tfms is not None:\n            augmented = self.tfms(image=img,mask=mask)\n            img,mask = augmented['image'],augmented['mask']\n        return img2tensor((img/255.0 - mean)/std),img2tensor(mask)\n    \n# def get_aug(p=1.0):\n#     return Compose([\n#         HorizontalFlip(),\n#         VerticalFlip(),\n#         RandomRotate90(),\n#         ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=15, p=0.9, \n#                          border_mode=cv2.BORDER_REFLECT),\n#         OneOf([\n#             OpticalDistortion(p=0.3),\n#             GridDistortion(p=.1),\n#             IAAPiecewiseAffine(p=0.3),\n#         ], p=0.3),\n#         OneOf([\n#             HueSaturationValue(10,15,10),\n#             CLAHE(clip_limit=2),\n#             RandomBrightnessContrast(),            \n#         ], p=0.3),\n#     ], p=p)\n\n# version 1 0.65 version\ndef get_aug(p=1.0):\n    return Compose([\n        HorizontalFlip(p=0.5),\n        VerticalFlip(),\n        RandomRotate90(p=1),\n        #Morphology\n        ShiftScaleRotate(shift_limit=0, scale_limit=(-0.2,0.2), rotate_limit=(-30,30), \n                         interpolation=1, border_mode=0, value=(0,0,0), p=0.5),\n        GaussNoise(var_limit=(0,50.0), mean=0, p=0.5),\n        GaussianBlur(blur_limit=(3,7), p=0.5),\n        #Color\n        RandomBrightnessContrast(brightness_limit=0.35, contrast_limit=0.5, \n                                 brightness_by_max=True,p=0.5),\n        HueSaturationValue(hue_shift_limit=30, sat_shift_limit=30, \n                           val_shift_limit=0, p=0.5),\n        OneOf([\n            OpticalDistortion(p=0.3),\n            GridDistortion(p=.1),\n            IAAPiecewiseAffine(p=0.3),\n        ], p=0.3),\n    ], p=p)","metadata":{"execution":{"iopub.status.busy":"2022-07-11T10:52:12.458498Z","iopub.execute_input":"2022-07-11T10:52:12.459022Z","iopub.status.idle":"2022-07-11T10:52:12.534548Z","shell.execute_reply.started":"2022-07-11T10:52:12.458964Z","shell.execute_reply":"2022-07-11T10:52:12.533656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#example of train images with masks\nds = HuBMAPDataset(tfms=get_aug())\ndl = DataLoader(ds,batch_size=64,shuffle=False,num_workers=NUM_WORKERS)\nimgs,masks = next(iter(dl))\n\nplt.figure(figsize=(16,16))\nfor i,(img,mask) in enumerate(zip(imgs,masks)):\n    img = ((img.permute(1,2,0)*std + mean)*255.0).numpy().astype(np.uint8)\n    plt.subplot(8,8,i+1)\n    plt.imshow(img,vmin=0,vmax=255)\n    plt.imshow(mask.squeeze().numpy(), alpha=0.2)\n    plt.axis('off')\n    plt.subplots_adjust(wspace=None, hspace=None)\n    \ndel ds,dl,imgs,masks","metadata":{"execution":{"iopub.status.busy":"2022-07-11T10:52:12.537675Z","iopub.execute_input":"2022-07-11T10:52:12.538055Z","iopub.status.idle":"2022-07-11T10:52:24.691320Z","shell.execute_reply.started":"2022-07-11T10:52:12.538019Z","shell.execute_reply":"2022-07-11T10:52:24.690016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model\nThe model used in this kernel is based on a U-shape network (UneXt50, see image below), which I used in Severstal and Understanding Clouds competitions. The idea of a U-shape network is coming from a [Unet](https://arxiv.org/pdf/1505.04597.pdf) architecture proposed in 2015 for medical images: the encoder part creates a representation of features at different levels, while the decoder combines the features and generates a prediction as a segmentation mask. The skip connections between encoder and decoder allow us to utilize features from the intermediate conv layers of the encoder effectively, without a need for the information to go the full way through entire encoder and decoder. The latter is especially important to link the predicted mask to the specific pixels of the detected object. Later people realized that ImageNet pretrained computer vision models could drastically improve the quality of a segmentation model because of optimized architecture of the encoder, high encoder capacity (in contrast to one used in the original Unet), and the power of the transfer learning.\n\nThere are several important things that must be added to a Unet network, however, to make it able to reach competitive results with current state of the art approaches. First, it is **Feature Pyramid Network (FPN)**: additional skip connection between different upscaling blocks of the decoder and the output layer. So, the final prediction is produced based on the concatenation of U-net output with resized outputs of the intermediate layers. These skip-connections provide a shortcut for gradient flow improving model performance and convergence speed. Since intermediate layers have many channels, their upscaling and use as an input for the final layer would introduce a significant overhead in terms of the computational time and memory. Therefore, 3x3+3x3 convolutions are applied (factorization) before the resize to reduce the number of channels.\n\nAnother very important thing is the **Atrous Spatial Pyramid Pooling (ASPP) block** added between encoder and decoder. The flaw of the traditional U-shape networks is resulted by a small receptive field. Therefore, if a model needs to make a decision about a segmentation of a large object, especially for a large image resolution, it can get confused being able to look only into parts of the object. A way to increase the receptive field and enable interactions between different parts of the image is use of a block combining convolutions with different dilatations ([Atrous convolutions](https://arxiv.org/pdf/1606.00915.pdf) with various rates in ASPP block). While the original paper uses 6,12,18 rates, they may be customized for a particular task and a particular image resolution to maximize the performance. One more thing I added is using group convolutions in ASPP block to reduce the number of model parameters.\n\nFinally, the decoder upscaling blocks are based on [pixel shuffle](https://arxiv.org/pdf/1609.05158.pdf) rather than transposed convolution used in the first Unet models. It allows to avoid artifacts in the produced masks. And I use [semisupervised Imagenet pretrained ResNeXt50](https://github.com/facebookresearch/semi-supervised-ImageNet1K-models) model as a backbone. In Pytorch it provides the performance of EfficientNet B2-B3 with much faster convergence for the computational cost and GPU RAM requirements of EfficientNet B0 (though, in TF EfficientNet is highly optimized and may be a good thing to use).","metadata":{}},{"cell_type":"markdown","source":"![](https://i.ibb.co/z5KxDzm/Une-Xt50-1.png)","metadata":{}},{"cell_type":"code","source":"class FPN(nn.Module):\n    def __init__(self, input_channels:list, output_channels:list):\n        super().__init__()\n        self.convs = nn.ModuleList(\n            [nn.Sequential(nn.Conv2d(in_ch, out_ch*2, kernel_size=3, padding=1),\n             nn.ReLU(inplace=True), nn.BatchNorm2d(out_ch*2),\n             nn.Conv2d(out_ch*2, out_ch, kernel_size=3, padding=1))\n            for in_ch, out_ch in zip(input_channels, output_channels)])\n        \n    def forward(self, xs:list, last_layer):\n        hcs = [F.interpolate(c(x),scale_factor=2**(len(self.convs)-i),mode='bilinear') \n               for i,(c,x) in enumerate(zip(self.convs, xs))]\n        hcs.append(last_layer)\n        return torch.cat(hcs, dim=1)\n\nclass UnetBlock(Module):\n    def __init__(self, up_in_c:int, x_in_c:int, nf:int=None, blur:bool=False,\n                 self_attention:bool=False, **kwargs):\n        super().__init__()\n        self.shuf = PixelShuffle_ICNR(up_in_c, up_in_c//2, blur=blur, **kwargs)\n        self.bn = nn.BatchNorm2d(x_in_c)\n        ni = up_in_c//2 + x_in_c\n        nf = nf if nf is not None else max(up_in_c//2,32)\n        self.conv1 = ConvLayer(ni, nf, norm_type=None, **kwargs)\n        self.conv2 = ConvLayer(nf, nf, norm_type=None,\n            xtra=SelfAttention(nf) if self_attention else None, **kwargs)\n        self.relu = nn.ReLU(inplace=True)\n\n    def forward(self, up_in:Tensor, left_in:Tensor) -> Tensor:\n        s = left_in\n        up_out = self.shuf(up_in)\n        cat_x = self.relu(torch.cat([up_out, self.bn(s)], dim=1))\n        return self.conv2(self.conv1(cat_x))\n        \nclass _ASPPModule(nn.Module):\n    def __init__(self, inplanes, planes, kernel_size, padding, dilation, groups=1):\n        super().__init__()\n        self.atrous_conv = nn.Conv2d(inplanes, planes, kernel_size=kernel_size,\n                stride=1, padding=padding, dilation=dilation, bias=False, groups=groups)\n        self.bn = nn.BatchNorm2d(planes)\n        self.relu = nn.ReLU()\n\n        self._init_weight()\n\n    def forward(self, x):\n        x = self.atrous_conv(x)\n        x = self.bn(x)\n\n        return self.relu(x)\n\n    def _init_weight(self):\n        for m in self.modules():\n            if isinstance(m, nn.Conv2d):\n                torch.nn.init.kaiming_normal_(m.weight)\n            elif isinstance(m, nn.BatchNorm2d):\n                m.weight.data.fill_(1)\n                m.bias.data.zero_()\n\nclass ASPP(nn.Module):\n    def __init__(self, inplanes=512, mid_c=256, dilations=[6, 12, 18, 24], out_c=None):\n        super().__init__()\n        self.aspps = [_ASPPModule(inplanes, mid_c, 1, padding=0, dilation=1)] + \\\n            [_ASPPModule(inplanes, mid_c, 3, padding=d, dilation=d,groups=4) for d in dilations]\n        self.aspps = nn.ModuleList(self.aspps)\n        self.global_pool = nn.Sequential(nn.AdaptiveMaxPool2d((1, 1)),\n                        nn.Conv2d(inplanes, mid_c, 1, stride=1, bias=False),\n                        nn.BatchNorm2d(mid_c), nn.ReLU())\n        out_c = out_c if out_c is not None else mid_c\n        self.out_conv = nn.Sequential(nn.Conv2d(mid_c*(2+len(dilations)), out_c, 1, bias=False),\n                                    nn.BatchNorm2d(out_c), nn.ReLU(inplace=True))\n        self.conv1 = nn.Conv2d(mid_c*(2+len(dilations)), out_c, 1, bias=False)\n        self._init_weight()\n\n    def forward(self, x):\n        x0 = self.global_pool(x)\n        xs = [aspp(x) for aspp in self.aspps]\n        x0 = F.interpolate(x0, size=xs[0].size()[2:], mode='bilinear', align_corners=True)\n        x = torch.cat([x0] + xs, dim=1)\n        return self.out_conv(x)\n    \n    def _init_weight(self):\n        for m in self.modules():\n            if isinstance(m, nn.Conv2d):\n                torch.nn.init.kaiming_normal_(m.weight)\n            elif isinstance(m, nn.BatchNorm2d):\n                m.weight.data.fill_(1)\n                m.bias.data.zero_()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-11T10:52:24.692678Z","iopub.execute_input":"2022-07-11T10:52:24.693008Z","iopub.status.idle":"2022-07-11T10:52:24.799253Z","shell.execute_reply.started":"2022-07-11T10:52:24.692974Z","shell.execute_reply":"2022-07-11T10:52:24.798138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pretrained_root = '../input/efficientnet-pytorch/'\nefficient_net_encoders = {\n    \"efficientnet-b0\": {\n        \"out_channels\": (3, 32, 24, 40, 112, 320),\n        \"stage_idxs\": (3, 5, 9, 16),\n        \"weight_path\": pretrained_root + \"efficientnet-b0-08094119.pth\"\n    },\n    \"efficientnet-b1\": {\n        \"out_channels\": (3, 32, 24, 40, 112, 320),\n        \"stage_idxs\": (5, 8, 16, 23),\n        \"weight_path\": pretrained_root + \"efficientnet-b1-dbc7070a.pth\"\n    },\n    \"efficientnet-b2\": {\n        \"out_channels\": (3, 32, 24, 48, 120, 352),\n        \"stage_idxs\": (5, 8, 16, 23),\n        \"weight_path\": pretrained_root + \"efficientnet-b2-27687264.pth\"\n    },\n    \"efficientnet-b3\": {\n        \"out_channels\": (3, 40, 32, 48, 136, 384),\n        \"stage_idxs\": (5, 8, 18, 26),\n        \"weight_path\": pretrained_root + \"efficientnet-b3-c8376fa2.pth\"\n    },\n    \"efficientnet-b4\": {\n        \"out_channels\": (3, 48, 32, 56, 160, 448),\n        \"stage_idxs\": (6, 10, 22, 32),\n        \"weight_path\": pretrained_root + \"efficientnet-b4-e116e8b3.pth\"\n    },\n    \"efficientnet-b5\": {\n        \"out_channels\": (3, 48, 40, 64, 176, 512),\n        \"stage_idxs\": (8, 13, 27, 39),\n        \"weight_path\": pretrained_root + \"efficientnet-b5-586e6cc6.pth\"\n    },\n    \"efficientnet-b6\": {\n        \"out_channels\": (3, 56, 40, 72, 200, 576),\n        \"stage_idxs\": (9, 15, 31, 45),\n        \"weight_path\": pretrained_root + \"efficientnet-b6-c76e70fd.pth\"\n    },\n    \"efficientnet-b7\": {\n        \"out_channels\": (3, 64, 48, 80, 224, 640),\n        \"stage_idxs\": (11, 18, 38, 55),\n        \"weight_path\": pretrained_root + \"efficientnet-b7-dcc49843.pth\"\n    }\n}","metadata":{"execution":{"iopub.status.busy":"2022-07-11T10:52:24.800728Z","iopub.execute_input":"2022-07-11T10:52:24.801192Z","iopub.status.idle":"2022-07-11T10:52:24.876118Z","shell.execute_reply.started":"2022-07-11T10:52:24.801131Z","shell.execute_reply":"2022-07-11T10:52:24.874880Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nsys.path.insert(0, '../input/efficientnet-pytorch/EfficientNet-PyTorch/EfficientNet-PyTorch-master')\n\nfrom efficientnet_pytorch import EfficientNet\nfrom efficientnet_pytorch.utils import url_map, url_map_advprop, get_model_params\n\n\nclass EfficientNetEncoder(EfficientNet):\n    def __init__(self, stage_idxs, out_channels, model_name, depth=5):\n\n        blocks_args, global_params = get_model_params(model_name, override_params=None)\n        super().__init__(blocks_args, global_params)\n        \n        cfg = efficient_net_encoders[model_name]\n\n        self._stage_idxs = stage_idxs\n        self._out_channels = out_channels\n        self._depth = depth\n        self._in_channels = 3\n\n        del self._fc\n        self.load_state_dict(torch.load(cfg['weight_path']))\n\n    def get_stages(self):\n        return [\n            nn.Identity(),\n            nn.Sequential(self._conv_stem, self._bn0, self._swish),\n            self._blocks[:self._stage_idxs[0]],\n            self._blocks[self._stage_idxs[0]:self._stage_idxs[1]],\n            self._blocks[self._stage_idxs[1]:self._stage_idxs[2]],\n            self._blocks[self._stage_idxs[2]:],\n        ]\n\n    def forward(self, x):\n        stages = self.get_stages()\n\n        block_number = 0.\n        drop_connect_rate = self._global_params.drop_connect_rate\n\n        features = []\n        for i in range(self._depth + 1):\n\n            # Identity and Sequential stages\n            if i < 2:\n                x = stages[i](x)\n\n            # Block stages need drop_connect rate\n            else:\n                for module in stages[i]:\n                    drop_connect = drop_connect_rate * block_number / len(self._blocks)\n                    block_number += 1.\n                    x = module(x, drop_connect)\n\n            features.append(x)\n\n        return features\n\n    def load_state_dict(self, state_dict, **kwargs):\n        state_dict.pop(\"_fc.bias\")\n        state_dict.pop(\"_fc.weight\")\n        super().load_state_dict(state_dict, **kwargs)  \n        \n\nclass EffUnet(nn.Module):\n    def __init__(self, model_name, stride=1):\n        super().__init__()\n        \n        cfg = efficient_net_encoders[model_name]\n        stage_idxs = cfg['stage_idxs']\n        out_channels = cfg['out_channels']\n        \n        self.encoder = EfficientNetEncoder(stage_idxs, out_channels, model_name)\n\n        #aspp with customized dilatations\n        self.aspp = ASPP(out_channels[-1], 256, out_c=384, \n                         dilations=[stride*1, stride*2, stride*3, stride*4])\n        self.drop_aspp = nn.Dropout2d(0.5)\n        #decoder\n        self.dec4 = UnetBlock(384, out_channels[-2], 256)\n        self.dec3 = UnetBlock(256, out_channels[-3], 128)\n        self.dec2 = UnetBlock(128, out_channels[-4], 64)\n        self.dec1 = UnetBlock(64, out_channels[-5], 32)\n        self.fpn = FPN([384, 256, 128, 64], [16]*4)\n        self.drop = nn.Dropout2d(0.1)\n        self.final_conv = ConvLayer(32+16*4, 1, ks=1, norm_type=None, act_cls=None)\n        \n    def forward(self, x):\n        enc0, enc1, enc2, enc3, enc4 = self.encoder(x)[-5:]\n        enc5 = self.aspp(enc4)\n        dec3 = self.dec4(self.drop_aspp(enc5), enc3)\n        dec2 = self.dec3(dec3,enc2)\n        dec1 = self.dec2(dec2,enc1)\n        dec0 = self.dec1(dec1,enc0)\n        x = self.fpn([enc5, dec3, dec2, dec1], dec0)\n        x = self.final_conv(self.drop(x))\n        x = F.interpolate(x,scale_factor=2,mode='bilinear')\n        return x\n    \n#split the model to encoder and decoder for fast.ai\nsplit_layers = lambda m: [\n                list(m.encoder.parameters()),\n                list(m.aspp.parameters())+list(m.dec4.parameters())+\n                list(m.dec3.parameters())+list(m.dec2.parameters())+\n                list(m.dec1.parameters())+list(m.fpn.parameters())+\n                list(m.final_conv.parameters())\n            ]","metadata":{"execution":{"iopub.status.busy":"2022-07-11T10:52:24.877287Z","iopub.execute_input":"2022-07-11T10:52:24.877554Z","iopub.status.idle":"2022-07-11T10:52:25.016852Z","shell.execute_reply.started":"2022-07-11T10:52:24.877521Z","shell.execute_reply":"2022-07-11T10:52:25.016112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loss and metric\nThe loss that works the best for image segmentation in most of the cases is [Lovász loss](https://arxiv.org/pdf/1705.08790.pdf), a differentiable surrogate of IoU. However, ReLU in it must be replaced by (ELU + 1), like I did [here](https://www.kaggle.com/iafoss/lovasz). Another trick is consideration of a symmetric Lovász loss: consider not only a predicted segmentation and a provided mask but also the inverse prediction and the inverse mask (predict mask for negative case).","metadata":{}},{"cell_type":"code","source":"def symmetric_lovasz(outputs, targets):\n    return 0.5*(lovasz_hinge(outputs, targets) + lovasz_hinge(-outputs, 1.0 - targets))","metadata":{"execution":{"iopub.status.busy":"2022-07-11T10:52:25.018171Z","iopub.execute_input":"2022-07-11T10:52:25.018532Z","iopub.status.idle":"2022-07-11T10:52:25.082012Z","shell.execute_reply.started":"2022-07-11T10:52:25.018496Z","shell.execute_reply":"2022-07-11T10:52:25.080994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Dice_soft(Metric):\n    def __init__(self, axis=1): \n        self.axis = axis \n    def reset(self): self.inter,self.union = 0,0\n    def accumulate(self, learn):\n        pred,targ = flatten_check(torch.sigmoid(learn.pred), learn.y)\n        self.inter += (pred*targ).float().sum().item()\n        self.union += (pred+targ).float().sum().item()\n    @property\n    def value(self): return 2.0 * self.inter/self.union if self.union > 0 else None\n    \n# dice with automatic threshold selection\nclass Dice_th(Metric):\n    def __init__(self, ths=np.arange(0.1,0.9,0.05), axis=1): \n        self.axis = axis\n        self.ths = ths\n        \n    def reset(self): \n        self.inter = torch.zeros(len(self.ths))\n        self.union = torch.zeros(len(self.ths))\n        \n    def accumulate(self, learn):\n        pred,targ = flatten_check(torch.sigmoid(learn.pred), learn.y)\n        for i,th in enumerate(self.ths):\n            p = (pred > th).float()\n            self.inter[i] += (p*targ).float().sum().item()\n            self.union[i] += (p+targ).float().sum().item()\n\n    @property\n    def value(self):\n        dices = torch.where(self.union > 0.0, \n                2.0*self.inter/self.union, torch.zeros_like(self.union))\n        return dices.max()","metadata":{"execution":{"iopub.status.busy":"2022-07-11T10:52:25.083826Z","iopub.execute_input":"2022-07-11T10:52:25.084498Z","iopub.status.idle":"2022-07-11T10:52:25.153646Z","shell.execute_reply.started":"2022-07-11T10:52:25.084459Z","shell.execute_reply":"2022-07-11T10:52:25.152797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model evaluation","metadata":{}},{"cell_type":"code","source":"#iterator like wrapper that returns predicted and gt masks\nclass Model_pred:\n    def __init__(self, model, dl, tta:bool=True, half:bool=False):\n        self.model = model\n        self.dl = dl\n        self.tta = tta\n        self.half = half\n        \n    def __iter__(self):\n        self.model.eval()\n        name_list = self.dl.dataset.fnames\n        count=0\n        with torch.no_grad():\n            for x,y in iter(self.dl):\n                x = x.cuda()\n                if self.half: x = x.half()\n                p = self.model(x)\n                py = torch.sigmoid(p).detach()\n                if self.tta:\n                    #x,y,xy flips as TTA\n                    flips = [[-1],[-2],[-2,-1]]\n                    for f in flips:\n                        p = self.model(torch.flip(x,f))\n                        p = torch.flip(p,f)\n                        py += torch.sigmoid(p).detach()\n                    py /= (1+len(flips))\n                if y is not None and len(y.shape)==4 and py.shape != y.shape:\n                    py = F.upsample(py, size=(y.shape[-2],y.shape[-1]), mode=\"bilinear\")\n                py = py.permute(0,2,3,1).float().cpu()\n                batch_size = len(py)\n                for i in range(batch_size):\n                    taget = y[i].detach().cpu() if y is not None else None\n                    yield py[i],taget,name_list[count]\n                    count += 1\n                    \n    def __len__(self):\n        return len(self.dl.dataset)\n    \nclass Dice_th_pred(Metric):\n    def __init__(self, ths=np.arange(0.1,0.9,0.01), axis=1): \n        self.axis = axis\n        self.ths = ths\n        self.reset()\n        \n    def reset(self): \n        self.inter = torch.zeros(len(self.ths))\n        self.union = torch.zeros(len(self.ths))\n        \n    def accumulate(self,p,t):\n        pred,targ = flatten_check(p, t)\n        for i,th in enumerate(self.ths):\n            p = (pred > th).float()\n            self.inter[i] += (p*targ).float().sum().item()\n            self.union[i] += (p+targ).float().sum().item()\n\n    @property\n    def value(self):\n        dices = torch.where(self.union > 0.0, 2.0*self.inter/self.union, \n                            torch.zeros_like(self.union))\n        return dices\n    \ndef save_img(data,name,out):\n    data = data.float().cpu().numpy()\n    img = cv2.imencode('.png',(data*255).astype(np.uint8))[1]\n    out.writestr(name, img)","metadata":{"execution":{"iopub.status.busy":"2022-07-11T10:52:25.155336Z","iopub.execute_input":"2022-07-11T10:52:25.155776Z","iopub.status.idle":"2022-07-11T10:52:25.237351Z","shell.execute_reply.started":"2022-07-11T10:52:25.155738Z","shell.execute_reply":"2022-07-11T10:52:25.236467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train","metadata":{}},{"cell_type":"code","source":"dice = Dice_th_pred(np.arange(0.2,0.7,0.01))\nfor fold in range(nfolds):\n    ds_t = HuBMAPDataset(fold=fold, train=True, tfms=get_aug())\n    ds_v = HuBMAPDataset(fold=fold, train=False)\n    data = ImageDataLoaders.from_dsets(ds_t,ds_v,bs=bs,\n                num_workers=NUM_WORKERS,pin_memory=True).cuda()\n    model = EffUnet('efficientnet-b5').cuda()\n    learn = Learner(data, model, loss_func=symmetric_lovasz,\n                metrics=[Dice_soft(),Dice_th()], \n                splitter=split_layers).to_fp16(clip=0.5)\n    \n    #start with training the head\n    learn.freeze_to(-1) #doesn't work\n    for param in learn.opt.param_groups[0]['params']:\n        param.requires_grad = False\n    learn.fit_one_cycle(6, lr_max=0.5e-2)\n\n    #continue training full model\n    learn.unfreeze()\n    learn.fit_one_cycle(32, lr_max=slice(2e-4,2e-3),\n        cbs=SaveModelCallback(monitor='dice_th',comp=np.greater))\n    torch.save(learn.model.state_dict(),f'model_{fold}.pth')\n    \n    #model evaluation on val and saving the masks\n    mp = Model_pred(learn.model,learn.dls.loaders[1])\n    with zipfile.ZipFile('val_masks_tta.zip', 'a') as out:\n        for p in progress_bar(mp):\n            dice.accumulate(p[0],p[1])\n            save_img(p[0],p[2],out)\n    gc.collect()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-11T10:52:25.238642Z","iopub.execute_input":"2022-07-11T10:52:25.239062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dices = dice.value\nnoise_ths = dice.ths\nbest_dice = dices.max()\nbest_thr = noise_ths[dices.argmax()]\nplt.figure(figsize=(8,4))\nplt.plot(noise_ths, dices, color='blue')\nplt.vlines(x=best_thr, ymin=dices.min(), ymax=dices.max(), colors='black')\nd = dices.max() - dices.min()\nplt.text(noise_ths[-1]-0.1, best_dice-0.1*d, f'DICE = {best_dice:.3f}', fontsize=12);\nplt.text(noise_ths[-1]-0.1, best_dice-0.2*d, f'TH = {best_thr:.3f}', fontsize=12);\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"execution_count":null,"outputs":[]}]}