{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":9988,"databundleVersionId":868324,"sourceType":"competition"},{"sourceId":5328625,"sourceType":"kernelVersion"}],"dockerImageVersionId":438,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Overview","metadata":{"_uuid":"1b8c18e1d49e580b84b194084a08d59e2a497b54"}},{"cell_type":"markdown","source":"There are two most obvious network architectures to approach this competition: U-net and SSD. Each of them has pros and cons. In particular, U-net provides a relatively simple way to solve the competition challenge using image segmentation. However, this competition requires prediction of an individual mask for each ship rather than one mask for entire image. Therefore, some creative postprocessing may be needed, especially to separate ships with overlapping masks, if it is even possible. Another drawback is that the data is labeled with using pixelized bounding boxes rather than real ship masks, therefore the score of U-net based models is lowered. Meanwhile, implementation of SSD requires usage of rotating bounding boxes (https://arxiv.org/pdf/1711.09405.pdf) that is not common and, therefore, would take additional efforts for development of the model and the corresponding loss function. In addition, bounding boxes are not provided in this competition and must be generated based on the pixel masks. Nevertheless, this approach is expected to provide higher score than U-net, especially since the data is labeled based on pixelized bounding boxes (I expect organizers used SSD with rotating bounding boxes to label train and test data).\nSince the first approach is more straightforward, I'll begin with presenting a kernel about U-net. In this post I will describe how to use pretrained ResNet34 to build a high accuracy image segmentation model. In particular, after training only a decoder for 1 epoch (15 min) on 256x256 rescaled images, the dice coefficient reaches ~0.8 (IoU ~0.67) that significantly outperforms all publicly available models posted so far in this competition. After training the entire model for 6 additional epochs with learning rate annealing, the dice coefficient reaches ~0.86 (IoU ~0.75). Due to the kernel run time limit, the model is further trained only for two epochs on 384x384 (dice ~0.87) followed by one epoch on 768x768 images. In an independent run I trained a model on 384x384 images for 12 epochs that boosted dice to 0.89 followed by training on full resolution images that increased dice further to 0.905.","metadata":{"_uuid":"3f0a329467823b3dc6d5f86b79f55500cc44acc8"}},{"cell_type":"code","source":"from fastai.conv_learner import *\nfrom fastai.dataset import *\n\nimport pandas as pd\nimport numpy as np\nimport os\nfrom PIL import Image\nfrom sklearn.model_selection import train_test_split","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2024-11-08T04:00:37.562607Z","iopub.execute_input":"2024-11-08T04:00:37.562892Z","iopub.status.idle":"2024-11-08T04:00:37.569594Z","shell.execute_reply.started":"2024-11-08T04:00:37.562851Z","shell.execute_reply":"2024-11-08T04:00:37.568834Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Data","metadata":{"_uuid":"ddd71a95c05902b29ce4e3768a8127c2a0b7098d"}},{"cell_type":"code","source":"PATH = './'\nTRAIN = '/kaggle/input/airbus-ship-detection/train_v2'\nTEST = '/kaggle/input/airbus-ship-detection/test_v2'\nSEGMENTATION = '/kaggle/input/airbus-ship-detection/train_ship_segmentations_v2.csv'\nPRETRAINED = '/kaggle/input/fine-tuning-resnet34-on-ship-detection/models/Resnet34_lable_256_1.h5'\nexclude_list = ['6384c3e78.jpg','13703f040.jpg', '14715c06d.jpg',  '33e0ff2d5.jpg',\n                '4d4e09f2a.jpg', '877691df8.jpg', '8b909bb20.jpg', 'a8d99130e.jpg', \n                'ad55c3143.jpg', 'c8260c541.jpg', 'd6c7f17c7.jpg', 'dc3e7c901.jpg',\n                'e44dffe88.jpg', 'ef87bad36.jpg', 'f083256d8.jpg'] #corrupted images","metadata":{"trusted":true,"_uuid":"b93d5422dfd31f43df9cdd3e301547cc95f25980","execution":{"iopub.status.busy":"2024-11-08T04:08:12.281372Z","iopub.execute_input":"2024-11-08T04:08:12.281654Z","iopub.status.idle":"2024-11-08T04:08:12.288130Z","shell.execute_reply.started":"2024-11-08T04:08:12.281603Z","shell.execute_reply":"2024-11-08T04:08:12.287408Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"nw = 2   #number of workers for data loader\narch = resnet34 #specify target architecture","metadata":{"trusted":true,"_uuid":"9a72e076dc04cf1786edae26a1d2f15ec3de234a","execution":{"iopub.status.busy":"2024-11-08T04:08:13.180680Z","iopub.execute_input":"2024-11-08T04:08:13.180955Z","iopub.status.idle":"2024-11-08T04:08:13.184709Z","shell.execute_reply.started":"2024-11-08T04:08:13.180915Z","shell.execute_reply":"2024-11-08T04:08:13.184005Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_names = [f for f in os.listdir(TRAIN)]\ntest_names = [f for f in os.listdir(TEST)]\nfor el in exclude_list:\n    if(el in train_names): train_names.remove(el)\n    if(el in test_names): test_names.remove(el)\n#5% of data in the validation set is sufficient for model evaluation\ntr_n, val_n = train_test_split(train_names, test_size=0.05, random_state=42)\nsegmentation_df = pd.read_csv(os.path.join(PATH, SEGMENTATION)).set_index('ImageId')","metadata":{"trusted":true,"_uuid":"49199ea17e9e9ba9893c10d06c1fb419e22aeb1b","execution":{"iopub.status.busy":"2024-11-08T04:08:14.143738Z","iopub.execute_input":"2024-11-08T04:08:14.144005Z","iopub.status.idle":"2024-11-08T04:08:15.615934Z","shell.execute_reply.started":"2024-11-08T04:08:14.143964Z","shell.execute_reply":"2024-11-08T04:08:15.615237Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"One of the challenges of this competition is strong data unbalance. Even if only images with ships are considered, the ratio of mask pixels to the total number of pixels is ~1:1000. If images with no ships are included, this ratio goes to ~1:10000, which is quite tough to handle. Therefore, I drop all images without ships, that makes the training set more balanced and also reduces the time per each epoch almost by 4 times. In an independent run, when the dice of my model reached 0.895, I ran it on images without ships and identified ~3600 false positive predictions out ~70k images. The incorrectly predicted images were incorporated to the training set as negative examples, and training was continued. The problem of false positive predictions can be further mitigated by stacking U-net model with a classification model predicting if ships are present in a particular image (https://www.kaggle.com/iafoss/fine-tuning-resnet34-on-ship-detection - ~98% accuracy). \nI also noticed that in some kernels the dataset is tried to be balanced by keeping approximately the same number of images with 0, 1, 2, etc. ships. However, this strategy would be effective for such task as ship counting rather than training U-net or SSD.  One possible way to balance the dataset is creative cropping the images that keeps approximately the same number of pixels corresponding to a ship or something else. However, I doubt that such approach will effective in this competition. Therefore, a special loss function must be used to mitigate the data unbalance.","metadata":{"_uuid":"b95102ddd563191f9b15eb115f736e93c837d080"}},{"cell_type":"code","source":"def cut_empty(names):\n    return [name for name in names \n            if(type(segmentation_df.loc[name]['EncodedPixels']) != float)]\n\ntr_n = cut_empty(tr_n)\nval_n = cut_empty(val_n)","metadata":{"trusted":true,"_uuid":"6a3d8c70e03964738322ca99084f088adc1c5f3e","execution":{"iopub.status.busy":"2024-11-08T04:09:11.736229Z","iopub.execute_input":"2024-11-08T04:09:11.736540Z","iopub.status.idle":"2024-11-08T04:09:38.370235Z","shell.execute_reply.started":"2024-11-08T04:09:11.736496Z","shell.execute_reply":"2024-11-08T04:09:38.369560Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def get_mask(img_id, df):\n    shape = (768,768)\n    img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n    masks = df.loc[img_id]['EncodedPixels']\n    if(type(masks) == float): return img.reshape(shape)\n    if(type(masks) == str): masks = [masks]\n    for mask in masks:\n        s = mask.split()\n        for i in range(len(s)//2):\n            start = int(s[2*i]) - 1\n            length = int(s[2*i+1])\n            img[start:start+length] = 1\n    return img.reshape(shape).T","metadata":{"trusted":true,"_uuid":"505f803555b8d9d0378a73d227da4b8174f2086d","execution":{"iopub.status.busy":"2024-11-08T04:09:43.874692Z","iopub.execute_input":"2024-11-08T04:09:43.874957Z","iopub.status.idle":"2024-11-08T04:09:43.893690Z","shell.execute_reply.started":"2024-11-08T04:09:43.874917Z","shell.execute_reply":"2024-11-08T04:09:43.892845Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class pdFilesDataset(FilesDataset):\n    def __init__(self, fnames, path, transform):\n        self.segmentation_df = pd.read_csv(SEGMENTATION).set_index('ImageId')\n        super().__init__(fnames, transform, path)\n    \n    def get_x(self, i):\n        img = open_image(os.path.join(self.path, self.fnames[i]))\n        if self.sz == 768: return img \n        else: return cv2.resize(img, (self.sz, self.sz))\n    \n    def get_y(self, i):\n        mask = np.zeros((768,768), dtype=np.uint8) if (self.path == TEST) \\\n            else get_mask(self.fnames[i], self.segmentation_df)\n        img = Image.fromarray(mask).resize((self.sz, self.sz)).convert('RGB')\n        return np.array(img).astype(np.float32)\n    \n    def get_c(self): return 0","metadata":{"trusted":true,"_uuid":"f67a326e1269bffca392fcf4ac10bf4a532ca56b","execution":{"iopub.status.busy":"2024-11-08T04:09:47.719204Z","iopub.execute_input":"2024-11-08T04:09:47.719480Z","iopub.status.idle":"2024-11-08T04:09:47.747420Z","shell.execute_reply.started":"2024-11-08T04:09:47.719439Z","shell.execute_reply":"2024-11-08T04:09:47.746653Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The carrently availible on kaggle version of fastai has a bug in RandomLighting data agmentation class. It would be nice if kaggle updated fastai version to the last one, where this and other bugs are fixed.","metadata":{"_uuid":"f51da2928183042aa99adfe9d1da494f6c624e57"}},{"cell_type":"code","source":"class RandomLighting(Transform):\n    def __init__(self, b, c, tfm_y=TfmType.NO):\n        super().__init__(tfm_y)\n        self.b,self.c = b,c\n\n    def set_state(self):\n        self.store.b_rand = rand0(self.b)\n        self.store.c_rand = rand0(self.c)\n\n    def do_transform(self, x, is_y):\n        if is_y and self.tfm_y != TfmType.PIXEL: return x  #add this line to fix the bug\n        b = self.store.b_rand\n        c = self.store.c_rand\n        c = -1/(c-1) if c<0 else c+1\n        x = lighting(x, b, c)\n        return x","metadata":{"trusted":true,"_uuid":"649601eb0bd110b02f15349cc9d26d60ddeee953","execution":{"iopub.status.busy":"2024-11-08T04:09:51.324921Z","iopub.execute_input":"2024-11-08T04:09:51.325259Z","iopub.status.idle":"2024-11-08T04:09:51.346658Z","shell.execute_reply.started":"2024-11-08T04:09:51.325189Z","shell.execute_reply":"2024-11-08T04:09:51.345983Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def get_data(sz,bs):\n    #data augmentation\n    aug_tfms = [RandomRotate(20, tfm_y=TfmType.CLASS),\n                RandomDihedral(tfm_y=TfmType.CLASS),\n                RandomLighting(0.05, 0.05, tfm_y=TfmType.CLASS)]\n    tfms = tfms_from_model(arch, sz, crop_type=CropType.NO, tfm_y=TfmType.CLASS, \n                aug_tfms=aug_tfms)\n    tr_names = tr_n if (len(tr_n)%bs == 0) else tr_n[:-(len(tr_n)%bs)] #cut incomplete batch\n    ds = ImageData.get_ds(pdFilesDataset, (tr_names,TRAIN), \n                (val_n,TRAIN), tfms, test=(test_names,TEST))\n    md = ImageData(PATH, ds, bs, num_workers=nw, classes=None)\n    md.is_multi = False\n    return md","metadata":{"trusted":true,"_uuid":"835b75b3508f418bac53ee6725694af208e1f28c","execution":{"iopub.status.busy":"2024-11-08T04:09:59.891446Z","iopub.execute_input":"2024-11-08T04:09:59.891772Z","iopub.status.idle":"2024-11-08T04:09:59.907909Z","shell.execute_reply.started":"2024-11-08T04:09:59.891712Z","shell.execute_reply":"2024-11-08T04:09:59.907253Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Model","metadata":{"trusted":true,"collapsed":true,"_uuid":"2b77a79e1e1e3f4c8a9e7120feb9a843b41492f8","jupyter":{"outputs_hidden":true}}},{"cell_type":"markdown","source":"The model used in this kernel is inspired by a Carvana example from FastAI course (http://course.fast.ai/index.html). It is composed of a ResNet34 based encoder and a simple upsampling decoder. Similar to the original U-net, skip connections are added between encoder and decoder to facilitate the information flow at different detalization levels. Meanwhile, using a pretrained ResNet34 model allows us to have a powerful encoder capable of handling elaborated feature, in comparison with the original U-net, without a risk of overfitting and necessity of training a big model from scratch. The total capacity of the model is ~21M parameters. Before using, the original ResNet34 model was further fine-tuned on ship/no-ship classification task (https://www.kaggle.com/iafoss/fine-tuning-resnet34-on-ship-detection).","metadata":{"_uuid":"cc66537bd5382d9e874cfa8c0616965e8d9f5f7d"}},{"cell_type":"code","source":"cut,lr_cut = model_meta[arch]","metadata":{"trusted":true,"_uuid":"5bfefb757e1d40da7d8761c824170214a4bec08b","execution":{"iopub.status.busy":"2024-11-08T04:10:04.622651Z","iopub.execute_input":"2024-11-08T04:10:04.622918Z","iopub.status.idle":"2024-11-08T04:10:04.626597Z","shell.execute_reply.started":"2024-11-08T04:10:04.622878Z","shell.execute_reply":"2024-11-08T04:10:04.625785Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def get_base():                   #load ResNet34 model\n    layers = cut_model(arch(True), cut)\n    return nn.Sequential(*layers)\n\ndef load_pretrained(model, path): #load a model pretrained on ship/no-ship classification\n    weights = torch.load(PRETRAINED, map_location=lambda storage, loc: storage)\n    model.load_state_dict(weights, strict=False)\n            \n    return model","metadata":{"trusted":true,"_uuid":"7b0649b17721919f475a99f41f47f09c4fcc41d6","execution":{"iopub.status.busy":"2024-11-08T04:10:05.926875Z","iopub.execute_input":"2024-11-08T04:10:05.927133Z","iopub.status.idle":"2024-11-08T04:10:05.936700Z","shell.execute_reply.started":"2024-11-08T04:10:05.927095Z","shell.execute_reply":"2024-11-08T04:10:05.935992Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class UnetBlock(nn.Module):\n    def __init__(self, up_in, x_in, n_out):\n        super().__init__()\n        up_out = x_out = n_out//2\n        self.x_conv  = nn.Conv2d(x_in,  x_out,  1)\n        self.tr_conv = nn.ConvTranspose2d(up_in, up_out, 2, stride=2)\n        self.bn = nn.BatchNorm2d(n_out)\n        \n    def forward(self, up_p, x_p):\n        up_p = self.tr_conv(up_p)\n        x_p = self.x_conv(x_p)\n        cat_p = torch.cat([up_p,x_p], dim=1)\n        return self.bn(F.relu(cat_p))\n\nclass SaveFeatures():\n    features=None\n    def __init__(self, m): self.hook = m.register_forward_hook(self.hook_fn)\n    def hook_fn(self, module, input, output): self.features = output\n    def remove(self): self.hook.remove()\n    \nclass Unet34(nn.Module):\n    def __init__(self, rn):\n        super().__init__()\n        self.rn = rn\n        self.sfs = [SaveFeatures(rn[i]) for i in [2,4,5,6]]\n        self.up1 = UnetBlock(512,256,256)\n        self.up2 = UnetBlock(256,128,256)\n        self.up3 = UnetBlock(256,64,256)\n        self.up4 = UnetBlock(256,64,256)\n        self.up5 = nn.ConvTranspose2d(256, 1, 2, stride=2)\n        \n    def forward(self,x):\n        x = F.relu(self.rn(x))\n        x = self.up1(x, self.sfs[3].features)\n        x = self.up2(x, self.sfs[2].features)\n        x = self.up3(x, self.sfs[1].features)\n        x = self.up4(x, self.sfs[0].features)\n        x = self.up5(x)\n        return x[:,0]\n    \n    def close(self):\n        for sf in self.sfs: sf.remove()\n            \nclass UnetModel():\n    def __init__(self,model,name='Unet'):\n        self.model,self.name = model,name\n\n    def get_layer_groups(self, precompute):\n        lgs = list(split_by_idxs(children(self.model.rn), [lr_cut]))\n        return lgs + [children(self.model)[1:]]","metadata":{"trusted":true,"_uuid":"77ca73006aec0c9dfe4be6deb2d6cf524840cd62","execution":{"iopub.status.busy":"2024-11-08T04:10:08.933545Z","iopub.execute_input":"2024-11-08T04:10:08.933867Z","iopub.status.idle":"2024-11-08T04:10:09.094890Z","shell.execute_reply.started":"2024-11-08T04:10:08.933807Z","shell.execute_reply":"2024-11-08T04:10:09.094133Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Loss function","metadata":{"trusted":true,"collapsed":true,"_uuid":"9c71a3af2a83b1122902a3f39e6fda1af5afbb99","jupyter":{"outputs_hidden":true}}},{"cell_type":"markdown","source":"Loss function is one of the most crucial parts of the completion. Due to strong data unbalance, simple loss functions, such as Binary Cross-Entropy loss, do not really work. Soft dice loss can be helpful since it boosts prediction of correct masks, but it leads to unstable training. Winners of image segmentation challenges typically combine BCE loss with dice (http://blog.kaggle.com/2017/12/22/carvana-image-masking-first-place-interview/). Similar loss function is used in publicly available models in this completion. I would agree that this combined loss function works perfectly for Carvana completion, where the number of pixels in the mask is about half of the total number of pixels. However, 1:1000 pixel unbalance deteriorates training with BCE. \nIf one tries to recall what is the loss function that should be used for strongly unbalanced data set, it is focal loss (https://arxiv.org/pdf/1708.02002.pdf), which revolutionized one stage object localization method in 2017. This loss function demonstrates amazing results on datasets with unbalance level 1:10-1000. In addition to focal loss, I include -log(soft dice loss). Log is important in the convex of the current competition since it boosts the loss for the cases when objects are not detected correctly and dice is close to zero. It allows to avoid false negative predictions or completely incorrect masks for images with one ship (the major part of the training set). Also, since the loss for such objects is very high, the model more effectively incorporates the knowledge about such objects and handles them even in images with multiple ships. To bring two losses to similar scale, focal loss is multiplied by 10. The implementation of focal loss is borrowed from https://becominghuman.ai/investigating-focal-and-dice-loss-for-the-kaggle-2018-data-science-bowl-65fb9af4f36c .","metadata":{"_uuid":"19d0fe1a3e00cea135163b5540338e2744615c3d"}},{"cell_type":"code","source":"def dice_loss(input, target):\n    input = torch.sigmoid(input)\n    smooth = 1.0\n\n    iflat = input.view(-1)\n    tflat = target.view(-1)\n    intersection = (iflat * tflat).sum()\n    \n    return ((2.0 * intersection + smooth) / (iflat.sum() + tflat.sum() + smooth))","metadata":{"trusted":true,"_uuid":"8de4222fd28596dbae5d55f8172ac28271f678f2","execution":{"iopub.status.busy":"2024-11-08T04:10:15.196038Z","iopub.execute_input":"2024-11-08T04:10:15.196370Z","iopub.status.idle":"2024-11-08T04:10:15.205270Z","shell.execute_reply.started":"2024-11-08T04:10:15.196315Z","shell.execute_reply":"2024-11-08T04:10:15.204613Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class FocalLoss(nn.Module):\n    def __init__(self, gamma):\n        super().__init__()\n        self.gamma = gamma\n        \n    def forward(self, input, target):\n        if not (target.size() == input.size()):\n            raise ValueError(\"Target size ({}) must be the same as input size ({})\"\n                             .format(target.size(), input.size()))\n\n        max_val = (-input).clamp(min=0)\n        loss = input - input * target + max_val + \\\n            ((-max_val).exp() + (-input - max_val).exp()).log()\n\n        invprobs = F.logsigmoid(-input * (target * 2.0 - 1.0))\n        loss = (invprobs * self.gamma).exp() * loss\n        \n        return loss.mean()","metadata":{"trusted":true,"_uuid":"456acf8ffe7b80cd94eb73bfaeb6b1061bbb44c7","execution":{"iopub.status.busy":"2024-11-08T04:10:16.672256Z","iopub.execute_input":"2024-11-08T04:10:16.672569Z","iopub.status.idle":"2024-11-08T04:10:16.695310Z","shell.execute_reply.started":"2024-11-08T04:10:16.672507Z","shell.execute_reply":"2024-11-08T04:10:16.694458Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class MixedLoss(nn.Module):\n    def __init__(self, alpha, gamma):\n        super().__init__()\n        self.alpha = alpha\n        self.focal = FocalLoss(gamma)\n        \n    def forward(self, input, target):\n        loss = self.alpha*self.focal(input, target) - torch.log(dice_loss(input, target))\n        return loss.mean()","metadata":{"trusted":true,"_uuid":"1cfb00a50821861c69534ee7398bd220c02900d8","execution":{"iopub.status.busy":"2024-11-08T04:10:17.978474Z","iopub.execute_input":"2024-11-08T04:10:17.978751Z","iopub.status.idle":"2024-11-08T04:10:17.988537Z","shell.execute_reply.started":"2024-11-08T04:10:17.978709Z","shell.execute_reply":"2024-11-08T04:10:17.987717Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def dice(pred, targs):\n    pred = (pred>0).float()\n    return 2.0 * (pred*targs).sum() / ((pred+targs).sum() + 1.0)\n\ndef IoU(pred, targs):\n    pred = (pred>0).float()\n    intersection = (pred*targs).sum()\n    return intersection / ((pred+targs).sum() - intersection + 1.0)","metadata":{"trusted":true,"_uuid":"37baddf38bb569cbf2d2f186a5171767402b94fa","execution":{"iopub.status.busy":"2024-11-08T04:10:19.124230Z","iopub.execute_input":"2024-11-08T04:10:19.124496Z","iopub.status.idle":"2024-11-08T04:10:19.134734Z","shell.execute_reply.started":"2024-11-08T04:10:19.124458Z","shell.execute_reply":"2024-11-08T04:10:19.134039Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Training","metadata":{"_uuid":"0b3a20e5e2fdedf1439737f40c14c78f9a69e29a"}},{"cell_type":"code","source":"m_base = load_pretrained(get_base(),PRETRAINED)\nm = to_gpu(Unet34(m_base))\nmodels = UnetModel(m)","metadata":{"trusted":true,"_uuid":"1a3b8f6b454a8d8b48505fbc5964e900a59ef09e","execution":{"iopub.status.busy":"2024-11-08T04:10:22.853241Z","iopub.execute_input":"2024-11-08T04:10:22.853522Z","iopub.status.idle":"2024-11-08T04:10:30.200729Z","shell.execute_reply.started":"2024-11-08T04:10:22.853481Z","shell.execute_reply":"2024-11-08T04:10:30.199931Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"models.model","metadata":{"trusted":true,"_uuid":"3d52f2e1e45ec597d901a814f592ac153168ebff","execution":{"iopub.status.busy":"2024-11-08T04:10:30.201729Z","iopub.execute_input":"2024-11-08T04:10:30.201982Z","iopub.status.idle":"2024-11-08T04:10:30.209862Z","shell.execute_reply.started":"2024-11-08T04:10:30.201930Z","shell.execute_reply":"2024-11-08T04:10:30.209288Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sz = 256 #image size\nbs = 64  #batch size\n\nmd = get_data(sz,bs)","metadata":{"trusted":true,"_uuid":"ab92ee6aaa8d31587d1bc6d2cad9e0407444c5de","execution":{"iopub.status.busy":"2024-11-08T04:10:37.759837Z","iopub.execute_input":"2024-11-08T04:10:37.760147Z","iopub.status.idle":"2024-11-08T04:10:41.482914Z","shell.execute_reply.started":"2024-11-08T04:10:37.760088Z","shell.execute_reply":"2024-11-08T04:10:41.482149Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn = ConvLearner(md, models)\nlearn.opt_fn=optim.Adam\nlearn.crit = MixedLoss(10.0, 2.0)\nlearn.metrics=[accuracy_thresh(0.5),dice,IoU]\nwd=1e-7\nlr = 1e-2","metadata":{"trusted":true,"_uuid":"eb1016214b0de75f56d2036241a27684b4c4bd1b","execution":{"iopub.status.busy":"2024-11-08T04:10:43.084742Z","iopub.execute_input":"2024-11-08T04:10:43.085067Z","iopub.status.idle":"2024-11-08T04:10:43.095828Z","shell.execute_reply.started":"2024-11-08T04:10:43.085005Z","shell.execute_reply":"2024-11-08T04:10:43.095090Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.freeze_to(1)","metadata":{"trusted":true,"_uuid":"4d4aa02ec617bdf974fcdd77497af1ce5d9993aa","execution":{"iopub.status.busy":"2024-11-08T04:10:45.686238Z","iopub.execute_input":"2024-11-08T04:10:45.686528Z","iopub.status.idle":"2024-11-08T04:10:45.694742Z","shell.execute_reply.started":"2024-11-08T04:10:45.686486Z","shell.execute_reply":"2024-11-08T04:10:45.693898Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Training only the decoder part for 1 epoch (15 min) leads to ~0.8 dice that outperforms all publicly available models in this competition.","metadata":{"_uuid":"0d2d28f3482827318f673aee6fe3e60e2cf0ac44"}},{"cell_type":"code","source":"learn.fit(lr,1,wds=wd,cycle_len=1,use_clr=(5,8))","metadata":{"trusted":true,"_uuid":"fd9510104082f97119adebc00bb16750710266a2","scrolled":true,"collapsed":true,"jupyter":{"outputs_hidden":true},"execution":{"iopub.status.busy":"2024-11-08T04:10:49.989592Z","iopub.execute_input":"2024-11-08T04:10:49.989876Z","iopub.status.idle":"2024-11-08T04:25:48.109089Z","shell.execute_reply.started":"2024-11-08T04:10:49.989833Z","shell.execute_reply":"2024-11-08T04:25:48.108415Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.save('Unet34_256_0')","metadata":{"trusted":true,"_uuid":"a06ef9bc205d434ef5be901ea489a368e24d1435","execution":{"iopub.status.busy":"2024-11-08T04:25:51.502753Z","iopub.execute_input":"2024-11-08T04:25:51.503035Z","iopub.status.idle":"2024-11-08T04:25:51.599988Z","shell.execute_reply.started":"2024-11-08T04:25:51.502993Z","shell.execute_reply":"2024-11-08T04:25:51.599417Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Unfreeze the model and train it with differential learning rate. The lr of the head part is still 1e-3, while the middle layers of the model are trained with 1e-4 lr, and the base is trained with even smaller lr, 1e-5, since low level detectors do not vary much from one image data set to another.","metadata":{"_uuid":"a1288146ed507f7a6876d756e5472f8dda15cc58"}},{"cell_type":"code","source":"lrs = np.array([lr/100,lr/10,lr])\nlearn.unfreeze() #unfreeze the encoder\nlearn.bn_freeze(True)","metadata":{"trusted":true,"_uuid":"eef154a2aa260f370c9665ec804b1abecdef3ccf","execution":{"iopub.status.busy":"2024-11-08T04:25:56.883916Z","iopub.execute_input":"2024-11-08T04:25:56.884205Z","iopub.status.idle":"2024-11-08T04:25:56.896582Z","shell.execute_reply.started":"2024-11-08T04:25:56.884146Z","shell.execute_reply":"2024-11-08T04:25:56.895568Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.fit(lrs,2,wds=wd,cycle_len=1,use_clr=(20,8))","metadata":{"trusted":true,"_uuid":"8498665da79f8a147b0599142bb4034e399c38aa","execution":{"iopub.status.busy":"2024-11-08T04:26:00.101473Z","iopub.execute_input":"2024-11-08T04:26:00.101755Z","iopub.status.idle":"2024-11-08T04:48:21.715141Z","shell.execute_reply.started":"2024-11-08T04:26:00.101713Z","shell.execute_reply":"2024-11-08T04:48:21.714436Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.fit(lrs/3,2,wds=wd,cycle_len=2,use_clr=(20,8))","metadata":{"trusted":true,"_uuid":"f8d9a6f6ecb66054d32e8c41d316672dbd9bb03b","execution":{"iopub.status.busy":"2024-11-08T04:48:24.968575Z","iopub.execute_input":"2024-11-08T04:48:24.968906Z","iopub.status.idle":"2024-11-08T05:33:49.529101Z","shell.execute_reply.started":"2024-11-08T04:48:24.968837Z","shell.execute_reply":"2024-11-08T05:33:49.528441Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The training has been run with learning rate annealing. Periodic lr increase followed by slow decrease drives the system out of steep minima (when lr is high) towards broader ones (which are explored when lr decreases) that enhances the ability of the model to generalize and reduces overfitting.","metadata":{"_uuid":"5c2b287d62042002e17cf7797ae48ccd05dd36cd"}},{"cell_type":"code","source":"learn.sched.plot_lr()","metadata":{"trusted":true,"_uuid":"1fd829270a788812cb9f3f7f5fe7c6e43d934ab7","execution":{"iopub.status.busy":"2024-11-08T05:35:29.489610Z","iopub.execute_input":"2024-11-08T05:35:29.489893Z","iopub.status.idle":"2024-11-08T05:35:29.648754Z","shell.execute_reply.started":"2024-11-08T05:35:29.489850Z","shell.execute_reply":"2024-11-08T05:35:29.648067Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Saved model can be ued for further training or for making predictions.","metadata":{"_uuid":"16ea8e6c7a593be13aad3cc6fe88e4cb657db35d"}},{"cell_type":"code","source":"learn.save('Unet34_256_1')","metadata":{"trusted":true,"_uuid":"df25dc961326f3e554b9c4d38525705664ec5075","execution":{"iopub.status.busy":"2024-11-08T05:35:33.471457Z","iopub.execute_input":"2024-11-08T05:35:33.471727Z","iopub.status.idle":"2024-11-08T05:35:33.573630Z","shell.execute_reply.started":"2024-11-08T05:35:33.471693Z","shell.execute_reply":"2024-11-08T05:35:33.573020Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Visualization","metadata":{"_uuid":"a5f92ae7e3e348f95129a3742ad76e4a502bf6d1"}},{"cell_type":"code","source":"def Show_images(x,yp,yt):\n    columns = 3\n    rows = min(bs,8)\n    fig=plt.figure(figsize=(columns*4, rows*4))\n    for i in range(rows):\n        fig.add_subplot(rows, columns, 3*i+1)\n        plt.axis('off')\n        plt.imshow(x[i])\n        fig.add_subplot(rows, columns, 3*i+2)\n        plt.axis('off')\n        plt.imshow(yp[i])\n        fig.add_subplot(rows, columns, 3*i+3)\n        plt.axis('off')\n        plt.imshow(yt[i])\n    plt.show()","metadata":{"trusted":true,"_uuid":"fb4a8386560ebde1f2277d94ecb5ee4a1bbc50d2","execution":{"iopub.status.busy":"2024-11-08T05:35:36.029307Z","iopub.execute_input":"2024-11-08T05:35:36.029576Z","iopub.status.idle":"2024-11-08T05:35:36.049771Z","shell.execute_reply.started":"2024-11-08T05:35:36.029536Z","shell.execute_reply":"2024-11-08T05:35:36.049093Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.model.eval();\nx,y = next(iter(md.val_dl))\nyp = to_np(F.sigmoid(learn.model(V(x))))","metadata":{"trusted":true,"_uuid":"0206fb587e6bb9ddfc7366f6d04e47101b275514","execution":{"iopub.status.busy":"2024-11-08T05:35:37.397907Z","iopub.execute_input":"2024-11-08T05:35:37.398253Z","iopub.status.idle":"2024-11-08T05:35:40.230517Z","shell.execute_reply.started":"2024-11-08T05:35:37.398166Z","shell.execute_reply":"2024-11-08T05:35:40.229757Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"Show_images(np.asarray(md.val_ds.denorm(x)), yp, y)","metadata":{"trusted":true,"_uuid":"8cda6a85fcbe5263406777a19729d0aec890ab99","execution":{"iopub.status.busy":"2024-11-08T05:35:42.389087Z","iopub.execute_input":"2024-11-08T05:35:42.389424Z","iopub.status.idle":"2024-11-08T05:35:44.087795Z","shell.execute_reply.started":"2024-11-08T05:35:42.389365Z","shell.execute_reply":"2024-11-08T05:35:44.086812Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The results are not ideal, but almost all ships are captured correctly even if the model is making the prediction on a very low resolution (256x256) images.","metadata":{"_uuid":"9850c2014ba1980bc0cca9b9a1177af5caa24da0"}},{"cell_type":"markdown","source":"### Training (384x384)","metadata":{"_uuid":"8f0b3a89ccb06468283d8e2611c3981895aae24a"}},{"cell_type":"markdown","source":"Fortunately, modern convolutional nets support input images of arbitrary resolution. To decrease the training time, one can start training the model on low resolution images first and continue training on higher resolution images for fewer epochs. In addition, a model pretrained on low resolution images first generalizes better since a pixel information is less available and high order features are tended to be used.","metadata":{"_uuid":"43deeffeee35bde4b11ae3a1aae13ecfabeebf4c"}},{"cell_type":"code","source":"sz = 384 #image size\nbs = 32  #batch size\n\nmd = get_data(sz,bs)\nlearn.set_data(md)\nlearn.unfreeze()\nlearn.bn_freeze(True)","metadata":{"trusted":true,"_uuid":"99f0972e31576ba82335a54825cc71853e7a5afd","execution":{"iopub.status.busy":"2024-11-08T05:37:11.880455Z","iopub.execute_input":"2024-11-08T05:37:11.880782Z","iopub.status.idle":"2024-11-08T05:37:15.637766Z","shell.execute_reply.started":"2024-11-08T05:37:11.880723Z","shell.execute_reply":"2024-11-08T05:37:15.637011Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Due to the kernel run time limit, the model was further trained only for two epochs on 384x384 (dice ~0.87) followed by one epoch on 768x768 images. In an independent run I trained a model on 384x384 images for 12 epochs that boosted dice to 0.89 followed by training on full resolution images that increased dice further to 0.905.","metadata":{"_uuid":"2cc8a6dd9fbf4d03893088d3be3e259942089d30"}},{"cell_type":"code","source":"learn.fit(lrs/5,1,wds=wd,cycle_len=2,use_clr=(10,8))","metadata":{"trusted":true,"_uuid":"5d40a6e24844002d52bb844d2a4873c164d06e39","execution":{"iopub.status.busy":"2024-11-08T05:38:36.870505Z","iopub.execute_input":"2024-11-08T05:38:36.870836Z","iopub.status.idle":"2024-11-08T06:10:59.780166Z","shell.execute_reply.started":"2024-11-08T05:38:36.870768Z","shell.execute_reply":"2024-11-08T06:10:59.779475Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.save('Unet34_384_1')","metadata":{"trusted":true,"_uuid":"fce1cccfb4fd63853d31ae3af9f2e2897ae1e802","execution":{"iopub.status.busy":"2024-11-08T06:11:43.237333Z","iopub.execute_input":"2024-11-08T06:11:43.237595Z","iopub.status.idle":"2024-11-08T06:11:43.340980Z","shell.execute_reply.started":"2024-11-08T06:11:43.237557Z","shell.execute_reply":"2024-11-08T06:11:43.340415Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Visualization","metadata":{"_uuid":"0e129b20579e8dff1523f9da36e4b54bcbde9b58"}},{"cell_type":"code","source":"learn.model.eval();\nx,y = next(iter(md.val_dl))\nyp = to_np(F.sigmoid(learn.model(V(x))))","metadata":{"trusted":true,"_uuid":"c92425b7455925580e6d03b373babaebc261047e","execution":{"iopub.status.busy":"2024-11-08T06:11:52.212791Z","iopub.execute_input":"2024-11-08T06:11:52.213054Z","iopub.status.idle":"2024-11-08T06:11:54.368792Z","shell.execute_reply.started":"2024-11-08T06:11:52.213014Z","shell.execute_reply":"2024-11-08T06:11:54.368100Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"Show_images(np.asarray(md.val_ds.denorm(x)), yp, y)","metadata":{"trusted":true,"_uuid":"507ef05ee2301e0ebcc69a679a4eb57a99a88e4f","execution":{"iopub.status.busy":"2024-11-08T06:11:56.034152Z","iopub.execute_input":"2024-11-08T06:11:56.034470Z","iopub.status.idle":"2024-11-08T06:11:57.749378Z","shell.execute_reply.started":"2024-11-08T06:11:56.034424Z","shell.execute_reply":"2024-11-08T06:11:57.748630Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Training (768x768)","metadata":{"_uuid":"8a2b5a8a6d4826f8ef2ccb1b1e26c749edae3aa9"}},{"cell_type":"code","source":"sz = 768 #image size\nbs = 6  #batch size\n\nmd = get_data(sz,bs)\nlearn.set_data(md)\nlearn.unfreeze()\nlearn.bn_freeze(True)","metadata":{"trusted":true,"_uuid":"82744b9640856bc71a861240263f8d314544ebe6","execution":{"iopub.status.busy":"2024-11-08T06:12:01.710203Z","iopub.execute_input":"2024-11-08T06:12:01.710481Z","iopub.status.idle":"2024-11-08T06:12:05.422892Z","shell.execute_reply.started":"2024-11-08T06:12:01.710440Z","shell.execute_reply":"2024-11-08T06:12:05.422298Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.fit(lrs/10,1,wds=wd,cycle_len=1,use_clr=(10,8))","metadata":{"trusted":true,"_uuid":"dfbb04dc51276c62a97b5a02ff92dbb20b206a3f","execution":{"iopub.status.busy":"2024-11-08T06:12:08.286117Z","iopub.execute_input":"2024-11-08T06:12:08.286442Z","iopub.status.idle":"2024-11-08T07:07:41.273482Z","shell.execute_reply.started":"2024-11-08T06:12:08.286382Z","shell.execute_reply":"2024-11-08T07:07:41.272713Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Training for just one epoch is insufficient to achieve high dice. However, if the training is continued, the dice can reach 0.90+.","metadata":{"_uuid":"addc63f66aa829048a2e285964771967d409aa93"}},{"cell_type":"code","source":"learn.save('Unet34_768_1')","metadata":{"trusted":true,"_uuid":"b6bc3ceacb7e616594fd2c8ea960e2e733ba81de","execution":{"iopub.status.busy":"2024-11-08T07:08:01.059291Z","iopub.execute_input":"2024-11-08T07:08:01.059569Z","iopub.status.idle":"2024-11-08T07:08:01.170904Z","shell.execute_reply.started":"2024-11-08T07:08:01.059528Z","shell.execute_reply":"2024-11-08T07:08:01.170303Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"And finally, I put a picture (original image, prediction, ground truth) obtained by the model with dice 0.895 trained further on full resolution images in an independent run. Apart from one tiny ship in the last image, everything is captured. When I zoomed it in, it is really not clear if it is a ship of just a small island: I see only several white pixels, and there are several small islands under the water. Another interesting thing is that the model is able to capture details that it was not trained for. In particular, in 4-th image the model captures antennas (upper right ship) and the shape of ships, even if training set is composed of pixelized bounding boxes.","metadata":{"_uuid":"c7584a0ba0ad6667987757dbe5cf3a3dcfea6614"}},{"cell_type":"markdown","source":"![1](https://image.ibb.co/mrqdze/Ship_Detection.png)","metadata":{"_uuid":"4e23a4bf43cf33af8e36b93035642b30e3853bea"}}]}