{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Baseline Torch U-net with Resnet34\n\n\nThis is a very fast implementation of a U-net with a Resnet34, using torch. Both train an inference happen in this notebook.\n\nThe code is simple enough to be the starting building block of a more robust system, but I wanted it to cover all the relevant steps at least to a certain extent.\nIt correctly articulates the train dataset, trains an U-net for some epochs, obtains results correctly and post-process them to get a valid submission.\n\nThere is no validation and no IoU measurements, and also the resizing is done in a very rough manner. There are various low-hanging fruits with data augmentation too. The post-processing (the splitting of the segmentation mask into different \"individual\" predictions) is done with conected components.\n\nThis is a full pipeline from zero to a submission with a score greater than zero though, with all the internet dependencies removed, and a lot of places for improvements and quick-wins. \n\n\nThe code is mostly an adaption from [this](https://www.kaggle.com/rishabhiitbhu/unet-with-resnet34-encoder-pytorch/notebook) with various specificities for this competition.\n\n### DO upvote!!\n\n\n<h3 style=\"background-color:#C8FF33;padding:40px;border-radius: 30px;\">\nThe U-net model hits a very low ceiling. Check the following starter notebooks, based on a pure pytorch Mask R-CNN:\n    <br>\n* <a href=\"https://www.kaggle.com/julian3833/sartorius-starter-torch-mask-r-cnn-lb-0-273\">🦠 Sartorius - Starter Torch Mask R-CNN [LB=0.273] </a>\n    <br>\n* <b><a href=\"https://www.kaggle.com/julian3833/sartorius-classifier-mask-r-cnn-lb-0-28\">🦠 Sartorius - Classifier + Mask R-CNN [LB=0.28]</a></b>\n    <br>\n* See also: <a href=\"https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/279790\"> this topic</a>\n</h3>\n\n\n&nbsp;\n&nbsp;\n&nbsp;\n&nbsp;\n\n\n\n## Resources\n\n* [UNet with ResNet34 encoder (Pytorch)](https://www.kaggle.com/rishabhiitbhu/unet-with-resnet34-encoder-pytorch/notebook)\nThe code is an adaption of this notebook\n\n* [pytorch-pretrained-image-models](https://www.kaggle.com/bminixhofer/pytorch-pretrained-image-models)\nThe \"offline\" pytorch's resnet weights come from this dataset\n* [segmentation-models-wheels](https://www.kaggle.com/arunmohan003/segmentation-models-wheels)\nThe \"offline\" segmentation library comes from here\n* [hubmap-hacking-the-kidney](https://www.kaggle.com/c7934597/hubmap-hacking-the-kidney)\nI learned how to use the the previous library from this notebook\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"# Install libraries (offline)","metadata":{}},{"cell_type":"code","source":"%%capture\n!pip install ../input/segmentation-models-wheels/efficientnet_pytorch-0.6.3-py3-none-any.whl\n!pip install ../input/segmentation-models-wheels/pretrainedmodels-0.7.4-py3-none-any.whl\n!pip install ../input/segmentation-models-wheels/timm-0.3.2-py3-none-any.whl\n!pip install ../input/segmentation-models-wheels/segmentation_models_pytorch-0.1.3-py3-none-any.whl","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:48:53.650193Z","iopub.execute_input":"2021-10-19T00:48:53.65053Z","iopub.status.idle":"2021-10-19T00:50:44.048079Z","shell.execute_reply.started":"2021-10-19T00:48:53.650459Z","shell.execute_reply":"2021-10-19T00:50:44.047096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Import Dependencies","metadata":{}},{"cell_type":"code","source":"import os\nimport cv2\nimport pdb\nimport time\nimport warnings\nimport random\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm_notebook as tqdm\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\nfrom sklearn.model_selection import KFold\nimport torch\nimport torch.nn as nn\nfrom torch.nn import functional as F\nimport torch.optim as optim\nimport torch.backends.cudnn as cudnn\nfrom torch.utils.data import DataLoader, Dataset, sampler\nfrom matplotlib import pyplot as plt\nfrom albumentations import (HorizontalFlip, VerticalFlip, ShiftScaleRotate, Normalize, Resize, Compose, GaussNoise)\nfrom albumentations.pytorch import ToTensorV2\nwarnings.filterwarnings(\"ignore\")\n\n\ndef fix_all_seeds(seed):\n    np.random.seed(seed)\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\n    \nfix_all_seeds(2021)\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-10-19T00:50:44.050107Z","iopub.execute_input":"2021-10-19T00:50:44.050444Z","iopub.status.idle":"2021-10-19T00:50:47.593865Z","shell.execute_reply.started":"2021-10-19T00:50:44.050341Z","shell.execute_reply":"2021-10-19T00:50:47.5929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Define constants and load df","metadata":{}},{"cell_type":"code","source":"SAMPLE_SUBMISSION  = '../input/sartorius-cell-instance-segmentation/sample_submission.csv'\nTRAIN_CSV = \"../input/sartorius-cell-instance-segmentation/train.csv\"\nTRAIN_PATH = \"../input/sartorius-cell-instance-segmentation/train\"\nTEST_PATH = \"../input/sartorius-cell-instance-segmentation/test\"\n\nRESNET_MEAN = (0.485, 0.456, 0.406)\nRESNET_STD = (0.229, 0.224, 0.225)\n\n# (336, 336)\nIMAGE_RESIZE = (224, 224)\n\nLEARNING_RATE = 5e-4\nEPOCHS = 12","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:50:47.595378Z","iopub.execute_input":"2021-10-19T00:50:47.595624Z","iopub.status.idle":"2021-10-19T00:50:47.60085Z","shell.execute_reply.started":"2021-10-19T00:50:47.59559Z","shell.execute_reply":"2021-10-19T00:50:47.600121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = pd.read_csv(TRAIN_CSV)\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:50:47.602963Z","iopub.execute_input":"2021-10-19T00:50:47.603634Z","iopub.status.idle":"2021-10-19T00:50:48.16544Z","shell.execute_reply.started":"2021-10-19T00:50:47.603594Z","shell.execute_reply":"2021-10-19T00:50:48.164634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training Dataset and Dataloader","metadata":{}},{"cell_type":"markdown","source":"## Utilities","metadata":{}},{"cell_type":"code","source":"def rle_decode(mask_rle, shape, color=1):\n    '''\n    mask_rle: run-length as string formated (start length)\n    shape: (height,width) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n\n    '''\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0] * shape[1], dtype=np.float32)\n    for lo, hi in zip(starts, ends):\n        img[lo : hi] = color\n    return img.reshape(shape)\n\n\ndef build_masks(df_train, image_id, input_shape):\n    height, width = input_shape\n    labels = df_train[df_train[\"id\"] == image_id][\"annotation\"].tolist()\n    mask = np.zeros((height, width))\n    for label in labels:\n        mask += rle_decode(label, shape=(height, width))\n    mask = mask.clip(0, 1)\n    return mask","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:50:48.166839Z","iopub.execute_input":"2021-10-19T00:50:48.167082Z","iopub.status.idle":"2021-10-19T00:50:48.17571Z","shell.execute_reply.started":"2021-10-19T00:50:48.167049Z","shell.execute_reply":"2021-10-19T00:50:48.174889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dataset and Dataloader","metadata":{}},{"cell_type":"code","source":"class CellDataset(Dataset):\n    def __init__(self, df):\n        self.df = df\n        self.base_path = TRAIN_PATH\n        self.transforms = Compose([Resize(IMAGE_RESIZE[0], IMAGE_RESIZE[1]), \n                                   Normalize(mean=RESNET_MEAN, std=RESNET_STD, p=1), \n                                   HorizontalFlip(p=0.5),\n                                   VerticalFlip(p=0.5),\n                                   ToTensorV2()])\n        self.gb = self.df.groupby('id')\n        self.image_ids = df.id.unique().tolist()\n\n    def __getitem__(self, idx):\n        image_id = self.image_ids[idx]\n        df = self.gb.get_group(image_id)\n        annotations = df['annotation'].tolist()\n        image_path = os.path.join(self.base_path, image_id + \".png\")\n        image = cv2.imread(image_path)\n        mask = build_masks(df_train, image_id, input_shape=(520, 704))\n        mask = (mask >= 1).astype('float32')\n        augmented = self.transforms(image=image, mask=mask)\n        image = augmented['image']\n        mask = augmented['mask']\n        return image, mask.reshape((1, IMAGE_RESIZE[0], IMAGE_RESIZE[1]))\n\n    def __len__(self):\n        return len(self.image_ids)","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:50:48.177368Z","iopub.execute_input":"2021-10-19T00:50:48.177703Z","iopub.status.idle":"2021-10-19T00:50:48.188333Z","shell.execute_reply.started":"2021-10-19T00:50:48.177669Z","shell.execute_reply":"2021-10-19T00:50:48.18768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds_train = CellDataset(df_train)\nimage, mask = ds_train[1]\nimage.shape, mask.shape","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:50:48.189638Z","iopub.execute_input":"2021-10-19T00:50:48.190095Z","iopub.status.idle":"2021-10-19T00:50:48.333245Z","shell.execute_reply.started":"2021-10-19T00:50:48.19006Z","shell.execute_reply":"2021-10-19T00:50:48.332597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(image[0], cmap='bone')\nplt.show()\nplt.imshow(mask[0], alpha=0.3)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:50:48.3345Z","iopub.execute_input":"2021-10-19T00:50:48.334739Z","iopub.status.idle":"2021-10-19T00:50:48.876002Z","shell.execute_reply.started":"2021-10-19T00:50:48.334708Z","shell.execute_reply":"2021-10-19T00:50:48.875364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dl_train = DataLoader(ds_train, batch_size=64, num_workers=4, pin_memory=True, shuffle=False)","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:50:48.877353Z","iopub.execute_input":"2021-10-19T00:50:48.877605Z","iopub.status.idle":"2021-10-19T00:50:48.88257Z","shell.execute_reply.started":"2021-10-19T00:50:48.87757Z","shell.execute_reply":"2021-10-19T00:50:48.881797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(dl_train)","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:50:48.885841Z","iopub.execute_input":"2021-10-19T00:50:48.886298Z","iopub.status.idle":"2021-10-19T00:50:48.896258Z","shell.execute_reply.started":"2021-10-19T00:50:48.886261Z","shell.execute_reply":"2021-10-19T00:50:48.895571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get a batch from the dataloader\nbatch = next(iter(dl_train))\nimages, masks = batch","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:50:48.897483Z","iopub.execute_input":"2021-10-19T00:50:48.897739Z","iopub.status.idle":"2021-10-19T00:50:56.518308Z","shell.execute_reply.started":"2021-10-19T00:50:48.897705Z","shell.execute_reply":"2021-10-19T00:50:56.5174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"idx=1\nplt.imshow(images[idx][0], cmap='bone')\nplt.show()\nplt.imshow(masks[idx][0], alpha=0.3)\nplt.show()\nplt.imshow(images[idx][0], cmap='bone')\nplt.imshow(masks[idx][0], alpha=0.3)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:50:56.520836Z","iopub.execute_input":"2021-10-19T00:50:56.521287Z","iopub.status.idle":"2021-10-19T00:50:57.136699Z","shell.execute_reply.started":"2021-10-19T00:50:56.521245Z","shell.execute_reply":"2021-10-19T00:50:57.136003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Losses","metadata":{}},{"cell_type":"code","source":"def dice_loss(input, target):\n    input = torch.sigmoid(input)\n    smooth = 1.0\n    iflat = input.view(-1)\n    tflat = target.view(-1)\n    intersection = (iflat * tflat).sum()\n    return ((2.0 * intersection + smooth) / (iflat.sum() + tflat.sum() + smooth))\n\n\nclass FocalLoss(nn.Module):\n    def __init__(self, gamma):\n        super().__init__()\n        self.gamma = gamma\n\n    def forward(self, input, target):\n        if not (target.size() == input.size()):\n            raise ValueError(\"Target size ({}) must be the same as input size ({})\"\n                             .format(target.size(), input.size()))\n        max_val = (-input).clamp(min=0)\n        loss = input - input * target + max_val + \\\n            ((-max_val).exp() + (-input - max_val).exp()).log()\n        invprobs = F.logsigmoid(-input * (target * 2.0 - 1.0))\n        loss = (invprobs * self.gamma).exp() * loss\n        return loss.mean()\n\n\nclass MixedLoss(nn.Module):\n    def __init__(self, alpha, gamma):\n        super().__init__()\n        self.alpha = alpha\n        self.focal = FocalLoss(gamma)\n\n    def forward(self, input, target):\n        loss = self.alpha*self.focal(input, target) - torch.log(dice_loss(input, target))\n        return loss.mean()","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:50:57.137988Z","iopub.execute_input":"2021-10-19T00:50:57.138328Z","iopub.status.idle":"2021-10-19T00:50:57.150138Z","shell.execute_reply.started":"2021-10-19T00:50:57.138296Z","shell.execute_reply":"2021-10-19T00:50:57.149331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model: U-net\n\nIn order to comply with the no-Internet restriction I am placing the resnet from a dataset into the `.cache` path.\nAlso, there is an error when importing `segmentation_models_pytorch`, that I solved setting a private attributre in `torch` 🤦‍♂️.\n\nI solved both problems in a quick-and-dirty way for now.","metadata":{}},{"cell_type":"code","source":"!mkdir -p /root/.cache/torch/hub/checkpoints/\n!cp ../input/pytorch-pretrained-image-models/resnet34.pth /root/.cache/torch/hub/checkpoints/resnet34-333f7ec4.pth\n\nimport torch\nimport collections.abc as container_abcs\ntorch._six.container_abcs = container_abcs\nimport segmentation_models_pytorch as smp","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:50:57.151354Z","iopub.execute_input":"2021-10-19T00:50:57.151636Z","iopub.status.idle":"2021-10-19T00:51:02.021265Z","shell.execute_reply.started":"2021-10-19T00:50:57.151597Z","shell.execute_reply":"2021-10-19T00:51:02.020427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# U-Net","metadata":{}},{"cell_type":"code","source":"model = smp.Unet(\"resnet34\", encoder_weights=\"imagenet\", activation=None)","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:51:02.022972Z","iopub.execute_input":"2021-10-19T00:51:02.023235Z","iopub.status.idle":"2021-10-19T00:51:02.454793Z","shell.execute_reply.started":"2021-10-19T00:51:02.023198Z","shell.execute_reply":"2021-10-19T00:51:02.454038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check model details\n# model","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:51:02.456206Z","iopub.execute_input":"2021-10-19T00:51:02.456487Z","iopub.status.idle":"2021-10-19T00:51:02.460132Z","shell.execute_reply.started":"2021-10-19T00:51:02.456449Z","shell.execute_reply":"2021-10-19T00:51:02.459231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training loop\n\nNo validation or k-folds for now, just get it running for few epochs.","metadata":{}},{"cell_type":"code","source":"torch.set_default_tensor_type(\"torch.cuda.FloatTensor\")\nn_batches = len(dl_train)\n\nmodel.cuda()\nmodel.train()\n\ncriterion = MixedLoss(10.0, 2.0)\noptimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)\n\nfor epoch in range(1, EPOCHS + 1):\n    print(f\"Starting epoch: {epoch} / {EPOCHS}\")\n    running_loss = 0.0\n    optimizer.zero_grad()\n    \n    for batch_idx, batch in enumerate(dl_train):\n        \n        # Predict\n        images, masks = batch\n        images, masks = images.cuda(),  masks.cuda()\n        outputs = model(images)\n        loss = criterion(outputs, masks)\n        \n        # Back prop\n        loss.backward()\n        optimizer.step()\n        optimizer.zero_grad()\n        running_loss += loss.item()\n\n    epoch_loss = running_loss / n_batches\n    print(f\"Epoch: {epoch} - Train Loss {epoch_loss:.4f}\")","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:51:02.461483Z","iopub.execute_input":"2021-10-19T00:51:02.461728Z","iopub.status.idle":"2021-10-19T00:51:45.371509Z","shell.execute_reply.started":"2021-10-19T00:51:02.461696Z","shell.execute_reply":"2021-10-19T00:51:45.370623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predict","metadata":{}},{"cell_type":"markdown","source":"## Test Dataset and DataLoader","metadata":{}},{"cell_type":"code","source":"class TestCellDataset(Dataset):\n    def __init__(self):\n        self.test_path = TEST_PATH\n        \n        # I am not sure if they adapt the sample submission csv or only the test folder\n        # I am using the test folders as the ground truth for the images to predict, which should be always right\n        # The sample csv is ignored\n        self.image_ids = [f[:-4]for f in os.listdir(self.test_path)]\n        self.num_samples = len(self.image_ids)\n        self.transform = Compose([Resize(IMAGE_RESIZE[0], IMAGE_RESIZE[1]), Normalize(mean=RESNET_MEAN, std=RESNET_STD, p=1), ToTensorV2()])\n\n    def __getitem__(self, idx):\n        image_id = self.image_ids[idx]\n        path = os.path.join(self.test_path, image_id + \".png\")\n        image = cv2.imread(path)\n        image = self.transform(image=image)['image']\n        return {'image': image, 'id': image_id}\n\n    def __len__(self):\n        return self.num_samples","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:52:25.911414Z","iopub.execute_input":"2021-10-19T00:52:25.91211Z","iopub.status.idle":"2021-10-19T00:52:25.91938Z","shell.execute_reply.started":"2021-10-19T00:52:25.912069Z","shell.execute_reply":"2021-10-19T00:52:25.918692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del dl_train, ds_train, optimizer","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:51:45.384124Z","iopub.execute_input":"2021-10-19T00:51:45.384396Z","iopub.status.idle":"2021-10-19T00:51:45.393856Z","shell.execute_reply.started":"2021-10-19T00:51:45.384351Z","shell.execute_reply":"2021-10-19T00:51:45.392912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds_test = TestCellDataset()\ndl_test = DataLoader(ds_test, batch_size=64, shuffle=False, num_workers=4, pin_memory=True)","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:52:28.78556Z","iopub.execute_input":"2021-10-19T00:52:28.786345Z","iopub.status.idle":"2021-10-19T00:52:28.829065Z","shell.execute_reply.started":"2021-10-19T00:52:28.786303Z","shell.execute_reply":"2021-10-19T00:52:28.828382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Postprocessing: separate different components of the prediction mask","metadata":{}},{"cell_type":"markdown","source":"### Utilities","metadata":{}},{"cell_type":"code","source":"def post_process(probability, threshold=0.5, min_size=300):\n    mask = cv2.threshold(probability, threshold, 1, cv2.THRESH_BINARY)[1]\n    num_component, component = cv2.connectedComponents(mask.astype(np.uint8))\n    predictions = []\n    for c in range(1, num_component):\n        p = (component == c)\n        if p.sum() > min_size:\n            a_prediction = np.zeros((520, 704), np.float32)\n            a_prediction[p] = 1\n            predictions.append(a_prediction)\n    return predictions\n\n# Stolen from: https://www.kaggle.com/arunamenon/cell-instance-segmentation-unet-eda\n# Run-length encoding stolen from https://www.kaggle.com/rakhlin/fast-run-length-encoding-python\n# Modified by me\ndef rle_encoding(x):\n    dots = np.where(x.flatten() == 1)[0]\n    run_lengths = []\n    prev = -2\n    for b in dots:\n        if (b>prev+1): run_lengths.extend((b + 1, 0))\n        run_lengths[-1] += 1\n        prev = b\n    return ' '.join(map(str, run_lengths))","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:52:30.22891Z","iopub.execute_input":"2021-10-19T00:52:30.229579Z","iopub.status.idle":"2021-10-19T00:52:30.240543Z","shell.execute_reply.started":"2021-10-19T00:52:30.229539Z","shell.execute_reply":"2021-10-19T00:52:30.239783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# I am not sure if they adapt the sample submission csv or only the test folder\n# I am using the test folders as the ground truth for the images to predict, which should be always right\n# The sample csv is ignored\npd.read_csv(SAMPLE_SUBMISSION)","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:52:34.063202Z","iopub.execute_input":"2021-10-19T00:52:34.063911Z","iopub.status.idle":"2021-10-19T00:52:34.077856Z","shell.execute_reply.started":"2021-10-19T00:52:34.063874Z","shell.execute_reply":"2021-10-19T00:52:34.076918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def check_is_run_length(mask_rle):\n    if not mask_rle:\n        return True\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    start_prev = starts[0]\n    ok = True\n    for start in starts[1:]:\n        ok = ok and start > start_prev\n        start_prev = start\n        if not ok:\n            return False\n    return True\n\ndef create_empty_submission():\n    fs = os.listdir(\"../input/sartorius-cell-instance-segmentation/test\")\n    df = pd.DataFrame([(f[:-4], \"\") for f in fs], columns=['id', 'predicted'])\n    df.to_csv(\"submission.csv\", index=False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Predict & submit","metadata":{}},{"cell_type":"code","source":"model.eval()\n\nsubmission = []\nfor i, batch in enumerate(tqdm(dl_test)):\n    preds = torch.sigmoid(model(batch['image'].cuda()))\n    preds = preds.detach().cpu().numpy()[:, 0, :, :] # (batch_size, 1, size, size) -> (batch_size, size, size)\n    for image_id, probability_mask in zip(batch['id'], preds):\n        try:\n            #if probability_mask.shape != IMAGE_RESIZE:\n            #    probability_mask = cv2.resize(probability_mask, dsize=IMAGE_RESIZE, interpolation=cv2.INTER_LINEAR)\n            probability_mask = cv2.resize(probability_mask, dsize=(704, 520), interpolation=cv2.INTER_LINEAR)\n            predictions = post_process(probability_mask)\n            for prediction in predictions:\n                #plt.imshow(prediction)\n                #plt.show()\n                try:\n                    submission.append((image_id, rle_encoding(prediction)))\n                except:\n                    print(\"Error in RL encoding\")\n        except Exception as e:\n            print(f\"Exception for img: {image_id}: {e}\")\n        \n        # Fill images with no predictions\n        image_ids = [image_id for image_id, preds in submission]\n        if image_id not in image_ids:\n            submission.append((image_id, \"\"))\n            \ndf_submission = pd.DataFrame(submission, columns=['id', 'predicted'])\ndf_submission.to_csv('submission.csv', index=False)\n\nif df_submission['predicted'].apply(check_is_run_length).mean() != 1:\n    print(\"Check run lenght failed\")\n    create_empty_submission()","metadata":{"execution":{"iopub.status.busy":"2021-10-19T00:56:28.484322Z","iopub.execute_input":"2021-10-19T00:56:28.484581Z","iopub.status.idle":"2021-10-19T00:57:12.894461Z","shell.execute_reply.started":"2021-10-19T00:56:28.484552Z","shell.execute_reply":"2021-10-19T00:57:12.893637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# _Please DO upvote_","metadata":{}}]}