{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Cell Segmentation using 3 Mask R-CNNs and Resnet18","metadata":{"execution":{"iopub.status.busy":"2021-10-23T04:12:16.649005Z","iopub.execute_input":"2021-10-23T04:12:16.6494Z","iopub.status.idle":"2021-10-23T04:12:16.726081Z","shell.execute_reply.started":"2021-10-23T04:12:16.649296Z","shell.execute_reply":"2021-10-23T04:12:16.723107Z"}}},{"cell_type":"markdown","source":"Here, the approach we have taken is to use a Mask R-CNN model (of the state-of-the-art models for instance segmentation), which is based on Faster R-CNNs, to perform instance segmentation. This model, developed by the Facebook AI Research team, is based on a *instance first* strategy instead of *segmentation first* which has been done in other similar models, and has outperformed other approaches on instance segmentation task. \n\nThe R-CNN paper can be found [here](https://arxiv.org/pdf/1703.06870.pdf).\n\nIt has mainly two phases:\n* Region Proposal Network: Here multiple Regions-of-Interest (RoI) are generated by the models.\n* Predicting class, box, and masks: From each RoI, features are extracted which are used to make the predictions.\n ","metadata":{}},{"cell_type":"markdown","source":"The approach we take here is to first use a Resnet18 Classification model to predict the `cell_type`. We also train 3 Mask R-CNNs for the different `cell_type`s and based on the previous prediction, use one of them to get the masks.\n\nThe main intuition behind this is the fact that the shape and number of instances for each type is different, and an improvement is shown with this approach getting a score of 0.283 over the previous 0.275 (while using just a single Mask R-CNN).","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"import os\nimport time\nimport random\nimport collections\nimport cv2\n\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\n\nimport torch\nimport torchvision\nfrom torchvision.transforms import ToPILImage\nfrom torchvision.transforms import functional as F\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision.models.detection.faster_rcnn import FastRCNNPredictor\nfrom torchvision.models.detection.mask_rcnn import MaskRCNNPredictor\n\nfrom torchvision.models import resnet34\nfrom torch.nn import CrossEntropyLoss\nfrom albumentations import Normalize, Resize, Compose\nfrom albumentations.pytorch import ToTensorV2\n\nimport torch.nn as nn\n\nimport wandb\n\nimport fastai\nfrom fastai.vision.all import *","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-12-16T14:17:07.972560Z","iopub.execute_input":"2021-12-16T14:17:07.972884Z","iopub.status.idle":"2021-12-16T14:17:09.583410Z","shell.execute_reply.started":"2021-12-16T14:17:07.972798Z","shell.execute_reply":"2021-12-16T14:17:09.582547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nwandb_key = user_secrets.get_secret(\"wandb_key\")\nwandb.login(key=wandb_key)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:10.267088Z","iopub.execute_input":"2021-12-16T14:17:10.267362Z","iopub.status.idle":"2021-12-16T14:17:12.315263Z","shell.execute_reply.started":"2021-12-16T14:17:10.267330Z","shell.execute_reply":"2021-12-16T14:17:12.314555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# run = wandb.init(\n#     project=\"sartorius-cell-segmentation\", \n#     entity=\"manikya\", \n#     job_type=\"train\",\n#     name=\"maskRCNN_full_14/12_v5_OneCycleSched\",\n#     reinit=True,\n#     resume=True)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:14.630428Z","iopub.execute_input":"2021-12-16T14:17:14.631004Z","iopub.status.idle":"2021-12-16T14:17:14.634542Z","shell.execute_reply.started":"2021-12-16T14:17:14.630962Z","shell.execute_reply":"2021-12-16T14:17:14.633805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SEED = 3011\n\ndef fix_seeds(seed):\n    np.random.seed(seed)\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\n    \nfix_seeds(SEED)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:15.788839Z","iopub.execute_input":"2021-12-16T14:17:15.789412Z","iopub.status.idle":"2021-12-16T14:17:15.796645Z","shell.execute_reply.started":"2021-12-16T14:17:15.789371Z","shell.execute_reply":"2021-12-16T14:17:15.795088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Configuration","metadata":{}},{"cell_type":"code","source":"TRAIN_CSV = \"../input/sartorius-cell-instance-segmentation/train.csv\"\nTRAIN_PATH = \"../input/sartorius-cell-instance-segmentation/train\"\nTEST_PATH = \"../input/sartorius-cell-instance-segmentation/test\"","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:17.140891Z","iopub.execute_input":"2021-12-16T14:17:17.141480Z","iopub.status.idle":"2021-12-16T14:17:17.145482Z","shell.execute_reply.started":"2021-12-16T14:17:17.141436Z","shell.execute_reply":"2021-12-16T14:17:17.144807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv(TRAIN_CSV)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:17.467729Z","iopub.execute_input":"2021-12-16T14:17:17.468219Z","iopub.status.idle":"2021-12-16T14:17:17.748428Z","shell.execute_reply.started":"2021-12-16T14:17:17.468183Z","shell.execute_reply":"2021-12-16T14:17:17.747671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.width.unique(), train_df.height.unique()","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:17.808795Z","iopub.execute_input":"2021-12-16T14:17:17.809005Z","iopub.status.idle":"2021-12-16T14:17:17.819918Z","shell.execute_reply.started":"2021-12-16T14:17:17.808979Z","shell.execute_reply":"2021-12-16T14:17:17.819136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"WIDTH = 704\nHEIGHT = 520\n\nTEST = False\n\nDEVICE = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n\nBATCH_SIZE = 2\n\nMOMENTUM = 0.9\nLEARNING_RATE = 0.001\nWEIGHT_DECAY = 0.0005\n\nMASK_THRESHOLD = 0.6\n\nNUM_EPOCHS = 10\n\nBOX_DETECTIONS_PER_IMG = 539\n\nMIN_SCORE = 0.59","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:18.351980Z","iopub.execute_input":"2021-12-16T14:17:18.353145Z","iopub.status.idle":"2021-12-16T14:17:18.360391Z","shell.execute_reply.started":"2021-12-16T14:17:18.353099Z","shell.execute_reply":"2021-12-16T14:17:18.359331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = pd.read_csv(TRAIN_CSV, nrows=3000 if TEST else None)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:19.224122Z","iopub.execute_input":"2021-12-16T14:17:19.224671Z","iopub.status.idle":"2021-12-16T14:17:19.505565Z","shell.execute_reply.started":"2021-12-16T14:17:19.224629Z","shell.execute_reply":"2021-12-16T14:17:19.504749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:20.307969Z","iopub.execute_input":"2021-12-16T14:17:20.308689Z","iopub.status.idle":"2021-12-16T14:17:20.325465Z","shell.execute_reply.started":"2021-12-16T14:17:20.308651Z","shell.execute_reply":"2021-12-16T14:17:20.324787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading the Classification model\n\nWe load a `fastai` learner for a resnet18 model trained to classify the cell type present in the images. This is then used later in the pipeline along with multiple R-CNNs to make mask predictions.","metadata":{}},{"cell_type":"code","source":"learn = load_learner(\"../input/cell-classification-helper-notebook/cell-classification-learner.pkl\")","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:21.312707Z","iopub.execute_input":"2021-12-16T14:17:21.314310Z","iopub.status.idle":"2021-12-16T14:17:21.366882Z","shell.execute_reply.started":"2021-12-16T14:17:21.314259Z","shell.execute_reply":"2021-12-16T14:17:21.366065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Utilities\n\n\n### Transformations\n\nSome of the transformations have been referred from [mask r cnn utils](https://www.kaggle.com/abhishek/maskrcnn-utils) as it was not possible to direclty use the implementation for the transformaions from `torchvision`. The Instance Segmentation task here requires us to also transform the target bounding boxes along with the image, and thus custom transformations are required.\n\nHere, all the transformations all take in `target` as well.","metadata":{}},{"cell_type":"markdown","source":"We referred to [this mask r cnn utils](https://www.kaggle.com/abhishek/maskrcnn-utils?select=transforms.py) package for the transformations.","metadata":{}},{"cell_type":"code","source":"class Compose:\n    def __init__(self, transforms):\n        self.transforms = transforms\n\n    def __call__(self, image, target):\n        for t in self.transforms:\n            image, target = t(image, target)\n        return image, target\n\nclass VerticalFlip:\n    def __init__(self, prob):\n        self.prob = prob\n\n    def __call__(self, image, target):\n        if random.random() < self.prob:\n            height, width = image.shape[-2:]\n            image = image.flip(-2)\n            bbox = target[\"boxes\"]\n            bbox[:, [1, 3]] = height - bbox[:, [3, 1]]\n            target[\"boxes\"] = bbox\n            target[\"masks\"] = target[\"masks\"].flip(-2)\n        return image, target\n\nclass HorizontalFlip:\n    def __init__(self, prob):\n        self.prob = prob\n\n    def __call__(self, image, target):\n        if random.random() < self.prob:\n            height, width = image.shape[-2:]\n            image = image.flip(-1)\n            bbox = target[\"boxes\"]\n            bbox[:, [0, 2]] = width - bbox[:, [2, 0]]\n            target[\"boxes\"] = bbox\n            target[\"masks\"] = target[\"masks\"].flip(-1)\n        return image, target\n\nclass ToTensorNew:\n    def __call__(self, image, target):\n        image = torchvision.transforms.functional.to_tensor(image)\n        return image, target\n    \n\ndef get_transform(train):\n    transforms = [ToTensorNew()]\n\n    if train: \n        transforms.append(HorizontalFlip(0.5))\n        transforms.append(VerticalFlip(0.5))\n\n    return Compose(transforms)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:23.097757Z","iopub.execute_input":"2021-12-16T14:17:23.098527Z","iopub.status.idle":"2021-12-16T14:17:23.112473Z","shell.execute_reply.started":"2021-12-16T14:17:23.098486Z","shell.execute_reply":"2021-12-16T14:17:23.110953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"torchvision.transforms.functional.to_tensor","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:23.608736Z","iopub.execute_input":"2021-12-16T14:17:23.609271Z","iopub.status.idle":"2021-12-16T14:17:23.614771Z","shell.execute_reply.started":"2021-12-16T14:17:23.609233Z","shell.execute_reply":"2021-12-16T14:17:23.613980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def runlen_decoding(mask_rl, shape, color=1):\n    '''\n    mask_rl: run-length as string formated (start length)\n    shape: (height,width) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n    '''\n    s = mask_rl.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0] * shape[1], dtype=np.float32)\n    for lo, hi in zip(starts, ends):\n        img[lo : hi] = color\n    return img.reshape(shape)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:24.063748Z","iopub.execute_input":"2021-12-16T14:17:24.064603Z","iopub.status.idle":"2021-12-16T14:17:24.071774Z","shell.execute_reply.started":"2021-12-16T14:17:24.064554Z","shell.execute_reply":"2021-12-16T14:17:24.070948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.iloc[0].annotation","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:24.546224Z","iopub.execute_input":"2021-12-16T14:17:24.546710Z","iopub.status.idle":"2021-12-16T14:17:24.552438Z","shell.execute_reply.started":"2021-12-16T14:17:24.546672Z","shell.execute_reply":"2021-12-16T14:17:24.551647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"runlen_decoding(train_df.iloc[0].annotation, (HEIGHT, WIDTH)).shape","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:24.922040Z","iopub.execute_input":"2021-12-16T14:17:24.922736Z","iopub.status.idle":"2021-12-16T14:17:24.929856Z","shell.execute_reply.started":"2021-12-16T14:17:24.922687Z","shell.execute_reply":"2021-12-16T14:17:24.929079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training Dataset and DataLoader","metadata":{}},{"cell_type":"code","source":"temp = train_df.groupby('id')['annotation', 'cell_type'].agg(lambda x: list(x)).reset_index()","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:25.816683Z","iopub.execute_input":"2021-12-16T14:17:25.817153Z","iopub.status.idle":"2021-12-16T14:17:25.887278Z","shell.execute_reply.started":"2021-12-16T14:17:25.817108Z","shell.execute_reply":"2021-12-16T14:17:25.886473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"((np.unique(temp.cell_type[0]), len(temp.iloc[0].annotation)),\n(np.unique(temp.cell_type[1]), len(temp.iloc[1].annotation)),\n(np.unique(temp.cell_type[2]), len(temp.iloc[2].annotation)))","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:26.213115Z","iopub.execute_input":"2021-12-16T14:17:26.213788Z","iopub.status.idle":"2021-12-16T14:17:26.225394Z","shell.execute_reply.started":"2021-12-16T14:17:26.213739Z","shell.execute_reply":"2021-12-16T14:17:26.224432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For defining the custom dataset, `torchvision`'s tutorial notebook (available [here](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#)) for the PennFudan dataset was helpful.","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv(TRAIN_CSV, nrows=3000 if TEST else None)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:27.006550Z","iopub.execute_input":"2021-12-16T14:17:27.007291Z","iopub.status.idle":"2021-12-16T14:17:27.296579Z","shell.execute_reply.started":"2021-12-16T14:17:27.007246Z","shell.execute_reply":"2021-12-16T14:17:27.295712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class NeuronalCellDataset(Dataset):\n    def __init__(self, image_dir, df, transforms=None):\n        self.transforms = transforms\n        self.image_dir = image_dir\n        self.df = df\n        \n        self.height = HEIGHT\n        self.width = WIDTH\n        \n        self.image_info = collections.defaultdict(dict)\n        temp_df = self.df.groupby('id')['annotation'].agg(lambda x: list(x)).reset_index()\n        for index, row in temp_df.iterrows():\n            self.image_info[index] = {\n                'image_id': row['id'],\n                'image_path': os.path.join(self.image_dir, row['id'] + '.png'),\n                'annotations': row[\"annotation\"]\n                }\n\n    def __getitem__(self, idx):\n        \n        img_path = self.image_info[idx][\"image_path\"]\n        img = Image.open(img_path).convert(\"RGB\")\n\n        info = self.image_info[idx]\n\n        n_objects = len(info['annotations'])\n        masks = np.zeros((n_objects, self.height, self.width), dtype=np.uint8)\n        boxes = []\n        \n        for i, annotation in enumerate(info['annotations']):\n            a_mask = runlen_decoding(annotation, (HEIGHT, WIDTH))\n            a_mask = Image.fromarray(a_mask)\n            \n            a_mask = np.array(a_mask) > 0\n            masks[i, :, :] = a_mask\n            \n            pos = np.where(a_mask)\n            xmin = np.min(pos[1])\n            xmax = np.max(pos[1])\n            ymin = np.min(pos[0])\n            ymax = np.max(pos[0])\n            \n            boxes.append([xmin, ymin, xmax, ymax])\n\n        \n        boxes = torch.as_tensor(boxes, dtype=torch.float32)\n        masks = torch.as_tensor(masks, dtype=torch.uint8)\n        labels = torch.ones((n_objects,), dtype=torch.int64) # As there is only 1 class.\n        \n        image_id = torch.tensor([idx])\n        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])\n        iscrowd = torch.zeros((n_objects,), dtype=torch.int64)\n\n        # Required target for the Mask R-CNN model\n        target = {\n            'boxes': boxes,\n            'labels': labels,\n            'masks': masks,\n            'image_id': image_id,\n            'area': area,\n            'iscrowd': iscrowd\n        }\n\n        if self.transforms is not None:\n            img, target = self.transforms(img, target)\n\n        return img, target\n\n    def __len__(self):\n        return len(self.image_info)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:27.385871Z","iopub.execute_input":"2021-12-16T14:17:27.386308Z","iopub.status.idle":"2021-12-16T14:17:27.404264Z","shell.execute_reply.started":"2021-12-16T14:17:27.386275Z","shell.execute_reply":"2021-12-16T14:17:27.403497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds_train = NeuronalCellDataset(TRAIN_PATH, df_train, transforms=get_transform(train=True))\ndl_train = DataLoader(ds_train, batch_size=BATCH_SIZE, shuffle=True, \n                      num_workers=2, collate_fn=lambda x: tuple(zip(*x)))","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:27.782222Z","iopub.execute_input":"2021-12-16T14:17:27.782581Z","iopub.status.idle":"2021-12-16T14:17:27.856617Z","shell.execute_reply.started":"2021-12-16T14:17:27.782536Z","shell.execute_reply":"2021-12-16T14:17:27.855940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds_train[0]","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:28.262713Z","iopub.execute_input":"2021-12-16T14:17:28.263000Z","iopub.status.idle":"2021-12-16T14:17:29.094284Z","shell.execute_reply.started":"2021-12-16T14:17:28.262968Z","shell.execute_reply":"2021-12-16T14:17:29.092580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.shape","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:29.096869Z","iopub.execute_input":"2021-12-16T14:17:29.097095Z","iopub.status.idle":"2021-12-16T14:17:29.102031Z","shell.execute_reply.started":"2021-12-16T14:17:29.097069Z","shell.execute_reply":"2021-12-16T14:17:29.101332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train_shsy5y = df_train[df_train.cell_type=='shsy5y'].reset_index()\nds_train_shsy5y = NeuronalCellDataset(TRAIN_PATH, df_train_shsy5y, transforms=get_transform(train=True))\ndl_train_shsy5y = DataLoader(ds_train_shsy5y, batch_size=BATCH_SIZE, shuffle=True, \n                      num_workers=2, collate_fn=lambda x: tuple(zip(*x)))","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:29.309546Z","iopub.execute_input":"2021-12-16T14:17:29.310075Z","iopub.status.idle":"2021-12-16T14:17:29.408631Z","shell.execute_reply.started":"2021-12-16T14:17:29.310039Z","shell.execute_reply":"2021-12-16T14:17:29.407338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train_cort = df_train[df_train.cell_type=='cort'].reset_index()\nds_train_cort = NeuronalCellDataset(TRAIN_PATH, df_train_cort, transforms=get_transform(train=True))\ndl_train_cort = DataLoader(ds_train_cort, batch_size=BATCH_SIZE, shuffle=True, \n                      num_workers=2, collate_fn=lambda x: tuple(zip(*x)))","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:30.046128Z","iopub.execute_input":"2021-12-16T14:17:30.047031Z","iopub.status.idle":"2021-12-16T14:17:30.101225Z","shell.execute_reply.started":"2021-12-16T14:17:30.046957Z","shell.execute_reply":"2021-12-16T14:17:30.100518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train_astro = df_train[df_train.cell_type=='astro'].reset_index()\nds_train_astro = NeuronalCellDataset(TRAIN_PATH, df_train_astro, transforms=get_transform(train=True))\ndl_train_astro = DataLoader(ds_train_astro, batch_size=BATCH_SIZE, shuffle=True, \n                      num_workers=2, collate_fn=lambda x: tuple(zip(*x)))","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:30.424980Z","iopub.execute_input":"2021-12-16T14:17:30.425491Z","iopub.status.idle":"2021-12-16T14:17:30.474613Z","shell.execute_reply.started":"2021-12-16T14:17:30.425454Z","shell.execute_reply":"2021-12-16T14:17:30.473897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train_cort.shape, df_train_shsy5y.shape, df_train_astro.shape","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:30.851817Z","iopub.execute_input":"2021-12-16T14:17:30.852351Z","iopub.status.idle":"2021-12-16T14:17:30.859698Z","shell.execute_reply.started":"2021-12-16T14:17:30.852304Z","shell.execute_reply":"2021-12-16T14:17:30.858525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train loop","metadata":{}},{"cell_type":"markdown","source":"## Model","metadata":{}},{"cell_type":"code","source":"# Override pytorch checkpoint with an \"offline\" version of the file\n!mkdir -p /root/.cache/torch/hub/checkpoints/\n!cp ../input/cocopre/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth /root/.cache/torch/hub/checkpoints/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:33.505945Z","iopub.execute_input":"2021-12-16T14:17:33.506370Z","iopub.status.idle":"2021-12-16T14:17:35.306919Z","shell.execute_reply.started":"2021-12-16T14:17:33.506338Z","shell.execute_reply":"2021-12-16T14:17:35.305965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_model():\n    \n    # 1 class for the background, and one for the cell type. Here, each image will only have neurons of one type (cell_type).\n    NUM_CLASSES = 2 \n    \n    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True,\n                                                                  box_detections_per_img=BOX_DETECTIONS_PER_IMG)\n\n\n    in_features = model.roi_heads.box_predictor.cls_score.in_features\n\n    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, NUM_CLASSES)\n\n\n    in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels\n    hidden_layer = 256\n\n    model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask, hidden_layer, NUM_CLASSES)\n    return model","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:35.309189Z","iopub.execute_input":"2021-12-16T14:17:35.309486Z","iopub.status.idle":"2021-12-16T14:17:35.318050Z","shell.execute_reply.started":"2021-12-16T14:17:35.309447Z","shell.execute_reply":"2021-12-16T14:17:35.314988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ['shsy5y', 'astro', 'cort']","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:35.429491Z","iopub.execute_input":"2021-12-16T14:17:35.429716Z","iopub.status.idle":"2021-12-16T14:17:35.954851Z","shell.execute_reply.started":"2021-12-16T14:17:35.429685Z","shell.execute_reply":"2021-12-16T14:17:35.953802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# model_0_shsy5y = get_model()\n# model_0_shsy5y.to(DEVICE);\n\n# model_1_astro = get_model()\n# model_1_astro.to(DEVICE);\n\n# model_2_cort = get_model()\n# model_2_cort.to(DEVICE);","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:35.956541Z","iopub.execute_input":"2021-12-16T14:17:35.956838Z","iopub.status.idle":"2021-12-16T14:17:35.963632Z","shell.execute_reply.started":"2021-12-16T14:17:35.956799Z","shell.execute_reply":"2021-12-16T14:17:35.962829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_0_shsy5y = get_model()\nmodel_0_shsy5y.to(DEVICE);\nmodel_0_shsy5y.load_state_dict(torch.load(\"../input/r-cnn-models-for-cell-segmentation/model_shsy5y-e10.bin\", map_location=DEVICE))\nmodel_0_shsy5y.eval()\n\nmodel_1_astro = get_model()\nmodel_1_astro.to(DEVICE);\nmodel_1_astro.load_state_dict(torch.load(\"../input/r-cnn-models-for-cell-segmentation/model_astro-e10.bin\", map_location=DEVICE))\nmodel_1_astro.eval()\n\nmodel_2_cort = get_model()\nmodel_2_cort.to(DEVICE);\nmodel_2_cort.load_state_dict(torch.load(\"../input/r-cnn-models-for-cell-segmentation/model_cort-e10.bin\", map_location=DEVICE))\nmodel_2_cort.eval();","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:38.366313Z","iopub.execute_input":"2021-12-16T14:17:38.366805Z","iopub.status.idle":"2021-12-16T14:17:43.039257Z","shell.execute_reply.started":"2021-12-16T14:17:38.366767Z","shell.execute_reply":"2021-12-16T14:17:43.038427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp_count = 0\nfor i, param in enumerate(model_2_cort.parameters()):\n    if (not param.requires_grad):\n        temp_count+=1\n        print(i)\nprint(f'Count of frozen layers is {temp_count}')","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:43.040868Z","iopub.execute_input":"2021-12-16T14:17:43.041142Z","iopub.status.idle":"2021-12-16T14:17:43.052018Z","shell.execute_reply.started":"2021-12-16T14:17:43.041107Z","shell.execute_reply":"2021-12-16T14:17:43.051231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for param in model_2_cort.parameters():\n    param.requires_grad = True\n    \nmodel_2_cort.train();\n\nfor param in model_0_shsy5y.parameters():\n    param.requires_grad = True\n    \nmodel_0_shsy5y.train();\n\nfor param in model_1_astro.parameters():\n    param.requires_grad = True\n    \nmodel_1_astro.train();","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:43.053504Z","iopub.execute_input":"2021-12-16T14:17:43.053970Z","iopub.status.idle":"2021-12-16T14:17:43.065672Z","shell.execute_reply.started":"2021-12-16T14:17:43.053907Z","shell.execute_reply":"2021-12-16T14:17:43.065008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The model has multiple heads for predicting the bounding boxes, classification, and instance masks.","metadata":{}},{"cell_type":"code","source":"model_1_astro.roi_heads","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:43.066948Z","iopub.execute_input":"2021-12-16T14:17:43.069130Z","iopub.status.idle":"2021-12-16T14:17:43.076553Z","shell.execute_reply.started":"2021-12-16T14:17:43.069092Z","shell.execute_reply":"2021-12-16T14:17:43.075668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training","metadata":{}},{"cell_type":"code","source":"# wandb.config = {\n#     \"learning_rate\": LEARNING_RATE,\n#     \"epochs\": NUM_EPOCHS,\n#     \"batch_size\": BATCH_SIZE,\n#     \"momentum\": MOMENTUM,\n#     \"learning_rate\": LEARNING_RATE,\n#     \"weight_decay\": WEIGHT_DECAY,\n#     \"num_box_preds\": BOX_DETECTIONS_PER_IMG,\n#     \"seed\": SEED,\n#     \"scheduler\": \"OneCyclePolicy\"\n# }","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:43.078063Z","iopub.execute_input":"2021-12-16T14:17:43.078404Z","iopub.status.idle":"2021-12-16T14:17:43.083979Z","shell.execute_reply.started":"2021-12-16T14:17:43.078300Z","shell.execute_reply":"2021-12-16T14:17:43.083116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"params_1_astro = torch.nn.ParameterList([p for p in model_1_astro.parameters() if p.requires_grad])\noptimizer_1_astro = torch.optim.SGD(params_1_astro, lr=LEARNING_RATE, momentum=MOMENTUM, weight_decay=WEIGHT_DECAY)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"params_0_shsy5y = torch.nn.ParameterList([p for p in model_0_shsy5y.parameters() if p.requires_grad])\noptimizer_0_shsy5y = torch.optim.SGD(params_0_shsy5y, lr=LEARNING_RATE, momentum=MOMENTUM, weight_decay=WEIGHT_DECAY)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"params_2_cort = torch.nn.ParameterList([p for p in model_2_cort.parameters() if p.requires_grad])\noptimizer_2_cort = torch.optim.SGD(params_2_cort, lr=LEARNING_RATE, momentum=MOMENTUM, weight_decay=WEIGHT_DECAY)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_batches_cort = len(dl_train_cort)\nlr_scheduler_cort = torch.optim.lr_scheduler.OneCycleLR(optimizer_2_cort, max_lr=0.01, steps_per_epoch=num_batchs_cort, epochs=NUM_EPOCHS)\n\nfor epoch in range(1, NUM_EPOCHS + 1):\n    print(f\"Starting epoch {epoch} of {NUM_EPOCHS}\")\n    \n    time_start = time.time()\n    loss_accum = 0.0\n    loss_mask_accum = 0.0\n    \n    for batch_idx, (images, targets) in enumerate(dl_train_cort, 1):\n    \n        # Predict\n        images = list(image.to(DEVICE) for image in images)\n        targets = [{k: v.to(DEVICE) for k, v in t.items()} for t in targets]\n\n        loss_dict = model_2_cort(images, targets)\n        loss = sum(loss for loss in loss_dict.values())\n        \n        # Backprop\n        optimizer_2_cort.zero_grad()\n        loss.backward()\n        optimizer_2_cort.step()\n        lr_scheduler_cort.step()\n        \n        # Logging\n        loss_mask = loss_dict['loss_mask'].item()\n        loss_accum += loss.item()\n        loss_mask_accum += loss_mask\n        \n        if batch_idx % 50 == 0:\n            print(f\"    [Batch {batch_idx:3d} / {num_batches_cort:3d}] Batch train loss: {loss.item():7.3f}. Mask-only loss: {loss_mask:7.3f}\")\n            \n#         wandb.log({\"batch_train_loss\": loss.item(), \"mask_loss\":loss_mask})\n#         wandb.watch(model)\n    \n    # Train losses\n    train_loss = loss_accum / num_batches_cort\n    train_loss_mask = loss_mask_accum / num_batches_cort\n    \n    elapsed = time.time() - time_start\n    \n    torch.save(model_2_cort.state_dict(), f\"/kaggle/working/model_cort-e{epoch}.bin\")\n    \n    prefix = f\"[Epoch {epoch:2d} / {NUM_EPOCHS:2d}]\"\n    print(f\"{prefix} Train mask-only loss: {train_loss_mask:7.3f}\")\n    print(f\"{prefix} Train loss: {train_loss:7.3f}. [{elapsed:.0f} secs]\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_batches_shsy5y = len(dl_train_shsy5y)\nlr_scheduler_shsy5y = torch.optim.lr_scheduler.OneCycleLR(optimizer_0_shsy5y, max_lr=0.01, steps_per_epoch=num_batches_shsy5y, epochs=NUM_EPOCHS)\n\nfor epoch in range(1, NUM_EPOCHS + 1):\n    print(f\"Starting epoch {epoch} of {NUM_EPOCHS}\")\n    \n    time_start = time.time()\n    loss_accum = 0.0\n    loss_mask_accum = 0.0\n    \n    for batch_idx, (images, targets) in enumerate(dl_train_shsy5y, 1):\n    \n        # Predict\n        images = list(image.to(DEVICE) for image in images)\n        targets = [{k: v.to(DEVICE) for k, v in t.items()} for t in targets]\n\n        loss_dict = model_0_shsy5y(images, targets)\n        loss = sum(loss for loss in loss_dict.values())\n        \n        # Backprop\n        optimizer_0_shsy5y.zero_grad()\n        loss.backward()\n        optimizer_0_shsy5y.step()\n        lr_scheduler_shsy5y.step()\n        \n        # Logging\n        loss_mask = loss_dict['loss_mask'].item()\n        loss_accum += loss.item()\n        loss_mask_accum += loss_mask\n        \n        if batch_idx % 50 == 0:\n            print(f\"    [Batch {batch_idx:3d} / {num_batches_shsy5y:3d}] Batch train loss: {loss.item():7.3f}. Mask-only loss: {loss_mask:7.3f}\")\n            \n#         wandb.log({\"batch_train_loss\": loss.item(), \"mask_loss\":loss_mask})\n#         wandb.watch(model)\n    \n    # Train losses\n    train_loss = loss_accum / num_batches_shsy5y\n    train_loss_mask = loss_mask_accum / num_batches_shsy5y\n    \n    elapsed = time.time() - time_start\n    \n    torch.save(model_0_shsy5y.state_dict(), f\"/kaggle/working/model_shsy5y-e{epoch}.bin\")\n    \n    prefix = f\"[Epoch {epoch:2d} / {NUM_EPOCHS:2d}]\"\n    print(f\"{prefix} Train mask-only loss: {train_loss_mask:7.3f}\")\n    print(f\"{prefix} Train loss: {train_loss:7.3f}. [{elapsed:.0f} secs]\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_batches_astro = len(dl_train_astro)\nlr_scheduler_astro = torch.optim.lr_scheduler.OneCycleLR(optimizer_1_astro, max_lr=0.01, steps_per_epoch=num_batches_astro, epochs=NUM_EPOCHS)\n\nfor epoch in range(1, NUM_EPOCHS + 1):\n    print(f\"Starting epoch {epoch} of {NUM_EPOCHS}\")\n    \n    time_start = time.time()\n    loss_accum = 0.0\n    loss_mask_accum = 0.0\n    \n    for batch_idx, (images, targets) in enumerate(dl_train_astro, 1):\n    \n        # Predict\n        images = list(image.to(DEVICE) for image in images)\n        targets = [{k: v.to(DEVICE) for k, v in t.items()} for t in targets]\n\n        loss_dict = model_1_astro(images, targets)\n        loss = sum(loss for loss in loss_dict.values())\n        \n        # Backprop\n        optimizer_1_astro.zero_grad()\n        loss.backward()\n        optimizer_1_astro.step()\n        lr_scheduler_astro.step()\n        \n        # Logging\n        loss_mask = loss_dict['loss_mask'].item()\n        loss_accum += loss.item()\n        loss_mask_accum += loss_mask\n        \n        if batch_idx % 50 == 0:\n            print(f\"    [Batch {batch_idx:3d} / {num_batches_astro:3d}] Batch train loss: {loss.item():7.3f}. Mask-only loss: {loss_mask:7.3f}\")\n            \n#         wandb.log({\"batch_train_loss\": loss.item(), \"mask_loss\":loss_mask})\n#         wandb.watch(model)\n    \n    # Train losses\n    train_loss = loss_accum / num_batches_astro\n    train_loss_mask = loss_mask_accum / num_batches_astro\n    \n    elapsed = time.time() - time_start\n    \n    torch.save(model_1_astro.state_dict(), f\"/kaggle/working/model_astro-e{epoch}.bin\")\n    \n    prefix = f\"[Epoch {epoch:2d} / {NUM_EPOCHS:2d}]\"\n    print(f\"{prefix} Train mask-only loss: {train_loss_mask:7.3f}\")\n    print(f\"{prefix} Train loss: {train_loss:7.3f}. [{elapsed:.0f} secs]\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# run.finish()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img, targets = ds_train[2]\nmasks = np.zeros((HEIGHT, WIDTH))\nfor mask in targets['masks']:\n    masks = np.logical_or(masks, mask)\nplt.imshow(img.numpy().transpose((1,2,0)))\nplt.imshow(masks, alpha=0.3)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:54.481362Z","iopub.execute_input":"2021-12-16T14:17:54.482183Z","iopub.status.idle":"2021-12-16T14:17:54.990982Z","shell.execute_reply.started":"2021-12-16T14:17:54.482138Z","shell.execute_reply":"2021-12-16T14:17:54.990263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.predict(\"../input/sartorius-cell-instance-segmentation/train/0140b3c8f445.png\")","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:58.677155Z","iopub.execute_input":"2021-12-16T14:17:58.677427Z","iopub.status.idle":"2021-12-16T14:17:59.098890Z","shell.execute_reply.started":"2021-12-16T14:17:58.677396Z","shell.execute_reply":"2021-12-16T14:17:59.098068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.predict(Path(TRAIN_PATH)/'0140b3c8f445.png')[0]","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:17:59.978894Z","iopub.execute_input":"2021-12-16T14:17:59.979314Z","iopub.status.idle":"2021-12-16T14:18:00.367254Z","shell.execute_reply.started":"2021-12-16T14:17:59.979282Z","shell.execute_reply":"2021-12-16T14:18:00.366326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[df_train.cell_type=='astro'].head(3)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:18:00.397880Z","iopub.execute_input":"2021-12-16T14:18:00.398101Z","iopub.status.idle":"2021-12-16T14:18:00.427364Z","shell.execute_reply.started":"2021-12-16T14:18:00.398076Z","shell.execute_reply":"2021-12-16T14:18:00.426704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.model(torch.unsqueeze(img, dim=0))","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:18:00.843667Z","iopub.execute_input":"2021-12-16T14:18:00.844051Z","iopub.status.idle":"2021-12-16T14:18:01.195443Z","shell.execute_reply.started":"2021-12-16T14:18:00.844018Z","shell.execute_reply":"2021-12-16T14:18:01.194706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"models_dict={\n    'astro': model_1_astro,\n    'cort': model_2_cort,\n    'shsy5y': model_0_shsy5y}","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:18:01.365162Z","iopub.execute_input":"2021-12-16T14:18:01.365567Z","iopub.status.idle":"2021-12-16T14:18:01.370420Z","shell.execute_reply.started":"2021-12-16T14:18:01.365531Z","shell.execute_reply":"2021-12-16T14:18:01.369694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds_train.image_info[2][\"image_path\"]","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:18:02.515665Z","iopub.execute_input":"2021-12-16T14:18:02.516168Z","iopub.status.idle":"2021-12-16T14:18:02.524127Z","shell.execute_reply.started":"2021-12-16T14:18:02.516114Z","shell.execute_reply":"2021-12-16T14:18:02.523098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Analyze prediction results for train set","metadata":{}},{"cell_type":"code","source":"def analyze_train_sample(models_dict, learn, ds_train, sample_index):\n    \n    img, targets = ds_train[sample_index]\n    plt.imshow(img.numpy().transpose((1,2,0)))\n    plt.title(\"Image\")\n    plt.axis('off')\n    plt.show()\n    \n    masks = np.zeros((HEIGHT, WIDTH))\n    for mask in targets['masks']:\n        masks = np.logical_or(masks, mask)\n    plt.imshow(img.numpy().transpose((1,2,0)))\n    plt.imshow(masks, alpha=0.3)\n    plt.title(\"Ground truth\")\n    plt.axis('off')\n    plt.show()\n    \n    for model in models_dict.values():\n        model.eval()\n    cell_type = learn.predict(ds_train.image_info[sample_index][\"image_path\"])[0]\n    with torch.no_grad():\n        preds = models_dict[cell_type]([img.to(DEVICE)])[0]\n\n    plt.imshow(img.cpu().numpy().transpose((1,2,0)))\n    all_preds_masks = np.zeros((HEIGHT, WIDTH))\n    for mask in preds['masks'].cpu().detach().numpy():\n        all_preds_masks = np.logical_or(all_preds_masks, mask[0] > 0.8)\n    plt.imshow(all_preds_masks, alpha=0.4)\n    plt.title(f\"Predictions using {cell_type} model\")\n    plt.axis('off')\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:18:04.961805Z","iopub.execute_input":"2021-12-16T14:18:04.962094Z","iopub.status.idle":"2021-12-16T14:18:04.973335Z","shell.execute_reply.started":"2021-12-16T14:18:04.962059Z","shell.execute_reply":"2021-12-16T14:18:04.972609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"analyze_train_sample(models_dict, learn, ds_train, 200)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:18:07.072289Z","iopub.execute_input":"2021-12-16T14:18:07.072580Z","iopub.status.idle":"2021-12-16T14:18:10.621834Z","shell.execute_reply.started":"2021-12-16T14:18:07.072550Z","shell.execute_reply":"2021-12-16T14:18:10.621171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"analyze_train_sample(models_dict, learn, ds_train, 399)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:18:12.913901Z","iopub.execute_input":"2021-12-16T14:18:12.914640Z","iopub.status.idle":"2021-12-16T14:18:14.937633Z","shell.execute_reply.started":"2021-12-16T14:18:12.914599Z","shell.execute_reply":"2021-12-16T14:18:14.936881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"analyze_train_sample(models_dict, learn, ds_train, 395)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T14:18:14.939270Z","iopub.execute_input":"2021-12-16T14:18:14.940035Z","iopub.status.idle":"2021-12-16T14:18:17.855642Z","shell.execute_reply.started":"2021-12-16T14:18:14.939981Z","shell.execute_reply":"2021-12-16T14:18:17.854902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## References and Important Links\n\nFor creating and working with the R-CNN model, `torchvision`'s object detection tutorial notebook (available [here](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#)) was particularly helpful.  \nFor working with instance segmentation task, we referred to an entry from a previous kaggle challenge on segmenting fashion images (available [here](https://www.kaggle.com/abhishek/mask-rcnn-using-torchvision-0-17/notebook)).  \nWe also used pytorch utils for visualizing the masks and understanding the R-CNN output. [Check them here](https://pytorch.org/vision/stable/auto_examples/plot_visualization_utils.html#instance-seg-output).  \nSome helper functions and the Mask R-CNN model were referred from [this](https://www.kaggle.com/julian3833/sartorius-starter-torch-mask-r-cnn-lb-0-273) notebook as well.  \nThe various reference scripts ([avaiable here](https://github.com/pytorch/vision/tree/main/references/detection)) in torchvision were also helpful.\n\nThe Mask R-CNN model was first introduced in [this](https://arxiv.org/abs/1703.06870) paper.\nModel used for classification was trained in [this notebook](https://www.kaggle.com/manikyab/cell-classification-helper-notebook). Interpretation from this notebook has also been done and can be found [here](https://www.kaggle.com/manikyab/cell-classification-captum-interpretation).\n\nThe weights for the model trained can be found [here](https://www.kaggle.com/manikyab/r-cnn-models-for-cell-segmentation).","metadata":{"execution":{"iopub.status.busy":"2021-12-13T08:34:42.182831Z","iopub.execute_input":"2021-12-13T08:34:42.183562Z","iopub.status.idle":"2021-12-13T08:34:42.209676Z","shell.execute_reply.started":"2021-12-13T08:34:42.183456Z","shell.execute_reply":"2021-12-13T08:34:42.208785Z"}}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}