{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🦠 Sartorius - Torch - Classifier + Mask R-CNN\n\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/30201/logos/header.png)\n\n\n### Main insight: the `cell_type` of the image determines a very different number of individuals to predict. We can asses the `cell_type` with a classifier and use that information downstream.\n\n__In this notebook, we make the Mask R-CNN generate a lot of predictions, and we decide how many to keep based on the `cell_type` reported by the classifier. \nWe also apply a mask refinement method using information from the train set.__\n\n\n\n__This solution relies on two models:__\n1. __A classifier assessing the `cell_type` of a given image from [🦠 Sartorius - Resnet34 Classifier](https://www.kaggle.com/julian3833/sartorius-resnet34-classifier) (Trained using semi supervised data, 100% validation accuracy)__\n2. __The Mask R-CNN predicting the individual masks from [🦠 Sartorius - Starter Torch Mask R-CNN [LB=0.273]](https://www.kaggle.com/julian3833/sartorius-starter-torch-mask-r-cnn-lb-0-205)__\n\n\nAlthough the idea was interesting, it doesn't add much value to the original solution.\nPossibly the `Classifier + Something` idea has better applications than this one an I wish someone else can spot it.\n\n## Please _DO_ upvote!","metadata":{}},{"cell_type":"markdown","source":"# Imports and constants","metadata":{}},{"cell_type":"code","source":"import os\nimport time\nimport random\nimport collections\n\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\nimport matplotlib.pyplot as plt\n\nimport cv2\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\n\nimport torch\nimport torchvision\nfrom torchvision.transforms import ToPILImage\nfrom torchvision.transforms import functional as F\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision.models.detection.faster_rcnn import FastRCNNPredictor\nfrom torchvision.models.detection.mask_rcnn import MaskRCNNPredictor\n\ndef fix_all_seeds(seed):\n    np.random.seed(seed)\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\n\n# Fix randomness\nfix_all_seeds(2021)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-10-24T23:39:03.844864Z","iopub.execute_input":"2021-10-24T23:39:03.845431Z","iopub.status.idle":"2021-10-24T23:39:03.855599Z","shell.execute_reply.started":"2021-10-24T23:39:03.845392Z","shell.execute_reply":"2021-10-24T23:39:03.854675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TRAIN_CSV = \"../input/sartorius-cell-instance-segmentation/train.csv\"\nTEST_PATH = \"../input/sartorius-cell-instance-segmentation/test\"\n\nCLASSIFIER_CHK = \"../input/sartorius-resnet-34-classifier-finetuned/resnet34-finetuned.bin\"\nMASK_RCNN_CHK = \"../input/sartorius-starter-torch-mask-r-cnn/pytorch_model.bin\"\n\nCELL_TYPES  = {0: 'shsy5y', 1: 'astro', 2: 'cort'}\n\nDEVICE = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n\nRESNET_MEAN = (0.485, 0.456, 0.406)\nRESNET_STD = (0.229, 0.224, 0.225)\n\n\n# The maximum possible amount of predictions\n# 539 is the 90% percentile of the cell_type with more instances per image\nBOX_DETECTIONS_PER_IMG = 600","metadata":{"execution":{"iopub.status.busy":"2021-10-24T23:39:03.857426Z","iopub.execute_input":"2021-10-24T23:39:03.857666Z","iopub.status.idle":"2021-10-24T23:39:03.876796Z","shell.execute_reply.started":"2021-10-24T23:39:03.857638Z","shell.execute_reply":"2021-10-24T23:39:03.876145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load dataframe","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv(TRAIN_CSV)","metadata":{"execution":{"iopub.status.busy":"2021-10-24T23:39:03.878129Z","iopub.execute_input":"2021-10-24T23:39:03.878911Z","iopub.status.idle":"2021-10-24T23:39:04.217406Z","shell.execute_reply.started":"2021-10-24T23:39:03.878874Z","shell.execute_reply":"2021-10-24T23:39:04.216528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Simple statistics: number of instances per image per `cell_type` \n\nWe will use the values from this analysis to decide the number of predicted individuals to generate for each image","metadata":{}},{"cell_type":"code","source":"df_instances = df_train.groupby(['id']).agg({'annotation': 'count', 'cell_type': 'first'})\ndf_instances = df_instances.groupby(\"cell_type\")[['annotation']]\\\n                               .describe(percentiles=[0.1, 0.25, 0.75, 0.8, 0.85, 0.9, 0.95, 0.99]).astype(int)\\\n                               .T.droplevel(level=0).T.drop(['count', '50%', 'std'], axis=1)\ndf_instances","metadata":{"execution":{"iopub.status.busy":"2021-10-25T00:17:18.736096Z","iopub.execute_input":"2021-10-25T00:17:18.736389Z","iopub.status.idle":"2021-10-25T00:17:18.794163Z","shell.execute_reply.started":"2021-10-25T00:17:18.736361Z","shell.execute_reply":"2021-10-25T00:17:18.793293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Trying with different strategies\ndf_instances['90%'].to_dict()","metadata":{"execution":{"iopub.status.busy":"2021-10-24T23:39:04.280454Z","iopub.execute_input":"2021-10-24T23:39:04.281326Z","iopub.status.idle":"2021-10-24T23:39:04.287393Z","shell.execute_reply.started":"2021-10-24T23:39:04.281272Z","shell.execute_reply":"2021-10-24T23:39:04.28642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Simple statistics: Pixels per mask per `cell_type` \n\nWe will use the numbers of this table to refine the masks.","metadata":{}},{"cell_type":"code","source":"df_train['n_pixels'] = df_train.annotation.apply(lambda x: np.sum([int(e) for e in x.split()[1:][::2]]))\ndf_pixels = df_train.groupby(\"cell_type\")[['n_pixels']].describe(percentiles=[0.01, 0.02, 0.03, 0.05, 0.1, 0.9, 0.95, 0.97, 0.98, 0.99])\\\n                    .astype(int).T.droplevel(level=0).T.drop(['count', '50%', 'std'], axis=1)\ndf_pixels","metadata":{"execution":{"iopub.status.busy":"2021-10-24T23:50:40.3594Z","iopub.execute_input":"2021-10-24T23:50:40.359696Z","iopub.status.idle":"2021-10-24T23:50:42.182157Z","shell.execute_reply.started":"2021-10-24T23:50:40.359664Z","shell.execute_reply":"2021-10-24T23:50:42.181563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Models","metadata":{}},{"cell_type":"markdown","source":"# Mask R-CNN from [🦠 Sartorius - Starter Torch Mask R-CNN [LB=0.270]](https://www.kaggle.com/julian3833/sartorius-starter-torch-mask-r-cnn-lb-0-205)\n\n\n__The model is trained [here](https://www.kaggle.com/julian3833/sartorius-starter-torch-mask-r-cnn-lb-0-205) and provided as a dataset [here](https://www.kaggle.com/julian3833/sartorius-starter-torch-mask-r-cnn/).__\n\nIt comes from version 28, the epoch 18 (which is the one that performed the best).\n\nSee [this notebook]((https://www.kaggle.com/julian3833/sartorius-starter-torch-mask-r-cnn-lb-0-205)) for details about this model.","metadata":{}},{"cell_type":"code","source":"def get_pretrained_mask_cnn():\n    # This is just a dummy value for the classification head\n    NUM_CLASSES = 2\n    \n    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=False, \n                                                               pretrained_backbone=False,\n                                                               box_detections_per_img=BOX_DETECTIONS_PER_IMG)\n\n    in_features = model.roi_heads.box_predictor.cls_score.in_features\n    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, NUM_CLASSES)\n    in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels\n    hidden_layer = 256\n    model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask, hidden_layer, NUM_CLASSES)\n    model.load_state_dict(torch.load(MASK_RCNN_CHK, map_location=DEVICE))\n    model = model.to(DEVICE)\n    model.eval()\n    return model\n\nmodel = get_pretrained_mask_cnn()","metadata":{"execution":{"iopub.status.busy":"2021-10-24T23:39:06.083782Z","iopub.execute_input":"2021-10-24T23:39:06.084087Z","iopub.status.idle":"2021-10-24T23:39:06.807677Z","shell.execute_reply.started":"2021-10-24T23:39:06.08406Z","shell.execute_reply":"2021-10-24T23:39:06.807053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Classifier from [🦠 Sartorius - Resnet34 Classifier](https://www.kaggle.com/julian3833/sartorius-resnet34-classifier)\n\n__The model fine-tuned [here](https://www.kaggle.com/julian3833/sartorius-resnet34-classifier) and provided as a dataset [here](https://www.kaggle.com/julian3833/sartorius-resnet-34-classifier-finetuned).__\n\n\nI stored the full model instead of the state dict by mistake, so the load is done without defining the architecture, which is quite obscure.\nThe architecture is:\n\n```python\nfrom torchvision.models import resnet34\nm = resnet34(True)\nm.fc = nn.Linear(512, 3)\n```\n\nSee:\n* https://www.kaggle.com/julian3833/sartorius-resnet34-classifier\n* https://stackoverflow.com/questions/42703500/best-way-to-save-a-trained-model-in-pytorch","metadata":{}},{"cell_type":"code","source":"# Load the fine-tuned resnet34 classifier for cell_types\nclassifier = torch.load(CLASSIFIER_CHK, map_location=DEVICE)\nclassifier.to(DEVICE)\nclassifier.eval();","metadata":{"execution":{"iopub.status.busy":"2021-10-24T23:39:06.808739Z","iopub.execute_input":"2021-10-24T23:39:06.809051Z","iopub.status.idle":"2021-10-24T23:39:06.861705Z","shell.execute_reply.started":"2021-10-24T23:39:06.809024Z","shell.execute_reply":"2021-10-24T23:39:06.86099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Classifier utility functions","metadata":{}},{"cell_type":"code","source":"# Get the input of the classifier\n# The process overlaps a bit with the Mask R-CNN preprocessing\n# But they are different\ndef get_image_for_classifier(image_id):\n    image_path = os.path.join(TEST_PATH, image_id + '.png')\n    transforms = A.Compose([A.Resize(224, 224), \n                       A.Normalize(mean=RESNET_MEAN, std=RESNET_STD, p=1), \n                       ToTensorV2()])\n    image = transforms(image=cv2.imread(image_path))['image']\n    return image.unsqueeze(0).to(DEVICE)\n\n# Assess the image_id cell_type with the classifier\ndef get_image_cell_type(classifier, image_id):\n    img = get_image_for_classifier(image_id)\n    with torch.no_grad():\n        logits = classifier(img)[0]\n        cell_type_idx = torch.argmax(logits).item()\n    return CELL_TYPES[cell_type_idx]","metadata":{"execution":{"iopub.status.busy":"2021-10-24T23:39:06.862905Z","iopub.execute_input":"2021-10-24T23:39:06.863143Z","iopub.status.idle":"2021-10-24T23:39:06.869836Z","shell.execute_reply.started":"2021-10-24T23:39:06.863113Z","shell.execute_reply":"2021-10-24T23:39:06.869123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Test Dataset","metadata":{}},{"cell_type":"code","source":"class CellTestDataset(Dataset):\n    def __init__(self, image_dir):\n        self.image_dir = image_dir\n        self.image_ids = [f[:-4]for f in os.listdir(self.image_dir)]\n    \n    def __getitem__(self, idx):\n        image_id = self.image_ids[idx]\n        image_path = os.path.join(self.image_dir, image_id + '.png')\n        image = Image.open(image_path).convert(\"RGB\")\n        image = F.to_tensor(image)\n        return {'image': image, 'image_id': image_id}\n\n    def __len__(self):\n        return len(self.image_ids)\n    \nds_test = CellTestDataset(TEST_PATH)","metadata":{"execution":{"iopub.status.busy":"2021-10-24T23:39:06.87097Z","iopub.execute_input":"2021-10-24T23:39:06.871336Z","iopub.status.idle":"2021-10-24T23:39:06.88644Z","shell.execute_reply.started":"2021-10-24T23:39:06.871294Z","shell.execute_reply":"2021-10-24T23:39:06.885716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Utility functions","metadata":{}},{"cell_type":"code","source":"def rle_encoding(x):\n    dots = np.where(x.flatten() == 1)[0]\n    run_lengths = []\n    prev = -2\n    for b in dots:\n        if (b>prev+1): run_lengths.extend((b + 1, 0))\n        run_lengths[-1] += 1\n        prev = b\n    return ' '.join(map(str, run_lengths))\n\n\ndef remove_overlapping_pixels(mask, other_masks):\n    for other_mask in other_masks:\n        if np.sum(np.logical_and(mask, other_mask)) > 0:\n            mask[np.logical_and(mask, other_mask)] = 0\n    return mask","metadata":{"execution":{"iopub.status.busy":"2021-10-24T23:39:06.887587Z","iopub.execute_input":"2021-10-24T23:39:06.887983Z","iopub.status.idle":"2021-10-24T23:39:06.896307Z","shell.execute_reply.started":"2021-10-24T23:39:06.887948Z","shell.execute_reply":"2021-10-24T23:39:06.895748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction loop","metadata":{}},{"cell_type":"markdown","source":"# Mask refinement function\n\nThe masks provided by Mask R-CNN have a probability in each of the pixels and we turn that into a mask thresholding it.\n\nA simple method to do this is to use one fixed number. That is `MASK_THRESHOLD`, which was fixed to `0.5` till now.\n\nHere we propose a refined method. \n\nThe method changes the threshold value in border cases, making sure the number of pixels of the mask is not lower than the `5%` percentile and not higher than the `95%` percentile of the train data for the givel `cell_type`.\n","metadata":{}},{"cell_type":"code","source":"\ndef refine_mask(mask, df_pixels, cell_type):\n    \n    \n    # Minimum number of pixels:\n    # The percentile 0.02 of the cell_type in the train set\n    min_pixels = df_pixels.loc[cell_type, '2%']\n    # Max number of pixels\n    # The percentile 0.95 of the cell_type in the train set\n    max_pixels = df_pixels.loc[cell_type, '98%']\n    \n    binary_mask = mask > MASK_THRESHOLD\n    \n    # If the mask is too small, make the condition less strict\n    # increasing its size until it reaches a minimum number of pixels\n    if binary_mask.sum() < min_pixels:\n        for t in range(25):\n            binary_mask = mask > (MASK_THRESHOLD - t * 0.02)\n            if binary_mask.sum() > min_pixels:\n                break\n    \n    # If the mask is too large, make the condition more strict\n    # reducing its size until it has less than certain amount of pixels\n    if binary_mask.sum() > max_pixels:\n        for t in range(25):\n            binary_mask = mask > (MASK_THRESHOLD + t * 0.02)\n            if binary_mask.sum() < max_pixels:\n                break\n\n    return binary_mask\n        \n    ","metadata":{"execution":{"iopub.status.busy":"2021-10-25T00:16:28.024426Z","iopub.execute_input":"2021-10-25T00:16:28.024722Z","iopub.status.idle":"2021-10-25T00:16:28.032902Z","shell.execute_reply.started":"2021-10-25T00:16:28.024689Z","shell.execute_reply":"2021-10-25T00:16:28.03212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction","metadata":{}},{"cell_type":"code","source":"MIN_SCORE = 0.585\n\nMASK_THRESHOLD = 0.5","metadata":{"execution":{"iopub.status.busy":"2021-10-25T00:16:28.841197Z","iopub.execute_input":"2021-10-25T00:16:28.841608Z","iopub.status.idle":"2021-10-25T00:16:28.844758Z","shell.execute_reply.started":"2021-10-25T00:16:28.841579Z","shell.execute_reply":"2021-10-25T00:16:28.844251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = []\nfor sample in ds_test:\n    img = sample['image']\n    image_id = sample['image_id']\n    \n    # Get classifier prediction: cell_type\n    cell_type = get_image_cell_type(classifier, image_id)\n    \n    # Given the cell_type, determine the numnber of instances to predict\n    max_preds = df_instances.loc[cell_type, '99%']\n    # min_preds = df_instances.loc[cell_type, '10%']\n    \n    \n    # Get Mask R-CNN predictions\n    with torch.no_grad():\n        result = model([img.to(DEVICE)])[0]\n    \n    previous_masks = []\n    for i, mask in enumerate(result[\"masks\"]):\n        \n        score = result[\"scores\"][i].cpu().item()\n        \n        # # Predict at most the 90% number of instances per cell type\n        # if i >= max_preds:\n        #    break\n        \n        # Minimum score required for instance to be kept\n        if score < MIN_SCORE:\n            break\n        \n        mask = mask.cpu().numpy()\n        \n        # See above \"Mask refinement function\"\n        binary_mask = refine_mask(mask, df_pixels, cell_type)\n\n        binary_mask = remove_overlapping_pixels(binary_mask, previous_masks)\n        \n        previous_masks.append(binary_mask)\n        rle = rle_encoding(binary_mask)\n        submission.append((image_id, rle))\n\n    # Add empty prediction if no RLE was generated for this image\n    all_images_ids = [image_id for image_id, rle in submission]\n    if image_id not in all_images_ids:\n        submission.append((image_id, \"\"))\n\ndf_sub = pd.DataFrame(submission, columns=['id', 'predicted'])\ndf_sub.to_csv(\"submission.csv\", index=False)\ndf_sub.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-25T00:16:29.443524Z","iopub.execute_input":"2021-10-25T00:16:29.445489Z","iopub.status.idle":"2021-10-25T00:16:58.320583Z","shell.execute_reply.started":"2021-10-25T00:16:29.445447Z","shell.execute_reply":"2021-10-25T00:16:58.31977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Remember to upvote if you found this notebook useful or interesting!","metadata":{}}]}