{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.9","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceType":"competition","sourceId":19989,"databundleVersionId":1160143},{"sourceType":"competition","sourceId":22990,"datasetId":1136396,"databundleVersionId":2048213},{"sourceType":"competition","sourceId":24911,"databundleVersionId":2119030}],"dockerImageVersionId":30085,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction \n\nDeepMAC (Deep Mask-heads Above CenterNet) is model to tackle the problem of partially supervised instance segmentation in which we are given box annotations for all classes, but masks for only a subset of classes.\n\n@vighneshbgoogle and @sbeery shared [Run DeepMAC inference for iWildCam](https://www.kaggle.com/vighneshbgoogle/run-deepmac-inference-for-iwildcam) notebook, and I got interested in DeepMAC model.\n\nTo see how well DeepMAC can create instance masks, I tried it for data of [iWildcam 2021 - FGVC8](https://www.kaggle.com/c/iwildcam2021-fgvc8), [Global Wheat Detection](https://www.kaggle.com/c/global-wheat-detection) and [HuBMAP - Hacking the Kidney](https://www.kaggle.com/c/hubmap-kidney-segmentation) in Example1 ~ 3. I also show how we can use the model for more realistic applications in Application1 and 2.\n\nI have used most of the code of original notebook, especially for chapters 1 and 2 (see Contents list below). After chapter3, my code is main.\n\n## Contents\n\n0. [About DeepMAC](#0)\n1. [Preparation](#1)\n2. [Examle1 iWildcam 2021 - FGVC8](#2)\n3. [Examle2 Global Wheat Detection](#3)\n4. [Examle3 HuBMAP - Hacking the Kidney](#4)\n5. [Application1 Create instance mask of iWildcam 2021 for MOT](#5)\n5. [Application2 Create instance mask of HuBMAP in a way that can be automated as much as possible.](#6)\n\n\n### Reference for DeepMAC\n\n* github.io: [The surprising impact of mask-head architecture\non novel class segmentation](https://google.github.io/deepmac/).\n\n* Paper: [The surprising impact of mask-head architecture on novel class segmentation](https://arxiv.org/abs/2104.00613)\n\n* Sample code: [DeepMAC colab](https://github.com/tensorflow/models/blob/master/research/object_detection/colab_tutorials/deepmac_colab.ipynb) in the Object Detection API\n\n-----------------------","metadata":{"papermill":{"duration":0.031046,"end_time":"2021-04-28T12:48:30.425268","exception":false,"start_time":"2021-04-28T12:48:30.394222","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"<a id=\"0\"></a>\n# About DeepMAC\n\nThe idea of DeepMAC is based on [MASK R-CNN](https://arxiv.org/abs/1703.06870). DeepMAC has shown significant performance improvement in creating instance masks of unknown objects. There are some differences between previous study like following:\n\n* Deeper and hourglass network as mask-heads.\n\n* Train late stage networks using not proposals of RPN but ground truth.\n\nTo learn about DeepMAC, it seems be efficient to learn along the way how they have developed. I'll list the papers in the order below, but I think it's pretty easy to study up to Mask R-CNN, since there are already many explanatory articles on the Internet and notebooks on kaggle. \n\n1. R-CNN: [Rich feature hierarchies for accurate object detection and semantic segmentation](https://arxiv.org/pdf/1311.2524.pdf)\n\n2. [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://arxiv.org/abs/1506.01497)\n\n3. [Mask R-CNN](https://arxiv.org/abs/1703.06870)\n\n4. DeepMAC: [The surprising impact of mask-head architecture on novel class segmentation](https://arxiv.org/abs/1703.06870)","metadata":{}},{"cell_type":"markdown","source":"<a id=\"1\"></a>\n# Preparation\n## Imports and Definitions\n\nLoad libraries to use and define utility functions.","metadata":{"papermill":{"duration":0.029149,"end_time":"2021-04-28T12:48:30.483997","exception":false,"start_time":"2021-04-28T12:48:30.454848","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Load libraries\n\nimport collections\nfrom collections import defaultdict\nimport glob\nimport gc\nimport io\nimport json\nimport logging\nimport os\nimport random\nimport warnings\n\nimport imageio\nfrom IPython.display import display, Javascript\nfrom IPython.display import Image as IPyImage\nimport matplotlib\nfrom matplotlib import patches\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image, ImageDraw, ImageFont\nimport scipy.misc\nfrom six import BytesIO\nfrom skimage import color\nfrom skimage import measure\nfrom skimage import transform\nfrom skimage import util\nfrom skimage.color import rgb_colors\nimport tensorflow as tf\nimport tifffile as tiff ","metadata":{"papermill":{"duration":6.835529,"end_time":"2021-04-28T12:48:37.349906","exception":false,"start_time":"2021-04-28T12:48:30.514377","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# For checking GPU setting\n\nprint('tensorflow version:', tf.__version__)\nos.environ['CUDA_VISIBLE_DEVICES'] = '0'\ngpu_devices = tf.config.experimental.list_physical_devices('GPU')\nif gpu_devices:\n    for gpu_device in gpu_devices:\n        print('device available:', gpu_device)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define utilities\n\nCOLORS = ([rgb_colors.cyan, rgb_colors.orange, rgb_colors.pink,\n           rgb_colors.purple, rgb_colors.limegreen , rgb_colors.crimson] +\n          [(color) for (name, color) in color.color_dict.items()])\nrandom.shuffle(COLORS)\n\nlogging.disable(logging.WARNING)\n\n\ndef read_image(path):\n  \"\"\"Read an image and optionally resize it for better plotting.\"\"\"\n  with open(path, 'rb') as f:\n    img = Image.open(f)\n    return np.array(img, dtype=np.uint8)\n\ndef read_json(path):\n  with open(path) as f:\n    return json.load(f)\n\ndef create_detection_map(annotations):\n  \"\"\"Creates a dict mapping IDs to detections.\"\"\"\n\n  ann_map = {}\n  for image in annotations['images']:\n    ann_map[image['id']] = image['detections']\n  return ann_map\n\ndef get_mask_prediction_function(model):\n  \"\"\"Get single image mask prediction function using a model.\"\"\"\n\n  @tf.function\n  def predict_masks(image, boxes):\n    height, width, _ = image.shape.as_list()\n    batch = image[tf.newaxis]\n    boxes = boxes[tf.newaxis]\n\n    detections = model(batch, boxes)\n    masks = detections['detection_masks']\n\n    return reframe_box_masks_to_image_masks(masks[0], boxes[0],\n                                             height, width)\n\n  return predict_masks\n\ndef convert_boxes(boxes):\n  xmin, ymin, width, height = boxes[:, 0], boxes[:, 1], boxes[:, 2], boxes[:, 3]\n  ymax = ymin + height\n  xmax = xmin + width\n\n  return np.stack([ymin, xmin, ymax, xmax], axis=1).astype(np.float32)\n\n\n# Copied from tensorflow/models\ndef reframe_box_masks_to_image_masks(box_masks, boxes, image_height,\n                                     image_width, resize_method='bilinear'):\n  \"\"\"Transforms the box masks back to full image masks.\n  Embeds masks in bounding boxes of larger masks whose shapes correspond to\n  image shape.\n  Args:\n    box_masks: A tensor of size [num_masks, mask_height, mask_width].\n    boxes: A tf.float32 tensor of size [num_masks, 4] containing the box\n           corners. Row i contains [ymin, xmin, ymax, xmax] of the box\n           corresponding to mask i. Note that the box corners are in\n           normalized coordinates.\n    image_height: Image height. The output mask will have the same height as\n                  the image height.\n    image_width: Image width. The output mask will have the same width as the\n                 image width.\n    resize_method: The resize method, either 'bilinear' or 'nearest'. Note that\n      'bilinear' is only respected if box_masks is a float.\n  Returns:\n    A tensor of size [num_masks, image_height, image_width] with the same dtype\n    as `box_masks`.\n  \"\"\"\n  resize_method = 'nearest' if box_masks.dtype == tf.uint8 else resize_method\n  # TODO(rathodv): Make this a public function.\n  def reframe_box_masks_to_image_masks_default():\n    \"\"\"The default function when there are more than 0 box masks.\"\"\"\n    def transform_boxes_relative_to_boxes(boxes, reference_boxes):\n      boxes = tf.reshape(boxes, [-1, 2, 2])\n      min_corner = tf.expand_dims(reference_boxes[:, 0:2], 1)\n      max_corner = tf.expand_dims(reference_boxes[:, 2:4], 1)\n      denom = max_corner - min_corner\n      # Prevent a divide by zero.\n      denom = tf.math.maximum(denom, 1e-4)\n      transformed_boxes = (boxes - min_corner) / denom\n      return tf.reshape(transformed_boxes, [-1, 4])\n\n    box_masks_expanded = tf.expand_dims(box_masks, axis=3)\n    num_boxes = tf.shape(box_masks_expanded)[0]\n    unit_boxes = tf.concat(\n        [tf.zeros([num_boxes, 2]), tf.ones([num_boxes, 2])], axis=1)\n    reverse_boxes = transform_boxes_relative_to_boxes(unit_boxes, boxes)\n\n    # TODO(vighneshb) Use matmul_crop_and_resize so that the output shape\n    # is static. This will help us run and test on TPUs.\n    resized_crops = tf.image.crop_and_resize(\n        image=box_masks_expanded,\n        boxes=reverse_boxes,\n        box_indices=tf.range(num_boxes),\n        crop_size=[image_height, image_width],\n        method=resize_method,\n        extrapolation_value=0)\n    return tf.cast(resized_crops, box_masks.dtype)\n\n  image_masks = tf.cond(\n      tf.shape(box_masks)[0] > 0,\n      reframe_box_masks_to_image_masks_default,\n      lambda: tf.zeros([0, image_height, image_width, 1], box_masks.dtype))\n  return tf.squeeze(image_masks, axis=3)\n\ndef plot_image_annotations(image, boxes, masks, darken_image=0.5):\n  fig, ax = plt.subplots(figsize=(16, 12))\n  ax.set_axis_off()\n  image = (image * darken_image).astype(np.uint8)\n  ax.imshow(image)\n\n  height, width, _ = image.shape\n\n  num_colors = len(COLORS)\n  color_index = 0\n\n  for box, mask in zip(boxes, masks):\n    ymin, xmin, ymax, xmax = box\n    ymin *= height\n    ymax *= height\n    xmin *= width\n    xmax *= width\n\n    color = COLORS[color_index]\n    color = np.array(color)\n    rect = patches.Rectangle((xmin, ymin), xmax - xmin, ymax - ymin,\n                             linewidth=2.5, edgecolor=color, facecolor='none')\n    ax.add_patch(rect)\n    mask = (mask > 0.5).astype(np.float32)\n    color_image = np.ones_like(image) * color[np.newaxis, np.newaxis, :]\n    color_and_mask = np.concatenate(\n        [color_image, mask[:, :, np.newaxis]], axis=2)\n\n    ax.imshow(color_and_mask, alpha=0.5)\n\n    color_index = (color_index + 1) % num_colors\n\n  return ax","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.063441,"end_time":"2021-04-28T12:48:37.443128","exception":false,"start_time":"2021-04-28T12:48:37.379687","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Downloading the checkpoint\nMake sure internet access is enabled from the right panel.","metadata":{"papermill":{"duration":0.029158,"end_time":"2021-04-28T12:48:37.501781","exception":false,"start_time":"2021-04-28T12:48:37.472623","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!curl -o /kaggle/working/deepmac_1024x1024_coco17.tar.gz http://download.tensorflow.org/models/object_detection/tf2/20210329/deepmac_1024x1024_coco17.tar.gz\n!tar -xzf /kaggle/working/deepmac_1024x1024_coco17.tar.gz","metadata":{"papermill":{"duration":73.760617,"end_time":"2021-04-28T12:49:51.292175","exception":false,"start_time":"2021-04-28T12:48:37.531558","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load the model into memory\nThis can take a minute or so","metadata":{"papermill":{"duration":0.043547,"end_time":"2021-04-28T12:49:51.380311","exception":false,"start_time":"2021-04-28T12:49:51.336764","status":"completed"},"tags":[]}},{"cell_type":"code","source":"model = tf.keras.models.load_model('/kaggle/working/deepmac_1024x1024_coco17/saved_model')\nprediction_function = get_mask_prediction_function(model)","metadata":{"papermill":{"duration":91.813086,"end_time":"2021-04-28T12:51:23.237147","exception":false,"start_time":"2021-04-28T12:49:51.424061","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-------------------","metadata":{"papermill":{"duration":0.04346,"end_time":"2021-04-28T12:51:23.324914","exception":false,"start_time":"2021-04-28T12:51:23.281454","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"<a id=\"2\"></a>\n# Examle1 iWildcam 2021 - FGVC8\n## Load metadata information\n\nbounding boxes from [MegaDetector](https://github.com/microsoft/CameraTraps/blob/master/megadetector.md) model and use the\nDeepMAC model to generate instance masks.","metadata":{"papermill":{"duration":0.043009,"end_time":"2021-04-28T12:51:23.411556","exception":false,"start_time":"2021-04-28T12:51:23.368547","status":"completed"},"tags":[]}},{"cell_type":"code","source":"BOX_ANNOTATION_FILE = '/kaggle/input/iwildcam2021-fgvc8/metadata/iwildcam2021_megadetector_results.json'\ndetection_map = create_detection_map(read_json(BOX_ANNOTATION_FILE))","metadata":{"papermill":{"duration":3.345411,"end_time":"2021-04-28T12:51:26.800525","exception":false,"start_time":"2021-04-28T12:51:23.455114","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Running inference","metadata":{"papermill":{"duration":0.044533,"end_time":"2021-04-28T12:51:26.888828","exception":false,"start_time":"2021-04-28T12:51:26.844295","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Run DeepMAC and show result\n\nimage_path = '/kaggle/input/iwildcam2021-fgvc8/train/905e980e-21bc-11ea-a13a-137349068a90.jpg'\nimage_id = os.path.basename(image_path).rstrip('.jpg')\n\nif image_id not in detection_map:\n    print(f'Image {image_path} is missing detection data.')\nelif len(detection_map[image_id]) == 0:\n    print(f'There are no detected objects in the image {image_path}.')\nelse:\n    detections = detection_map[image_id]\n    image = read_image(image_path)\n    bboxes = np.array([det['bbox'] for det in detections])\n    bboxes = convert_boxes(bboxes)\n    masks = prediction_function(tf.convert_to_tensor(image),\n                                tf.convert_to_tensor(bboxes, dtype=tf.float32))\n    plot_image_annotations(image, bboxes, masks.numpy(), darken_image=0.75)","metadata":{"papermill":{"duration":31.586371,"end_time":"2021-04-28T12:51:58.518923","exception":false,"start_time":"2021-04-28T12:51:26.932552","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that masks have been created for the animals in the Bboxes.","metadata":{"papermill":{"duration":0.063617,"end_time":"2021-04-28T12:51:58.646831","exception":false,"start_time":"2021-04-28T12:51:58.583214","status":"completed"},"tags":[]}},{"cell_type":"code","source":"print(f\"There are {len(masks)} inastane masks.\")\n\nfig, axs = plt.subplots(2, 4, figsize=(20,5))\nfor i in range(8):\n    col_num = i % 4\n    row_num = i // 4\n    axs[row_num, col_num].imshow(masks[i])\n    axs[row_num, col_num].set_title(f'Instance mask of {i}-th animal.')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show result as one image\n\ndef overlap_masks(masks):\n    \"\"\"Overlap masks and return one mask\"\"\"\n    \n    masks = masks.numpy()\n    mask_overlapped = np.zeros_like(masks[0])\n\n    for mask in masks:\n        mask_overlapped = np.logical_or((mask > 0.5).astype(np.float32), mask_overlapped)\n    \n    return mask_overlapped\n    \nplt.figure(figsize=(8,8))\nplt.imshow(overlap_masks(masks))","metadata":{"papermill":{"duration":0.502027,"end_time":"2021-04-28T12:51:59.212928","exception":false,"start_time":"2021-04-28T12:51:58.710901","status":"completed"},"tags":[],"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Save memory\n\ndel detection_map\ngc.collect()","metadata":{"papermill":{"duration":1.402807,"end_time":"2021-04-28T12:52:00.681344","exception":false,"start_time":"2021-04-28T12:51:59.278537","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"--------------","metadata":{"papermill":{"duration":0.066276,"end_time":"2021-04-28T12:52:00.814884","exception":false,"start_time":"2021-04-28T12:52:00.748608","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"<a id=\"3\"></a>\n# Examle2 Global Wheat Detection\n\nWe'll see how it works in another example. [Global Wheat Detection](https://www.kaggle.com/c/global-wheat-detection) was competiton to detect wheat spikes with computer vision. To do so, it may be able to automatically detect the degree of growth and abnormalities. \n\nSince we are given the Bbox in train.csv, we will use it to create the masks in the same way as in Example1.","metadata":{"papermill":{"duration":0.066565,"end_time":"2021-04-28T12:52:00.948951","exception":false,"start_time":"2021-04-28T12:52:00.882386","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Load metadata information and process","metadata":{"papermill":{"duration":0.067174,"end_time":"2021-04-28T12:52:01.082258","exception":false,"start_time":"2021-04-28T12:52:01.015084","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Load bbox data\n\nglobal_wheat_detection_train = pd.read_csv(\"../input/global-wheat-detection/train.csv\")","metadata":{"papermill":{"duration":0.350918,"end_time":"2021-04-28T12:52:01.498974","exception":false,"start_time":"2021-04-28T12:52:01.148056","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to convert data format of bbox\n\ndef convert_deepmac_format_from_gwd_train(df):\n    \"\"\"Convert train csv of global_wheat_detection to the format same as Example1\"\"\"\n    image_ids = []\n    bboxs = []\n    result_df = defaultdict(list)\n    \n    for i in range(len(df)):\n        image_id = df.iloc[i][\"image_id\"]\n        bbox = df.iloc[i][\"bbox\"]\n        \n        # Converts string to list of 4 floats and normalizes them by 1024.\n        bbox = [float(item.strip())/1024 for item in bbox[1:-1].split(\",\")]\n        \n        image_ids.append(image_id)\n        bboxs.append(bbox)\n    \n    d = defaultdict(list)\n    for image_id, bbox in zip(image_ids, bboxs):\n        result_df[image_id].append({\"bbox\": bbox})\n    \n    return result_df","metadata":{"papermill":{"duration":0.07826,"end_time":"2021-04-28T12:52:01.643639","exception":false,"start_time":"2021-04-28T12:52:01.565379","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert data format of bbox\n\nglobal_wheat_detection_train = convert_deepmac_format_from_gwd_train(global_wheat_detection_train)\nbboxes = np.array([det['bbox'] for det in detections])","metadata":{"papermill":{"duration":42.240764,"end_time":"2021-04-28T12:52:43.950703","exception":false,"start_time":"2021-04-28T12:52:01.709939","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Running inference","metadata":{"papermill":{"duration":0.067514,"end_time":"2021-04-28T12:52:44.08623","exception":false,"start_time":"2021-04-28T12:52:44.018716","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Run DeepMAC and show result\n\nimage_path = '../input/global-wheat-detection/train/00333207f.jpg'\nimage_id = os.path.basename(image_path).rstrip('.jpg')\n\nif image_id not in global_wheat_detection_train:\n    print(f'Image {image_path} is missing detection data.')\nelif len(global_wheat_detection_train[image_id]) == 0:\n    print(f'There are no detected objects in the image {image_path}.')\nelse:\n    detections = global_wheat_detection_train[image_id]\n    image = read_image(image_path)\n    bboxes = np.array([det['bbox'] for det in detections])\n    bboxes = convert_boxes(bboxes)\n    masks = prediction_function(tf.convert_to_tensor(image),\n                                tf.convert_to_tensor(bboxes, dtype=tf.float32))\n    plot_image_annotations(image, bboxes, masks.numpy(), darken_image=0.75)","metadata":{"papermill":{"duration":43.679829,"end_time":"2021-04-28T12:53:27.83346","exception":false,"start_time":"2021-04-28T12:52:44.153631","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this example, too, we can see that we have created a mask for wheat spikes.","metadata":{"papermill":{"duration":0.09929,"end_time":"2021-04-28T12:53:28.032575","exception":false,"start_time":"2021-04-28T12:53:27.933285","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Show result as one image\n    \nplt.figure(figsize=(8,8))\nplt.imshow(overlap_masks(masks))","metadata":{"papermill":{"duration":0.585286,"end_time":"2021-04-28T12:53:28.717438","exception":false,"start_time":"2021-04-28T12:53:28.132152","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Save memory\n\ndel global_wheat_detection_train\ngc.collect()","metadata":{"papermill":{"duration":1.464895,"end_time":"2021-04-28T12:53:30.283415","exception":false,"start_time":"2021-04-28T12:53:28.81852","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"------------","metadata":{"papermill":{"duration":0.102139,"end_time":"2021-04-28T12:53:30.494413","exception":false,"start_time":"2021-04-28T12:53:30.392274","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"<a id=\"4\"></a>\n# Examle3 HuBMAP - Hacking the Kidney\n\nGoal of this competition is development of a segmentation algorithm to identify the \"Glomerulus\" in the kidney. We are given train image, segmentation mask and annotation data.\n\nHowever, some of the masks and annotation data are incomplete, and methods to correct them are being studied.\n\nIf we can use DeepMAC to create masks, it may be much easier than doing precise hand labeling, although we will need to create Bboxes. Also, the annotations can be dropped into the program, making it more reproducible.\n\nCurrently, solutions are image segmentation, but we may be able to choose detection as our strategy if DeepMAC generates good mask.","metadata":{"papermill":{"duration":0.102067,"end_time":"2021-04-28T12:53:30.698595","exception":false,"start_time":"2021-04-28T12:53:30.596528","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Load metadata information and process\n\n[image_id].json includes, annotation data (not bbox), I'll load this. I also load corresponding image.\n\n<div class=\"alert alert-block alert-warning\">I will implement some utility functions in following cells, but what I want to do is to create image and Bbox of a good size to be input to DeepMAC.</div>","metadata":{"papermill":{"duration":0.101895,"end_time":"2021-04-28T12:53:30.903","exception":false,"start_time":"2021-04-28T12:53:30.801105","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Select sample image id\n\nimage_id = \"aaa6a05cc\"","metadata":{"papermill":{"duration":0.110161,"end_time":"2021-04-28T12:53:31.116739","exception":false,"start_time":"2021-04-28T12:53:31.006578","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load annotation data\n\nwith open(f\"../input/hubmap-kidney-segmentation/train/{image_id}.json\") as f:\n    structure_aaa6a05cc = json.load(f)","metadata":{"papermill":{"duration":0.125054,"end_time":"2021-04-28T12:53:31.346424","exception":false,"start_time":"2021-04-28T12:53:31.22137","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load image\nimage_aaa6a05cc = tiff.imread(f\"../input/hubmap-kidney-segmentation/train/{image_id}.tiff\")\nprint( \"Shape: \", image_aaa6a05cc.shape)","metadata":{"papermill":{"duration":3.363011,"end_time":"2021-04-28T12:53:34.811301","exception":false,"start_time":"2021-04-28T12:53:31.44829","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I'll convert annotation data to bbox, because it is not bbox, but set of coordinates (If you want to know detail, please see [HuBMAP Let's Visualize and Understand Dataset](https://www.kaggle.com/nayuts/hubmap-let-s-visualize-and-understand-dataset#With-Annotation-json-file)). \n\nI will also create a function to display the bbox and the image together to make sure that the conversion is successful. In addition, since original image is so huge, I will also create a function to crop the image and generate the corresponding bbox. ","metadata":{"papermill":{"duration":0.103511,"end_time":"2021-04-28T12:53:35.016682","exception":false,"start_time":"2021-04-28T12:53:34.913171","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#Function to convert annotation data to bbox\n\ndef convert_bbox_from_coordinates(coordinates, image_shape):\n    \"\"\"convert coordinates to bbox\"\"\"\n    x_max = max([coor[0] for coor in coordinates])\n    x_min = min([coor[0] for coor in coordinates])\n    y_max = max([coor[1] for coor in coordinates])\n    y_min = min([coor[1] for coor in coordinates])\n    \n    h, w, _ = image_shape\n    \n    return [x_min/w, y_min/h, (x_max - x_min)/w, (y_max - y_min)/h]\n\ndef create_bbox_from_coordinates(structure, image_id, image):\n    \"\"\"convert coordinates to bbox of glomerulus\"\"\"\n    result_dict = {}\n    result = []\n    \n    image_shape = image.shape\n    \n    for glomerulus in structure:\n        for coordinates in glomerulus[\"geometry\"][\"coordinates\"]:\n            result.append(convert_bbox_from_coordinates(coordinates, image_shape))\n            \n    result_dict[image_id] = result\n    \n    return result_dict","metadata":{"papermill":{"duration":0.113482,"end_time":"2021-04-28T12:53:35.233655","exception":false,"start_time":"2021-04-28T12:53:35.120173","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Function to display the bbox and the image together\n\ndef draw_bboxs(im, bboxes):\n    \"\"\"\n    detections_list: list of set includes bbox.\n    im: image read by Pillow.\n    \"\"\"\n    h_image, w_image,  _ = im.shape\n    \n    im = Image.fromarray(im)\n    draw = ImageDraw.Draw(im)\n    \n    for bbox in bboxes:\n        x1, y1,w_box, h_box = bbox\n        ymin, xmin, ymax, xmax = y1, x1, y1 + h_box, x1 + w_box\n        ymin, xmin, ymax, xmax = ymin*h_image, xmin*w_image, ymax*h_image, xmax*w_image\n        (left, right, top, bottom) = (xmin, xmax, ymin, ymax)\n        \n        draw.line([(left, top), (left, bottom), (right, bottom),\n               (right, top), (left, top)], width=10, fill='Red')\n    return im","metadata":{"papermill":{"duration":0.117869,"end_time":"2021-04-28T12:53:35.455311","exception":false,"start_time":"2021-04-28T12:53:35.337442","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Function to crop the image and generate the corresponding bbox\n\ndef has_intersect(a, b):\n    \"\"\"Detect intersect of two rectangles\n    \n    Rectangle is represented as [x_min, x_max, y_min. y_max]\n    \"\"\"\n    #return max(a[0], b[0]) <= min(a[1], b[1]) and max(a[2], b[2]) <= min(a[3], b[3])\n    return (a[0] < b[1] and a[1] > b[0] and a[3] > b[2] and a[2] < b[3])\n\ndef crop_image_and_bboxes(image, bboxes, range_axis0, range_axis1):\n    \n    result_bboxes = []\n    range_axis1_min = range_axis1[0]\n    range_axis1_max = range_axis1[1]\n    range_axis0_min = range_axis0[0]\n    range_axis0_max = range_axis0[1]\n    \n    w_image = range_axis1_max - range_axis1_min\n    h_image = range_axis0_max - range_axis0_min\n    \n    h_org, w_org, _ = image.shape\n    \n    for bbox in bboxes:\n        x1, y1,w_box, h_box = bbox\n        ymin, xmin, ymax, xmax = y1, x1, y1 + h_box, x1 + w_box\n        ymin, xmin, ymax, xmax = ymin*h_org, xmin*w_org, ymax*h_org, xmax*w_org\n        \n        if not has_intersect([xmin, xmax, ymin, ymax],\n                             [range_axis1_min, range_axis1_max, range_axis0_min, range_axis0_max]):\n            continue\n        else:\n            new_rectangle = [max(xmin, range_axis1_min) - range_axis1_min,\n                             min(xmax, range_axis1_max) - range_axis1_min,\n                             max(ymin, range_axis0_min) - range_axis0_min, \n                             min(ymax, range_axis0_max) - range_axis0_min]\n        \n            new_bbox = [new_rectangle[0]/w_image,\n                        new_rectangle[2]/h_image,\n                        (new_rectangle[1]- new_rectangle[0])/w_image, \n                        (new_rectangle[3]- new_rectangle[2])/h_image]\n            result_bboxes.append(new_bbox)\n    \n    return image[range_axis0_min:range_axis0_max, range_axis1_min:range_axis1_max, :], result_bboxes","metadata":{"papermill":{"duration":0.117287,"end_time":"2021-04-28T12:53:35.674654","exception":false,"start_time":"2021-04-28T12:53:35.557367","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create bboxes and crop image and boxes.","metadata":{"papermill":{"duration":0.102789,"end_time":"2021-04-28T12:53:35.879645","exception":false,"start_time":"2021-04-28T12:53:35.776856","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#Process inage and Bbox\n\nbboxes = create_bbox_from_coordinates(structure_aaa6a05cc, image_id, image_aaa6a05cc)\nimage_croped, bboxes_croped = crop_image_and_bboxes(image_aaa6a05cc, bboxes[image_id], [6200, 7200], [6200, 7200])","metadata":{"papermill":{"duration":0.114115,"end_time":"2021-04-28T12:53:36.096682","exception":false,"start_time":"2021-04-28T12:53:35.982567","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check the results. It looks like we have successfully created bboxes.","metadata":{"papermill":{"duration":0.103676,"end_time":"2021-04-28T12:53:36.302482","exception":false,"start_time":"2021-04-28T12:53:36.198806","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Show cropping and Bbox\n\nplt.figure(figsize=(8,8))\nimage_croped_with_bbox = draw_bboxs(image_croped, bboxes_croped)\nplt.imshow(image_croped_with_bbox)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Running inference\n\nWe got croped image and bboxes, we can inference masks.","metadata":{"papermill":{"duration":0.121721,"end_time":"2021-04-28T12:53:37.126229","exception":false,"start_time":"2021-04-28T12:53:37.004508","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Run DeepMAC and show result\n\nbboxes_croped = np.array(bboxes_croped)\nbboxes_croped = convert_boxes(bboxes_croped)\nmasks = prediction_function(tf.convert_to_tensor(image_croped),\n                            tf.convert_to_tensor(bboxes_croped, dtype=tf.float32))\nplot_image_annotations(image_croped, bboxes_croped, masks.numpy(), darken_image=0.75)","metadata":{"papermill":{"duration":27.347205,"end_time":"2021-04-28T12:54:04.594205","exception":false,"start_time":"2021-04-28T12:53:37.247","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, we were able to create the masks better than we had imagined!","metadata":{"papermill":{"duration":0.156661,"end_time":"2021-04-28T12:54:04.909282","exception":false,"start_time":"2021-04-28T12:54:04.752621","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Show result as one image\n\nmask_overlapped = overlap_masks(masks)\n    \nplt.figure(figsize=(8,8))\nplt.imshow(mask_overlapped)","metadata":{"papermill":{"duration":0.412372,"end_time":"2021-04-28T12:54:05.477637","exception":false,"start_time":"2021-04-28T12:54:05.065265","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Make sure that it matches the test data.　Load encoded mask from train.csv and decode it.","metadata":{"papermill":{"duration":0.153388,"end_time":"2021-04-28T12:54:05.785438","exception":false,"start_time":"2021-04-28T12:54:05.63205","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#Load given mask as dataset.\n\n# https://www.kaggle.com/paulorzp/rle-functions-run-lenght-encode-decode\n\ndef rle2mask(mask_rle, shape=(1600,256)):\n    '''\n    mask_rle: run-length as string formated (start length)\n    shape: (width,height) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n\n    '''\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape).T\n\nhubmap_train = pd.read_csv(\"../input/hubmap-kidney-segmentation/train.csv\")\nmask_given = rle2mask(hubmap_train[hubmap_train[\"id\"]==image_id][\"encoding\"].iloc[-1],\n                      (image_aaa6a05cc.shape[1], image_aaa6a05cc.shape[0]))","metadata":{"papermill":{"duration":0.636282,"end_time":"2021-04-28T12:54:06.57612","exception":false,"start_time":"2021-04-28T12:54:05.939838","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here is a side-by-side comparison.","metadata":{"papermill":{"duration":0.154044,"end_time":"2021-04-28T12:54:06.884672","exception":false,"start_time":"2021-04-28T12:54:06.730628","status":"completed"},"tags":[]}},{"cell_type":"code","source":"fig, axs = plt.subplots(1, 2, figsize=(10,10))\naxs[0].imshow(mask_overlapped)\naxs[0].set_title('Result of DeepMAC')\naxs[1].imshow(mask_given[6200:7200, 6200:7200])\naxs[1].set_title('Given mask')","metadata":{"papermill":{"duration":0.614369,"end_time":"2021-04-28T12:54:07.653777","exception":false,"start_time":"2021-04-28T12:54:07.039408","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check how well they match.","metadata":{"papermill":{"duration":0.15726,"end_time":"2021-04-28T12:54:07.968012","exception":false,"start_time":"2021-04-28T12:54:07.810752","status":"completed"},"tags":[]}},{"cell_type":"code","source":"print(\"Percentage of match: \", np.sum(mask_overlapped == mask_given[6200:7200, 6200:7200]) / np.sum(np.ones_like(mask_overlapped)) * 100, \"%\")","metadata":{"papermill":{"duration":0.171581,"end_time":"2021-04-28T12:54:08.297383","exception":false,"start_time":"2021-04-28T12:54:08.125802","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check in other areas.","metadata":{}},{"cell_type":"code","source":"image_croped, bboxes_croped = crop_image_and_bboxes(image_aaa6a05cc, bboxes[image_id], \n                                                    [1000, 5000], [1200, 6200])\nbboxes_croped = np.array(bboxes_croped)\nbboxes_croped = convert_boxes(bboxes_croped)\nmasks = prediction_function(tf.convert_to_tensor(image_croped),\n                            tf.convert_to_tensor(bboxes_croped, dtype=tf.float32))\nmask_overlapped = overlap_masks(masks)\n\n# Show result\nfig, axs = plt.subplots(1, 2, figsize=(10,10))\naxs[0].imshow(mask_overlapped)\naxs[0].set_title('Result of DeepMAC')\naxs[1].imshow(mask_given[1000:5000, 1200:6200])\naxs[1].set_title('Given mask')\n\n# Percentage of match\nprint(f\"Percentage of match of [1000:5000, 1200:6200] of {image_id}: \",\n      np.sum(mask_overlapped == mask_given[1000:5000, 1200:6200]) /\n      np.sum(np.ones_like(mask_overlapped)) * 100, \"%\")","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_croped, bboxes_croped = crop_image_and_bboxes(image_aaa6a05cc, bboxes[image_id], \n                                                    [1000, 3000], [7200, 8600])\nbboxes_croped = np.array(bboxes_croped)\nbboxes_croped = convert_boxes(bboxes_croped)\nmasks = prediction_function(tf.convert_to_tensor(image_croped),\n                            tf.convert_to_tensor(bboxes_croped, dtype=tf.float32))\nmask_overlapped = overlap_masks(masks)\n\n# Show result\nfig, axs = plt.subplots(1, 2, figsize=(10,10))\naxs[0].imshow(mask_overlapped)\naxs[0].set_title('Result of DeepMAC')\naxs[1].imshow(mask_given[1000:3000, 7200:8600])\naxs[1].set_title('Given mask')\n\n# Percentage of match\nprint(f\"Percentage of match of [1000:3000, 7200:8600] of {image_id}: \",\n      np.sum(mask_overlapped == mask_given[1000:3000, 7200:8600]) /\n      np.sum(np.ones_like(mask_overlapped)) * 100, \"%\")","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_croped, bboxes_croped = crop_image_and_bboxes(image_aaa6a05cc, bboxes[image_id], \n                                                    [5000, 10000], [5000, 10000])\nbboxes_croped = np.array(bboxes_croped)\nbboxes_croped = convert_boxes(bboxes_croped)\nmasks = prediction_function(tf.convert_to_tensor(image_croped),\n                            tf.convert_to_tensor(bboxes_croped, dtype=tf.float32))\nmask_overlapped = overlap_masks(masks)\n\n# Show result\nfig, axs = plt.subplots(1, 2, figsize=(10,10))\naxs[0].imshow(mask_overlapped)\naxs[0].set_title('Result of DeepMAC')\naxs[1].imshow(mask_given[5000:10000, 5000:10000])\naxs[1].set_title('Given mask')\n\nprint(f\"Percentage of match of [5000:10000, 5000:10000] of {image_id}: \",\n      np.sum(mask_overlapped == mask_given[5000:10000, 5000:10000]) /\n      np.sum(np.ones_like(mask_overlapped)) * 100, \"%\")","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del image_aaa6a05cc, hubmap_train, \ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---------------------","metadata":{}},{"cell_type":"markdown","source":"<a id=\"5\"></a>\n# Application1 Create instance mask of iWildcam 2021 for MOTS\n\nIt is still challenging to track objects when multiple objects are involved. Multi-object tracking and segmentation (MOTS) was first introduced in [MOTS: Multi-Object Tracking and Segmentation](https://arxiv.org/abs/1902.03604) at 2019. Until then, the mainstream tracking method used banding boxes.  However, when the boxes overlap each other, the accuracy of the tracking is reduced. The segmentation method has the potential to solve this problem and is being studied.\n\nDataset of [iWildcam 2021 - FGVC8](https://www.kaggle.com/c/iwildcam2021-fgvc8/data) already provides instance masks as additional png files. But annotation file follows COCO format, so we also see how to create masks processed one by [RLE](https://en.wikipedia.org/wiki/Run-length_encoding) or extract contours COCO format usually provides.\n\n### Load data","metadata":{}},{"cell_type":"code","source":"BOX_ANNOTATION_FILE = '/kaggle/input/iwildcam2021-fgvc8/metadata/iwildcam2021_megadetector_results.json'\ndetection_map = create_detection_map(read_json(BOX_ANNOTATION_FILE))\n    \nimage_path = '/kaggle/input/iwildcam2021-fgvc8/train/90605482-21bc-11ea-a13a-137349068a90.jpg'\nimage_id = os.path.basename(image_path).rstrip('.jpg')\nim = Image.open(image_path)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Show target image with bbox\n\nIn this image (image id: 90605482-21bc-11ea-a13a-137349068a90), some bounding box overlaps  each other. Therefore, to achieve accurate tracking, we may choose to create segmentation as strategy.","metadata":{}},{"cell_type":"code","source":"# Show target image and bboxes\n\ndef draw_bboxs_app1(detections_list, im):\n    \"\"\"\n    detections_list: list of set includes bbox.\n    im: image read by Pillow.\n    \n    #Refered: https://www.kaggle.com/qinhui1999/how-to-use-bbox-for-iwildcam-2020 \n    \"\"\"\n    \n    for detection in detections_list:\n        x1, y1,w_box, h_box = detection[\"bbox\"]\n        ymin,xmin,ymax, xmax=y1, x1, y1 + h_box, x1 + w_box\n        draw = ImageDraw.Draw(im)\n        \n        imageWidth=im.size[0]\n        imageHeight= im.size[1]\n        (left, right, top, bottom) = (xmin * imageWidth, xmax * imageWidth,\n                                      ymin * imageHeight, ymax * imageHeight)\n        \n        draw.line([(left, top), (left, bottom), (right, bottom),\n               (right, top), (left, top)], width=4, fill='Red')\n        \ndraw_bboxs_app1(detection_map[image_id], im)\nplt.figure(figsize=(15, 15))\nplt.imshow(im)\nplt.title(f\"{image_id} image with bbox\")","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create instance masks\n\nWith DeepMAC, we will create instance masks.","metadata":{}},{"cell_type":"code","source":"# Run DeepMAC and show result\n\nif image_id not in detection_map:\n    print(f'Image {image_path} is missing detection data.')\nelif len(detection_map[image_id]) == 0:\n    print(f'There are no detected objects in the image {image_path}.')\nelse:\n    detections = detection_map[image_id]\n    image = read_image(image_path)\n    bboxes = np.array([det['bbox'] for det in detections])\n    bboxes = convert_boxes(bboxes)\n    masks = prediction_function(tf.convert_to_tensor(image),\n                                tf.convert_to_tensor(bboxes, dtype=tf.float32))\n    plot_image_annotations(image, bboxes, masks.numpy(), darken_image=0.75)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show result\n\nprint(f\"There are {len(masks)} inastane masks.\")\n\nfig, axs = plt.subplots(1, 5, figsize=(20,5))\nfor i in range(5):\n    axs[i].imshow(masks[i])\n    axs[i].set_title(f'Instance mask of {i}-th animal.')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, I'll try to convert the masks to following two format COCO format provides.\n\n1. [Contours of mask](#5-1)\n2. [Encoded with RLE](#5-2)\n\nI won't append the encoded mask to given annotation files(iwildcam2021_train_annotations.json and iwildcam2021_test_annotations.json) because it was a little difficult.\n\nIncidentally, the assumption of segmentation preservation according to the COCO format is as follows.","metadata":{}},{"cell_type":"code","source":"# Contours version\n\"\"\"\n\"annotations\": [\n    {\n        \"segmentation\": [[100, 300, ..., 500.4, 200.01]],\n        \"image_id\": 90605482-21bc-11ea-a13a-137349068a90,\n        \"category_id\": 9,\n        \"id\": 15\n    },\n    ...\n]\n\"\"\"\n\n# RLE encorded version\n\"\"\"\n\"annotations\": [\n    {\n        \"segmentation\": {\n            \"counts\": [179,27,392,41,…,55,20],\n            \"size\": [426,640]\n        },\n        \"image_id\": 90605482-21bc-11ea-a13a-137349068a90,\n        \"category_id\": 9,\n        \"id\": 15\n    }\n]\n\"\"\"","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"5-1\"></a>\n### 1. Create contours of mask\n\nFirst, we create contours of mask. by find_contours method of skimage, we can easily try this.","metadata":{}},{"cell_type":"code","source":"# Extract contours from instance masks\n\ncontours_results = [measure.find_contours(mask.numpy(), 0.5) for mask in masks]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show contours with image\n# https://scikit-image.org/docs/0.5/auto_examples/plot_contours.html\n\nim = Image.open(image_path)\nfig, ax = plt.subplots(figsize=(15,15))\nax.imshow(im, cmap=plt.cm.cividis, interpolation='gaussian',alpha=0.8)\nfor contours in contours_results:\n    for contour in contours:\n        plt.plot(contour[:, 1], contour[:, 0], linewidth=2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert the result to save COCO dataset\n# https://github.com/cocodataset/cocoapi/issues/131\n\nsegmentations = []\nfor contour in contours:\n    contour = np.flip(contour, axis=1)\n    segmentations.append(contour.ravel().tolist())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"segmentations[0]","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"5-2\"></a>\n### 2. Create RLE encorded mask","metadata":{}},{"cell_type":"code","source":"# Create instance mask\n\noverlaped_mask = overlap_masks(masks)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#https://www.kaggle.com/bguberfain/memory-aware-rle-encoding\n#with bug fix\ndef rle_encode_less_memory(img):\n    #watch out for the bug\n    pixels = img.T.flatten()\n    \n    # This simplified method requires first and last pixel to be zero\n    pixels[0] = 0\n    pixels[-1] = 0\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 2\n    runs[1::2] -= runs[::2]\n    \n    return ' '.join(str(x) for x in runs)\n\nrle_encode_less_memory(overlaped_mask)","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"6\"></a>\n# Application2 Create instance mask of HuBMAP in a way that can be automated as much as possible.\n\nAt the HuBMAP competition, there was unannotated data in train data that has been actively discussed. We don't know how private dataset is annotated, so we don't know if that makes sense, but from [this discussion](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616#1250442), adding handwritten annotations for training is allowed.  I haven't gotten around to it yet, if we can automate the creation of the bbox for glomerular with detection model, we can automate the creation of the mask, we can the result of process as pseudo-label. Let's apply DeepMAC to this problem.\n\n### Step1 Load image and find unannotated one","metadata":{}},{"cell_type":"code","source":"# Load image\nimage_id = \"aaa6a05cc\"\nimage_aaa6a05cc = tiff.imread(f\"../input/hubmap-kidney-segmentation/train/{image_id}.tiff\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this case, we assume that the glomerulus in the image below was not annotated. In the actual data, this glomerulus has been annotated.","metadata":{}},{"cell_type":"code","source":"# Crop the found unannotated parts.\nplt.figure(figsize=(8,8))\nplt.imshow(image_aaa6a05cc[6600:7100, 6200:6700, :])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Step2 Crop image and create bbox\n\nIn this article, I have created create_bbox() function as simple bbox creation tool. In practice, you can use anything you like.","metadata":{}},{"cell_type":"code","source":"# Crop image and create Bbox\n\nunannotated_image = image_aaa6a05cc[6600:7100, 6200:6700, :]\n\ndef create_bbox(xmin, ymin, w, h, image_shape):\n    \"\"\"return bbox\n    \n    xmin       : x-coordinate (axis1-coordinate) of the left edge of the bbox.\n    ymin       : y-coordinate (axis0-coordinate) of the top edge of the bbox.\n    w          : with of cropped image.  \n    h          : height of cropped image.  \n    image_shape: shape of cropped image.\n    \n    \"\"\"\n    return [xmin/image_shape[0], ymin/image_shape[1],\n            w/image_shape[0], h/image_shape[1]]\n\nannotate_bbox = create_bbox(30, 100, 270, 330, unannotated_image.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show our bbox and unannotated image\n\nplt.figure(figsize=(8,8))\nplt.imshow(draw_bboxs(unannotated_image, [annotate_bbox]))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Step3 Create mask\n\nCreate mask with DeepMAC.","metadata":{}},{"cell_type":"code","source":"# Create instance mask\n\ndef create_mask(image, bboxes):\n    \"\"\"Create mask with DeepMAC from given image and bbox\n    \n    image : Cropped image.\n    bboxes: List of bbox.\n    \n    \"\"\"\n    \n    bboxes = convert_boxes(np.array(bboxes))\n    masks = prediction_function(tf.convert_to_tensor(image),\n                                tf.convert_to_tensor(bboxes, dtype=tf.float32))\n    mask_overlapped = overlap_masks(masks)\n    return mask_overlapped\n\nmask_overlapped = create_mask(unannotated_image, [annotate_bbox])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check result.","metadata":{}},{"cell_type":"code","source":"# Show result\nfig, axs = plt.subplots(1, 2, figsize=(10,10))\naxs[0].imshow(mask_overlapped)\naxs[0].set_title('New mask')\naxs[1].imshow(unannotated_image)\naxs[1].imshow(mask_overlapped, cmap='coolwarm', alpha=0.5)\naxs[1].set_title('New mask on image')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Step4 intefrate mask with existing mask\n\nLoad mask and replace the corresponding part with the new mask.","metadata":{}},{"cell_type":"code","source":"# Integrate the mask with the existing one\nhubmap_train = pd.read_csv(\"../input/hubmap-kidney-segmentation/train.csv\")\nmask_given = rle2mask(hubmap_train[hubmap_train[\"id\"]==image_id][\"encoding\"].iloc[-1],\n                      (image_aaa6a05cc.shape[1], image_aaa6a05cc.shape[0]))\nmask_given[6600:7100, 6200:6700] = mask_overlapped","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems that the result of mask is largely effected by bbox. If we want to automate using the detection model, it is likely that the detection model will need to have reasonable performance.","metadata":{}},{"cell_type":"code","source":"annotate_bbox = create_bbox(30, 100, 300, 330, unannotated_image.shape)\n\n# Show our bbox and unannotated image\n\nplt.figure(figsize=(8,8))\nplt.imshow(draw_bboxs(unannotated_image, [annotate_bbox]))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mask_overlapped = create_mask(unannotated_image, [annotate_bbox])\n\n# Show result\nfig, axs = plt.subplots(1, 2, figsize=(10,10))\naxs[0].imshow(mask_overlapped)\naxs[0].set_title('New mask')\naxs[1].imshow(unannotated_image)\naxs[1].imshow(mask_overlapped, cmap='coolwarm', alpha=0.5)\naxs[1].set_title('New mask on image')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]}]}