I wanted to share my ensemble technique. I think it can be useful in similar projects. It is a very simple process and there are few steps.
\nIoU matrices can be calculated from masks or boxes, but using masks gave us a slight boost. IoU matrices are calculated with the code below.
\ndef get_iou_matrix_from_boxes(bounding_boxes1, bounding_boxes2):\n\n \"\"\"\n Calculate IoU matrix between two sets of bounding boxes\n\n Parameters\n ----------\n bounding_boxes1 [numpy.ndarray of shape (n_objects, 4)]: Bounding boxes\n bounding_boxes2 [numpy.ndarray of shape (m_objects, 4)]: Bounding boxes\n\n Returns\n -------\n iou_matrix [numpy.ndarray of shape (n_objects, m_objects)]: IoU matrix between two sets of bounding boxes\n \"\"\"\n\n bounding_boxes1_x1, bounding_boxes1_y1, bounding_boxes1_x2, bounding_boxes1_y2 = np.split(bounding_boxes1, 4, axis=1)\n bounding_boxes2_x1, bounding_boxes2_y1, bounding_boxes2_x2, bounding_boxes2_y2 = np.split(bounding_boxes2, 4, axis=1)\n\n xa = np.maximum(bounding_boxes1_x1, np.transpose(bounding_boxes2_x1))\n ya = np.maximum(bounding_boxes1_y1, np.transpose(bounding_boxes2_y1))\n xb = np.minimum(bounding_boxes1_x2, np.transpose(bounding_boxes2_x2))\n yb = np.minimum(bounding_boxes1_y2, np.transpose(bounding_boxes2_y2))\n\n inter_area = np.maximum((xb - xa + 1), 0) * np.maximum((yb - ya + 1), 0)\n box_a_area = (bounding_boxes1_x2 - bounding_boxes1_x1 + 1) * (bounding_boxes1_y2 - bounding_boxes1_y1 + 1)\n box_b_area = (bounding_boxes2_x2 - bounding_boxes2_x1 + 1) * (bounding_boxes2_y2 - bounding_boxes2_y1 + 1)\n iou_matrix = inter_area / (box_a_area + np.transpose(box_b_area) - inter_area)\n\n return iou_matrix\n\n\ndef get_iou_matrix_from_masks(masks1, masks2):\n\n \"\"\"\n Calculate IOU matrix between two sets of masks\n\n Parameters\n ----------\n masks1 [numpy.ndarray of shape (n_objects, height, width)]: 2d binary masks\n masks2 [numpy.ndarray of shape (m_objects, height, width)]: 2d binary masks\n\n Returns\n -------\n iou_matrix [numpy.ndarray of shape (n_objects, m_objects)]: IoU matrix between two sets of masks\n \"\"\"\n\n if len(list(masks1)) == 0 or len(list(masks2)) == 0:\n print(f'empty predictions - masks1 len {len(list(masks1))}, masks2 len {len(list(masks2))}')\n return np.array([[]])\n\n enc_masks1 = [mask_util.encode(np.asarray(p, order='F')) for p in (masks1 > 0.5).astype(np.uint8)]\n enc_masks2 = [mask_util.encode(np.asarray(p, order='F')) for p in (masks2 > 0.5).astype(np.uint8)]\n iou_matrix = mask_util.iou(enc_masks1, enc_masks2, [0] * len(enc_masks1))\n\n return iou_matrix\nThis is the function used for blending multiple models' predictions. Using 0.9 iou_threshold for boxes and 0.7 iou_threshold for masks worked best for us. If more than 5 models are ensembled, it is a good choice to drop single components (not connected to any other node), because it worked like a voting mechanism under the hood.
\nimport networkx as nx\n\n\ndef blend_masks(prediction_boxes, prediction_masks, iou_threshold=0.9, label_threshold=0.5, iou_method='boxes', drop_single_components=True):\n\n \"\"\"\n Blend prediction masks of multiple models based on IoU\n\n Parameters\n ----------\n prediction_boxes [list of shape (n_models)]: Bounding box predictions of multiple models\n prediction_masks [list of shape (n_models)]: Mask predictions of multiple models\n iou_threshold (int): IoU threshold for blending masks (0 <= iou_threshold <= 1)\n iou_method (str): boxes or masks\n label_threshold (int): Label threshold for converting soft predictions to labels (0 <= iou_threshold <= 1)\n drop_single_components (bool): Whether to discard predictions without connections or not\n\n Returns\n -------\n blended_masks [numpy.ndarray of shape (n_objects, height, width)]: Blended binary masks\n \"\"\"\n\n iou_matrices = {}\n\n # Create all combinations of IoU matrices from given predictions\n for i in range(len(prediction_masks)):\n for j in range(i, len(prediction_masks)):\n if i == j:\n continue\n\n if iou_method == 'boxes':\n iou_matrix = get_iou_matrix_from_boxes(prediction_boxes[i], prediction_boxes[j])\n elif iou_method == 'masks':\n iou_matrix = get_iou_matrix_from_masks(prediction_masks[i], prediction_masks[j])\n\n iou_matrices[f'{i + 1}_{j + 1}'] = iou_matrix\n\n # Create a graph to store connected bounding boxes\n bounding_box_graph = nx.Graph()\n\n # Add all masks from all models as nodes\n for model_idx, boxes in enumerate(prediction_masks, start=1):\n nodes = [f'model{model_idx}_box{box_idx}' for box_idx in np.arange(len(boxes))]\n bounding_box_graph.add_nodes_from(nodes)\n\n del prediction_boxes\n\n # Add edges between nodes with IoU >= iou_threshold\n for model_combination, iou_matrix in iou_matrices.items():\n matching_boxes_idx = np.where(iou_matrix >= iou_threshold)\n model1_idx, model2_idx = model_combination.split('_')\n edges = [(f'model{model1_idx}_box{box1}', f'model{model2_idx}_box{box2}') for box1, box2 in zip(*matching_boxes_idx)]\n bounding_box_graph.add_edges_from(edges)\n\n del iou_matrices\n blended_masks = []\n\n for connections in nx.connected_components(bounding_box_graph):\n if len(connections) == 1:\n # Skip mask if its bounding isn't connected to any other bounding box\n if drop_single_components:\n continue\n else:\n # Append mask directly if its bounding box isn't connected to any other bounding box\n model_idx, box_idx = list(connections)[0].split('_')\n model_idx = int(model_idx.replace('model', ''))\n box_idx = int(box_idx.replace('box', ''))\n blended_masks.append(prediction_masks[model_idx - 1][box_idx])\n else:\n # Blend mask with its connections and append\n blended_mask = np.zeros((520, 704), dtype=np.float32)\n for connection in connections:\n model_idx, box_idx = connection.split('_')\n model_idx = int(model_idx.replace('model', ''))\n box_idx = int(box_idx.replace('box', ''))\n # Divide soft predictions with number of connections and accumulate on blended_mask\n blended_mask += (prediction_masks[model_idx - 1][box_idx] / len(connections))\n blended_masks.append(blended_mask)\n\n del prediction_masks, bounding_box_graph\n blended_masks = np.stack(blended_masks)\n # Convert soft predictions to binary labels\n blended_masks = np.uint8(blended_masks >= label_threshold)\n\n return blended_masks\nThis was my first instance segmentation project and I learned a lot. It would be great if top teams can share their ensembling strategies as well. Congrats to winners and happy new year to everyone!
", "messages": [ { "id": "1633894", "postDate": "12/31/2021 06:22:01", "content": "I wanted to share my ensemble technique. I think it can be useful in similar projects. It is a very simple process and there are few steps.
\nIoU matrices can be calculated from masks or boxes, but using masks gave us a slight boost. IoU matrices are calculated with the code below.
\ndef get_iou_matrix_from_boxes(bounding_boxes1, bounding_boxes2):\n\n \"\"\"\n Calculate IoU matrix between two sets of bounding boxes\n\n Parameters\n ----------\n bounding_boxes1 [numpy.ndarray of shape (n_objects, 4)]: Bounding boxes\n bounding_boxes2 [numpy.ndarray of shape (m_objects, 4)]: Bounding boxes\n\n Returns\n -------\n iou_matrix [numpy.ndarray of shape (n_objects, m_objects)]: IoU matrix between two sets of bounding boxes\n \"\"\"\n\n bounding_boxes1_x1, bounding_boxes1_y1, bounding_boxes1_x2, bounding_boxes1_y2 = np.split(bounding_boxes1, 4, axis=1)\n bounding_boxes2_x1, bounding_boxes2_y1, bounding_boxes2_x2, bounding_boxes2_y2 = np.split(bounding_boxes2, 4, axis=1)\n\n xa = np.maximum(bounding_boxes1_x1, np.transpose(bounding_boxes2_x1))\n ya = np.maximum(bounding_boxes1_y1, np.transpose(bounding_boxes2_y1))\n xb = np.minimum(bounding_boxes1_x2, np.transpose(bounding_boxes2_x2))\n yb = np.minimum(bounding_boxes1_y2, np.transpose(bounding_boxes2_y2))\n\n inter_area = np.maximum((xb - xa + 1), 0) * np.maximum((yb - ya + 1), 0)\n box_a_area = (bounding_boxes1_x2 - bounding_boxes1_x1 + 1) * (bounding_boxes1_y2 - bounding_boxes1_y1 + 1)\n box_b_area = (bounding_boxes2_x2 - bounding_boxes2_x1 + 1) * (bounding_boxes2_y2 - bounding_boxes2_y1 + 1)\n iou_matrix = inter_area / (box_a_area + np.transpose(box_b_area) - inter_area)\n\n return iou_matrix\n\n\ndef get_iou_matrix_from_masks(masks1, masks2):\n\n \"\"\"\n Calculate IOU matrix between two sets of masks\n\n Parameters\n ----------\n masks1 [numpy.ndarray of shape (n_objects, height, width)]: 2d binary masks\n masks2 [numpy.ndarray of shape (m_objects, height, width)]: 2d binary masks\n\n Returns\n -------\n iou_matrix [numpy.ndarray of shape (n_objects, m_objects)]: IoU matrix between two sets of masks\n \"\"\"\n\n if len(list(masks1)) == 0 or len(list(masks2)) == 0:\n print(f'empty predictions - masks1 len {len(list(masks1))}, masks2 len {len(list(masks2))}')\n return np.array([[]])\n\n enc_masks1 = [mask_util.encode(np.asarray(p, order='F')) for p in (masks1 > 0.5).astype(np.uint8)]\n enc_masks2 = [mask_util.encode(np.asarray(p, order='F')) for p in (masks2 > 0.5).astype(np.uint8)]\n iou_matrix = mask_util.iou(enc_masks1, enc_masks2, [0] * len(enc_masks1))\n\n return iou_matrix\nThis is the function used for blending multiple models' predictions. Using 0.9 iou_threshold for boxes and 0.7 iou_threshold for masks worked best for us. If more than 5 models are ensembled, it is a good choice to drop single components (not connected to any other node), because it worked like a voting mechanism under the hood.
\nimport networkx as nx\n\n\ndef blend_masks(prediction_boxes, prediction_masks, iou_threshold=0.9, label_threshold=0.5, iou_method='boxes', drop_single_components=True):\n\n \"\"\"\n Blend prediction masks of multiple models based on IoU\n\n Parameters\n ----------\n prediction_boxes [list of shape (n_models)]: Bounding box predictions of multiple models\n prediction_masks [list of shape (n_models)]: Mask predictions of multiple models\n iou_threshold (int): IoU threshold for blending masks (0 <= iou_threshold <= 1)\n iou_method (str): boxes or masks\n label_threshold (int): Label threshold for converting soft predictions to labels (0 <= iou_threshold <= 1)\n drop_single_components (bool): Whether to discard predictions without connections or not\n\n Returns\n -------\n blended_masks [numpy.ndarray of shape (n_objects, height, width)]: Blended binary masks\n \"\"\"\n\n iou_matrices = {}\n\n # Create all combinations of IoU matrices from given predictions\n for i in range(len(prediction_masks)):\n for j in range(i, len(prediction_masks)):\n if i == j:\n continue\n\n if iou_method == 'boxes':\n iou_matrix = get_iou_matrix_from_boxes(prediction_boxes[i], prediction_boxes[j])\n elif iou_method == 'masks':\n iou_matrix = get_iou_matrix_from_masks(prediction_masks[i], prediction_masks[j])\n\n iou_matrices[f'{i + 1}_{j + 1}'] = iou_matrix\n\n # Create a graph to store connected bounding boxes\n bounding_box_graph = nx.Graph()\n\n # Add all masks from all models as nodes\n for model_idx, boxes in enumerate(prediction_masks, start=1):\n nodes = [f'model{model_idx}_box{box_idx}' for box_idx in np.arange(len(boxes))]\n bounding_box_graph.add_nodes_from(nodes)\n\n del prediction_boxes\n\n # Add edges between nodes with IoU >= iou_threshold\n for model_combination, iou_matrix in iou_matrices.items():\n matching_boxes_idx = np.where(iou_matrix >= iou_threshold)\n model1_idx, model2_idx = model_combination.split('_')\n edges = [(f'model{model1_idx}_box{box1}', f'model{model2_idx}_box{box2}') for box1, box2 in zip(*matching_boxes_idx)]\n bounding_box_graph.add_edges_from(edges)\n\n del iou_matrices\n blended_masks = []\n\n for connections in nx.connected_components(bounding_box_graph):\n if len(connections) == 1:\n # Skip mask if its bounding isn't connected to any other bounding box\n if drop_single_components:\n continue\n else:\n # Append mask directly if its bounding box isn't connected to any other bounding box\n model_idx, box_idx = list(connections)[0].split('_')\n model_idx = int(model_idx.replace('model', ''))\n box_idx = int(box_idx.replace('box', ''))\n blended_masks.append(prediction_masks[model_idx - 1][box_idx])\n else:\n # Blend mask with its connections and append\n blended_mask = np.zeros((520, 704), dtype=np.float32)\n for connection in connections:\n model_idx, box_idx = connection.split('_')\n model_idx = int(model_idx.replace('model', ''))\n box_idx = int(box_idx.replace('box', ''))\n # Divide soft predictions with number of connections and accumulate on blended_mask\n blended_mask += (prediction_masks[model_idx - 1][box_idx] / len(connections))\n blended_masks.append(blended_mask)\n\n del prediction_masks, bounding_box_graph\n blended_masks = np.stack(blended_masks)\n # Convert soft predictions to binary labels\n blended_masks = np.uint8(blended_masks >= label_threshold)\n\n return blended_masks\nThis was my first instance segmentation project and I learned a lot. It would be great if top teams can share their ensembling strategies as well. Congrats to winners and happy new year to everyone!
", "rawMarkdown": "I wanted to share my ensemble technique. I think it can be useful in similar projects. It is a very simple process and there are few steps.\n\n* Iterate over list of mask or box predictions of multiple models and calculate IoU matrix for every combination (e.g. if there are 3 models, there will be 3 IoU matrices of model1xmodel2, model1xmodel3, model2xmodel3)\n* Create a graph\n* Store all predictions (only indices) from all models as nodes in the graph (e.g. model1pred1, model3pred300)\n* Add edges between nodes if that point in IoU matrix is higher than the given IoU threshold\n* Blend connected components\n\nIoU matrices can be calculated from masks or boxes, but using masks gave us a slight boost. IoU matrices are calculated with the code below.\n\n```\ndef get_iou_matrix_from_boxes(bounding_boxes1, bounding_boxes2):\n\n \"\"\"\n Calculate IoU matrix between two sets of bounding boxes\n \n Parameters\n ----------\n bounding_boxes1 [numpy.ndarray of shape (n_objects, 4)]: Bounding boxes\n bounding_boxes2 [numpy.ndarray of shape (m_objects, 4)]: Bounding boxes\n \n Returns\n -------\n iou_matrix [numpy.ndarray of shape (n_objects, m_objects)]: IoU matrix between two sets of bounding boxes\n \"\"\"\n\n bounding_boxes1_x1, bounding_boxes1_y1, bounding_boxes1_x2, bounding_boxes1_y2 = np.split(bounding_boxes1, 4, axis=1)\n bounding_boxes2_x1, bounding_boxes2_y1, bounding_boxes2_x2, bounding_boxes2_y2 = np.split(bounding_boxes2, 4, axis=1)\n\n xa = np.maximum(bounding_boxes1_x1, np.transpose(bounding_boxes2_x1))\n ya = np.maximum(bounding_boxes1_y1, np.transpose(bounding_boxes2_y1))\n xb = np.minimum(bounding_boxes1_x2, np.transpose(bounding_boxes2_x2))\n yb = np.minimum(bounding_boxes1_y2, np.transpose(bounding_boxes2_y2))\n\n inter_area = np.maximum((xb - xa + 1), 0) * np.maximum((yb - ya + 1), 0)\n box_a_area = (bounding_boxes1_x2 - bounding_boxes1_x1 + 1) * (bounding_boxes1_y2 - bounding_boxes1_y1 + 1)\n box_b_area = (bounding_boxes2_x2 - bounding_boxes2_x1 + 1) * (bounding_boxes2_y2 - bounding_boxes2_y1 + 1)\n iou_matrix = inter_area / (box_a_area + np.transpose(box_b_area) - inter_area)\n\n return iou_matrix\n\n\ndef get_iou_matrix_from_masks(masks1, masks2):\n \n \"\"\"\n Calculate IOU matrix between two sets of masks\n \n Parameters\n ----------\n masks1 [numpy.ndarray of shape (n_objects, height, width)]: 2d binary masks\n masks2 [numpy.ndarray of shape (m_objects, height, width)]: 2d binary masks\n \n Returns\n -------\n iou_matrix [numpy.ndarray of shape (n_objects, m_objects)]: IoU matrix between two sets of masks\n \"\"\"\n \n if len(list(masks1)) == 0 or len(list(masks2)) == 0:\n print(f'empty predictions - masks1 len {len(list(masks1))}, masks2 len {len(list(masks2))}')\n return np.array([[]])\n \n enc_masks1 = [mask_util.encode(np.asarray(p, order='F')) for p in (masks1 > 0.5).astype(np.uint8)]\n enc_masks2 = [mask_util.encode(np.asarray(p, order='F')) for p in (masks2 > 0.5).astype(np.uint8)]\n iou_matrix = mask_util.iou(enc_masks1, enc_masks2, [0] * len(enc_masks1))\n \n return iou_matrix\n```\n\nThis is the function used for blending multiple models' predictions. Using 0.9 iou_threshold for boxes and 0.7 iou_threshold for masks worked best for us. If more than 5 models are ensembled, it is a good choice to drop single components (not connected to any other node), because it worked like a voting mechanism under the hood.\n\n```\nimport networkx as nx\n\n\ndef blend_masks(prediction_boxes, prediction_masks, iou_threshold=0.9, label_threshold=0.5, iou_method='boxes', drop_single_components=True):\n\n \"\"\"\n Blend prediction masks of multiple models based on IoU\n \n Parameters\n ----------\n prediction_boxes [list of shape (n_models)]: Bounding box predictions of multiple models\n prediction_masks [list of shape (n_models)]: Mask predictions of multiple models\n iou_threshold (int): IoU threshold for blending masks (0 <= iou_threshold <= 1)\n iou_method (str): boxes or masks\n label_threshold (int): Label threshold for converting soft predictions to labels (0 <= iou_threshold <= 1)\n drop_single_components (bool): Whether to discard predictions without connections or not\n \n Returns\n -------\n blended_masks [numpy.ndarray of shape (n_objects, height, width)]: Blended binary masks\n \"\"\"\n\n iou_matrices = {}\n\n # Create all combinations of IoU matrices from given predictions\n for i in range(len(prediction_masks)):\n for j in range(i, len(prediction_masks)):\n if i == j:\n continue\n \n if iou_method == 'boxes':\n iou_matrix = get_iou_matrix_from_boxes(prediction_boxes[i], prediction_boxes[j])\n elif iou_method == 'masks':\n iou_matrix = get_iou_matrix_from_masks(prediction_masks[i], prediction_masks[j])\n \n iou_matrices[f'{i + 1}_{j + 1}'] = iou_matrix\n\n # Create a graph to store connected bounding boxes\n bounding_box_graph = nx.Graph()\n\n # Add all masks from all models as nodes\n for model_idx, boxes in enumerate(prediction_masks, start=1):\n nodes = [f'model{model_idx}_box{box_idx}' for box_idx in np.arange(len(boxes))]\n bounding_box_graph.add_nodes_from(nodes)\n \n del prediction_boxes\n\n # Add edges between nodes with IoU >= iou_threshold\n for model_combination, iou_matrix in iou_matrices.items():\n matching_boxes_idx = np.where(iou_matrix >= iou_threshold)\n model1_idx, model2_idx = model_combination.split('_')\n edges = [(f'model{model1_idx}_box{box1}', f'model{model2_idx}_box{box2}') for box1, box2 in zip(*matching_boxes_idx)]\n bounding_box_graph.add_edges_from(edges)\n\n del iou_matrices\n blended_masks = []\n\n for connections in nx.connected_components(bounding_box_graph):\n if len(connections) == 1:\n # Skip mask if its bounding isn't connected to any other bounding box\n if drop_single_components:\n continue\n else:\n # Append mask directly if its bounding box isn't connected to any other bounding box\n model_idx, box_idx = list(connections)[0].split('_')\n model_idx = int(model_idx.replace('model', ''))\n box_idx = int(box_idx.replace('box', ''))\n blended_masks.append(prediction_masks[model_idx - 1][box_idx])\n else:\n # Blend mask with its connections and append\n blended_mask = np.zeros((520, 704), dtype=np.float32)\n for connection in connections:\n model_idx, box_idx = connection.split('_')\n model_idx = int(model_idx.replace('model', ''))\n box_idx = int(box_idx.replace('box', ''))\n # Divide soft predictions with number of connections and accumulate on blended_mask\n blended_mask += (prediction_masks[model_idx - 1][box_idx] / len(connections))\n blended_masks.append(blended_mask)\n \n del prediction_masks, bounding_box_graph\n blended_masks = np.stack(blended_masks)\n # Convert soft predictions to binary labels\n blended_masks = np.uint8(blended_masks >= label_threshold)\n\n return blended_masks\n```\n\nThis was my first instance segmentation project and I learned a lot. It would be great if top teams can share their ensembling strategies as well. Congrats to winners and happy new year to everyone!", "votes": null }, { "id": "1633907", "postDate": "12/31/2021 06:39:23", "content": "How much did your three models ensemble increase in terms of private LB? In our case, we had two model ensembles with a similar approach. But in the end, our single model actually had the best private LB, such a letdown 😭
", "rawMarkdown": "How much did your three models ensemble increase in terms of private LB? In our case, we had two model ensembles with a similar approach. But in the end, our single model actually had the best private LB, such a letdown 😭", "votes": null }, { "id": "1633924", "postDate": "12/31/2021 07:02:42", "content": "I wasn't using 3 models. I gave that as an example. In my earlier experiments, I submit every fold of my mask r-cnn model individually and they scored between 0.316-320 on public LB, and my blend scored 0.323 on public LB. I can't find and compare their private LB scores since I didn't add descriptions to them.
", "rawMarkdown": "I wasn't using 3 models. I gave that as an example. In my earlier experiments, I submit every fold of my mask r-cnn model individually and they scored between 0.316-320 on public LB, and my blend scored 0.323 on public LB. I can't find and compare their private LB scores since I didn't add descriptions to them.", "votes": null }, { "id": "1633927", "postDate": "12/31/2021 07:08:55", "content": "I see, thank you for the info :)
", "rawMarkdown": "I see, thank you for the info :)", "votes": null }, { "id": "1634077", "postDate": "12/31/2021 10:40:59", "content": "Thank you @gunesevitan, it is a very interesting approach based on graphs. I already kept note of it for future competitions and projects :-)
", "rawMarkdown": "Thank you @gunesevitan, it is a very interesting approach based on graphs. I already kept note of it for future competitions and projects :-)", "votes": null }, { "id": "1634224", "postDate": "12/31/2021 12:45:34", "content": "An interesting ensemble strategy. 👍. In our Solution, the ensemble process is divided into three steps. Firstly, we compute the box IOU for all predicted boxes( suppose N boxes in total) and get an IOU matrix N-by-N. For each row of the matrix, we select the boxes with IOU larger than a threshold and weighted their corresponding mask. The weighted mask will be converted into a bitmask, we then update the box by the bitmask. NMW is used to filter the proposed boxes from step one. Lastly, mask IOU is used to remove the overlap (Just like the NMW doses).
", "rawMarkdown": "An interesting ensemble strategy. 👍. In [our Solution](https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/297985), the ensemble process is divided into three steps. Firstly, we compute the box IOU for all predicted boxes( suppose N boxes in total) and get an IOU matrix N-by-N. For each row of the matrix, we select the boxes with IOU larger than a threshold and weighted their corresponding mask. The weighted mask will be converted into a bitmask, we then update the box by the bitmask. [NMW](https://github.com/ZFTurbo/Weighted-Boxes-Fusion/blob/master/ensemble_boxes/ensemble_boxes_nmw.py) is used to filter the proposed boxes from step one. Lastly, mask IOU is used to remove the overlap (Just like the NMW doses).", "votes": null } ], "comments": [ { "id": 1633907, "author_name": "woprime", "author_url": "", "post_date": "12/31/2021 06:39:23", "content": "How much did your three models ensemble increase in terms of private LB? In our case, we had two model ensembles with a similar approach. But in the end, our single model actually had the best private LB, such a letdown 😭
", "votes": null, "replies": [ { "id": 1633924, "author_name": "gunesevitan", "author_url": "", "post_date": "12/31/2021 07:02:42", "content": "I wasn't using 3 models. I gave that as an example. In my earlier experiments, I submit every fold of my mask r-cnn model individually and they scored between 0.316-320 on public LB, and my blend scored 0.323 on public LB. I can't find and compare their private LB scores since I didn't add descriptions to them.
", "votes": null, "replies": [] }, { "id": 1633927, "author_name": "woprime", "author_url": "", "post_date": "12/31/2021 07:08:55", "content": "I see, thank you for the info :)
", "votes": null, "replies": [] } ] }, { "id": 1634077, "author_name": "lucamassaron", "author_url": "", "post_date": "12/31/2021 10:40:59", "content": "Thank you @gunesevitan, it is a very interesting approach based on graphs. I already kept note of it for future competitions and projects :-)
", "votes": null, "replies": [] }, { "id": 1634224, "author_name": "xiuqi0", "author_url": "", "post_date": "12/31/2021 12:45:34", "content": "An interesting ensemble strategy. 👍. In our Solution, the ensemble process is divided into three steps. Firstly, we compute the box IOU for all predicted boxes( suppose N boxes in total) and get an IOU matrix N-by-N. For each row of the matrix, we select the boxes with IOU larger than a threshold and weighted their corresponding mask. The weighted mask will be converted into a bitmask, we then update the box by the bitmask. NMW is used to filter the proposed boxes from step one. Lastly, mask IOU is used to remove the overlap (Just like the NMW doses).
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1633894": "I wanted to share my ensemble technique. I think it can be useful in similar projects. It is a very simple process and there are few steps.\n\n* Iterate over list of mask or box predictions of multiple models and calculate IoU matrix for every combination (e.g. if there are 3 models, there will be 3 IoU matrices of model1xmodel2, model1xmodel3, model2xmodel3)\n* Create a graph\n* Store all predictions (only indices) from all models as nodes in the graph (e.g. model1pred1, model3pred300)\n* Add edges between nodes if that point in IoU matrix is higher than the given IoU threshold\n* Blend connected components\n\nIoU matrices can be calculated from masks or boxes, but using masks gave us a slight boost. IoU matrices are calculated with the code below.\n\n```\ndef get_iou_matrix_from_boxes(bounding_boxes1, bounding_boxes2):\n\n \"\"\"\n Calculate IoU matrix between two sets of bounding boxes\n \n Parameters\n ----------\n bounding_boxes1 [numpy.ndarray of shape (n_objects, 4)]: Bounding boxes\n bounding_boxes2 [numpy.ndarray of shape (m_objects, 4)]: Bounding boxes\n \n Returns\n -------\n iou_matrix [numpy.ndarray of shape (n_objects, m_objects)]: IoU matrix between two sets of bounding boxes\n \"\"\"\n\n bounding_boxes1_x1, bounding_boxes1_y1, bounding_boxes1_x2, bounding_boxes1_y2 = np.split(bounding_boxes1, 4, axis=1)\n bounding_boxes2_x1, bounding_boxes2_y1, bounding_boxes2_x2, bounding_boxes2_y2 = np.split(bounding_boxes2, 4, axis=1)\n\n xa = np.maximum(bounding_boxes1_x1, np.transpose(bounding_boxes2_x1))\n ya = np.maximum(bounding_boxes1_y1, np.transpose(bounding_boxes2_y1))\n xb = np.minimum(bounding_boxes1_x2, np.transpose(bounding_boxes2_x2))\n yb = np.minimum(bounding_boxes1_y2, np.transpose(bounding_boxes2_y2))\n\n inter_area = np.maximum((xb - xa + 1), 0) * np.maximum((yb - ya + 1), 0)\n box_a_area = (bounding_boxes1_x2 - bounding_boxes1_x1 + 1) * (bounding_boxes1_y2 - bounding_boxes1_y1 + 1)\n box_b_area = (bounding_boxes2_x2 - bounding_boxes2_x1 + 1) * (bounding_boxes2_y2 - bounding_boxes2_y1 + 1)\n iou_matrix = inter_area / (box_a_area + np.transpose(box_b_area) - inter_area)\n\n return iou_matrix\n\n\ndef get_iou_matrix_from_masks(masks1, masks2):\n \n \"\"\"\n Calculate IOU matrix between two sets of masks\n \n Parameters\n ----------\n masks1 [numpy.ndarray of shape (n_objects, height, width)]: 2d binary masks\n masks2 [numpy.ndarray of shape (m_objects, height, width)]: 2d binary masks\n \n Returns\n -------\n iou_matrix [numpy.ndarray of shape (n_objects, m_objects)]: IoU matrix between two sets of masks\n \"\"\"\n \n if len(list(masks1)) == 0 or len(list(masks2)) == 0:\n print(f'empty predictions - masks1 len {len(list(masks1))}, masks2 len {len(list(masks2))}')\n return np.array([[]])\n \n enc_masks1 = [mask_util.encode(np.asarray(p, order='F')) for p in (masks1 > 0.5).astype(np.uint8)]\n enc_masks2 = [mask_util.encode(np.asarray(p, order='F')) for p in (masks2 > 0.5).astype(np.uint8)]\n iou_matrix = mask_util.iou(enc_masks1, enc_masks2, [0] * len(enc_masks1))\n \n return iou_matrix\n```\n\nThis is the function used for blending multiple models' predictions. Using 0.9 iou_threshold for boxes and 0.7 iou_threshold for masks worked best for us. If more than 5 models are ensembled, it is a good choice to drop single components (not connected to any other node), because it worked like a voting mechanism under the hood.\n\n```\nimport networkx as nx\n\n\ndef blend_masks(prediction_boxes, prediction_masks, iou_threshold=0.9, label_threshold=0.5, iou_method='boxes', drop_single_components=True):\n\n \"\"\"\n Blend prediction masks of multiple models based on IoU\n \n Parameters\n ----------\n prediction_boxes [list of shape (n_models)]: Bounding box predictions of multiple models\n prediction_masks [list of shape (n_models)]: Mask predictions of multiple models\n iou_threshold (int): IoU threshold for blending masks (0 <= iou_threshold <= 1)\n iou_method (str): boxes or masks\n label_threshold (int): Label threshold for converting soft predictions to labels (0 <= iou_threshold <= 1)\n drop_single_components (bool): Whether to discard predictions without connections or not\n \n Returns\n -------\n blended_masks [numpy.ndarray of shape (n_objects, height, width)]: Blended binary masks\n \"\"\"\n\n iou_matrices = {}\n\n # Create all combinations of IoU matrices from given predictions\n for i in range(len(prediction_masks)):\n for j in range(i, len(prediction_masks)):\n if i == j:\n continue\n \n if iou_method == 'boxes':\n iou_matrix = get_iou_matrix_from_boxes(prediction_boxes[i], prediction_boxes[j])\n elif iou_method == 'masks':\n iou_matrix = get_iou_matrix_from_masks(prediction_masks[i], prediction_masks[j])\n \n iou_matrices[f'{i + 1}_{j + 1}'] = iou_matrix\n\n # Create a graph to store connected bounding boxes\n bounding_box_graph = nx.Graph()\n\n # Add all masks from all models as nodes\n for model_idx, boxes in enumerate(prediction_masks, start=1):\n nodes = [f'model{model_idx}_box{box_idx}' for box_idx in np.arange(len(boxes))]\n bounding_box_graph.add_nodes_from(nodes)\n \n del prediction_boxes\n\n # Add edges between nodes with IoU >= iou_threshold\n for model_combination, iou_matrix in iou_matrices.items():\n matching_boxes_idx = np.where(iou_matrix >= iou_threshold)\n model1_idx, model2_idx = model_combination.split('_')\n edges = [(f'model{model1_idx}_box{box1}', f'model{model2_idx}_box{box2}') for box1, box2 in zip(*matching_boxes_idx)]\n bounding_box_graph.add_edges_from(edges)\n\n del iou_matrices\n blended_masks = []\n\n for connections in nx.connected_components(bounding_box_graph):\n if len(connections) == 1:\n # Skip mask if its bounding isn't connected to any other bounding box\n if drop_single_components:\n continue\n else:\n # Append mask directly if its bounding box isn't connected to any other bounding box\n model_idx, box_idx = list(connections)[0].split('_')\n model_idx = int(model_idx.replace('model', ''))\n box_idx = int(box_idx.replace('box', ''))\n blended_masks.append(prediction_masks[model_idx - 1][box_idx])\n else:\n # Blend mask with its connections and append\n blended_mask = np.zeros((520, 704), dtype=np.float32)\n for connection in connections:\n model_idx, box_idx = connection.split('_')\n model_idx = int(model_idx.replace('model', ''))\n box_idx = int(box_idx.replace('box', ''))\n # Divide soft predictions with number of connections and accumulate on blended_mask\n blended_mask += (prediction_masks[model_idx - 1][box_idx] / len(connections))\n blended_masks.append(blended_mask)\n \n del prediction_masks, bounding_box_graph\n blended_masks = np.stack(blended_masks)\n # Convert soft predictions to binary labels\n blended_masks = np.uint8(blended_masks >= label_threshold)\n\n return blended_masks\n```\n\nThis was my first instance segmentation project and I learned a lot. It would be great if top teams can share their ensembling strategies as well. Congrats to winners and happy new year to everyone!", "1633907": "How much did your three models ensemble increase in terms of private LB? In our case, we had two model ensembles with a similar approach. But in the end, our single model actually had the best private LB, such a letdown 😭", "1633924": "I wasn't using 3 models. I gave that as an example. In my earlier experiments, I submit every fold of my mask r-cnn model individually and they scored between 0.316-320 on public LB, and my blend scored 0.323 on public LB. I can't find and compare their private LB scores since I didn't add descriptions to them.", "1633927": "I see, thank you for the info :)", "1634077": "Thank you @gunesevitan, it is a very interesting approach based on graphs. I already kept note of it for future competitions and projects :-)", "1634224": "An interesting ensemble strategy. 👍. In [our Solution](https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/297985), the ensemble process is divided into three steps. Firstly, we compute the box IOU for all predicted boxes( suppose N boxes in total) and get an IOU matrix N-by-N. For each row of the matrix, we select the boxes with IOU larger than a threshold and weighted their corresponding mask. The weighted mask will be converted into a bitmask, we then update the box by the bitmask. [NMW](https://github.com/ZFTurbo/Weighted-Boxes-Fusion/blob/master/ensemble_boxes/ensemble_boxes_nmw.py) is used to filter the proposed boxes from step one. Lastly, mask IOU is used to remove the overlap (Just like the NMW doses)." }, "source": "meta" }