{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n\n#
YOLOX + YOLOv5 Weighted Boxes Fusion Ensemble
\n\n
\n\n
\n
\n
\n\n","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.027817,"end_time":"2021-12-05T21:43:15.021008","exception":false,"start_time":"2021-12-05T21:43:14.993191","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"-----\n\n**Important Note:** Please note that the higest LB score is coming from an older version where a model in the ensemble had priority over the other (resulting in one model taking over most of the predictions). \nThe current version has the correct score (that is 0.447). Apologies for the inconvenience. \n\n-----","metadata":{}},{"cell_type":"markdown","source":"### How to ensemble object detection models?\n\nThis notebook shows how to detect starfish objects (COTS dataset) using multiple models on Kaggle (YOLOX & YOLOv5 for example). \n\n
\n\n**Original Notebooks Credits:**\nI have used the great models from [YoloX full training pipeline for COTS dataset](https://www.kaggle.com/remekkinas/yolox-full-training-pipeline-for-cots-dataset) and from [YoloX training pipeline COTS dataset [LB 0.507] !!\n](https://www.kaggle.com/remekkinas/yolox-training-pipeline-cots-dataset-lb-0-507).\nAnd the ensemble is from [WBF approach for ensemble](https://www.kaggle.com/shonenkov/wbf-approach-for-ensemble/notebook).\n
\n\n\nFollowing [the original notebook](https://www.kaggle.com/shonenkov/wbf-approach-for-ensemble/notebook), here is a super easy technique for bbox postprocessing ensembling object detection tasks.\n\nOriginal paper: [Weighted Boxes Fusion: ensembling boxes for object detection models](https://arxiv.org/pdf/1910.13302.pdf)\n\n\n**MAIN IDEA**\nWe know about NMS (Non-maximum Suppression) method and its [Soft-NMS](https://arxiv.org/pdf/1704.04503.pdf) extension. But on this notebook we will use WBF (Weighted Boxes Fusion), that gives really good boost in [Open Images Dataset](https://storage.googleapis.com/openimages/web/index.html) according to [paper](https://www.kaggleusercontent.com/kf/33938688/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..DAfPciYx8Oc7kJrKcYawig.uvFe9prASYZNcVo76gM8ut-7s5bGRQ-lUXSOWGU5EflkWsk-TpJKhYUObxe6ntB0_--EImt8-pehmt1tQEbdNeQPePCwwWz9fQhny16RsdDQvlz4dGFXVLMsW4lnbVjdZyhfmeXp-wLpf7yZPJ-9xQ9GnUDvCiy21i0O_V-RVFRlEUCvw9NDCBFawHNaOEGiRcltaPrN1fdlnfrj7OW-v8qYdMLE-rUjvl3Q19KHCkuh81EV7GgkHdHbeTXMG45hNAuuEo2WoFSvT7djDwxUvQUxPsGAPpZc7-hXrmw4IJFwtrhwqdatecFw6HD67Q8HObWPyCWtd5tihPK2gkMa7r7YSU4Nejgl293muJCMMlN6XnWfRKweK5T6z6df1-dTfSWxtUTHtGWV1RfGxT8NN-KV_7cpeGY2kMGRPdk4lbqreQCeGLjtbmUeBiNhgTX1acodY6JfN2kGSFPn15S22GqqQFOdKEaLnaNZGZ0c3IAHgyuJE4yfsAjC3wrFAPQz25y1u6E4k1e8yQlVklIhCdBKCl8zkvDffgXAzl-POm_N_AeW6zVPhVZ_86ipENK7eqcjD2VU5oxi2S7IclRdn_eEulBPrrhlpjl0E8axDoEkGE_R_uncF3uZQdAG6AYg2PwgoEGGxTo_uTk5ul3Op3CfdPwPC9NTvLZ4qxR9T8E.a8_2yDGoW9UBcfsWU7txuA/(https://arxiv.org/pdf/1910.13302.pdf).\n\n#### Weighted boxes fusion (WBF) Algorithm\n\n1. Each predicted box from each model is added to a single list B. The list is sorted in decreasing order of the confidence scores C.\n\n2. Create lists: L and F for boxes clusters and fused boxes. Each position in L contains a set of boxes (or single box), which form a cluster. Each position in F contains only one box, which is the fused box from the corresponding cluster in L. \n\n3. Iterate predicted boxes in B in cycle and try to find a matching box in the list F. Match is defined as a box with a large overlap with the box under question (IoU > thresh).\n\n4. If the match is not found, add the box from the list B to the end of lists L and F as new entries; proceed to the next box in the list B.\n\n5. If the match is found, add this box to the list L at the position pos corresponding to the matching box in the list F.\n\n6. Recalculate the box coordinates and confidence score in F[pos], using all T boxes accumulated in cluster L[pos]. (see the formula in the paper)\n\n7. After all boxes in B are processed, re-scale confidence scores in F list: multiply it by a number of boxes in a cluster and divide by a number of models N. (see the formula in the paper)\n\n\n**When WBF can be better than NMS or SoftNMS?**\nBoth NMS and Soft-NMS exclude some boxes, but WBF uses information from all boxes. It can fix some cases where all boxes are predicted inaccurate by all models. NMS will leave only one inaccurate box, while WBF will fix it using information from all 3 boxes (see the example in Fig. 1, red predictions, blue ground truth).\n\n\n![](https://i.ibb.co/d2P2pPL/2020-05-12-21-02-34.png)","metadata":{}},{"cell_type":"code","source":"import warnings; warnings.filterwarnings(\"ignore\")\n\nimport os\nimport cv2\nimport ast\nimport sys\nimport glob\nimport torch\nimport shutil\nimport importlib\nimport traceback\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\nfrom tqdm.notebook import tqdm\nimport matplotlib.pyplot as plt\nsys.path.append('../input/tensorflow-great-barrier-reef')\ntqdm.pandas()\n\nfrom PIL import Image\nfrom IPython.display import display","metadata":{"papermill":{"duration":1.669635,"end_time":"2021-12-05T21:43:16.721911","exception":false,"start_time":"2021-12-05T21:43:15.052276","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-30T12:53:31.740553Z","iopub.execute_input":"2021-12-30T12:53:31.741206Z","iopub.status.idle":"2021-12-30T12:53:33.232405Z","shell.execute_reply.started":"2021-12-30T12:53:31.741108Z","shell.execute_reply":"2021-12-30T12:53:33.231688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Installing YOLOX \n
Some Kaggle enviroment hacking :) due to competition limitation - no internet access during submission.
","metadata":{"papermill":{"duration":0.014038,"end_time":"2021-12-05T21:43:16.750638","exception":false,"start_time":"2021-12-05T21:43:16.7366","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%cp -r /kaggle/input/yolox-cots-models /kaggle/working/\n%cd /kaggle/working/yolox-cots-models/yolox-dep","metadata":{"papermill":{"duration":17.792999,"end_time":"2021-12-05T21:43:34.592784","exception":false,"start_time":"2021-12-05T21:43:16.799785","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-30T12:53:33.235614Z","iopub.execute_input":"2021-12-30T12:53:33.236204Z","iopub.status.idle":"2021-12-30T12:53:53.919591Z","shell.execute_reply.started":"2021-12-30T12:53:33.236152Z","shell.execute_reply":"2021-12-30T12:53:53.918756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Install YOLOX required modules\n!pip install pip-21.3.1-py3-none-any.whl -f ./ --no-index\n!pip install loguru-0.5.3-py3-none-any.whl -f ./ --no-index\n!pip install ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl -f ./ --no-index\n!pip install onnx-1.8.1-cp37-cp37m-manylinux2010_x86_64.whl -f ./ --no-index\n!pip install onnxruntime-1.8.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl -f ./ --no-index\n!pip install onnxoptimizer-0.2.6-cp37-cp37m-manylinux2014_x86_64.whl -f ./ --no-index\n!pip install thop-0.0.31.post2005241907-py3-none-any.whl -f ./ --no-index\n!pip install tabulate-0.8.9-py3-none-any.whl -f ./ --no-index","metadata":{"_kg_hide-output":true,"papermill":{"duration":74.488225,"end_time":"2021-12-05T21:44:49.096558","exception":false,"start_time":"2021-12-05T21:43:34.608333","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-30T12:53:53.923642Z","iopub.execute_input":"2021-12-30T12:53:53.924369Z","iopub.status.idle":"2021-12-30T12:55:07.199832Z","shell.execute_reply.started":"2021-12-30T12:53:53.924327Z","shell.execute_reply":"2021-12-30T12:55:07.198756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Install YOLOX\n%cd /kaggle/working/yolox-cots-models/YOLOX\n!pip install -r requirements.txt\n!pip install -v -e . ","metadata":{"_kg_hide-output":true,"papermill":{"duration":74.641611,"end_time":"2021-12-05T21:46:03.760925","exception":false,"start_time":"2021-12-05T21:44:49.119314","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-30T12:55:07.204036Z","iopub.execute_input":"2021-12-30T12:55:07.204282Z","iopub.status.idle":"2021-12-30T12:56:21.388931Z","shell.execute_reply.started":"2021-12-30T12:55:07.204251Z","shell.execute_reply":"2021-12-30T12:56:21.388039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Install CocoAPI tool\n%cd /kaggle/working/yolox-cots-models/yolox-dep/cocoapi/PythonAPI\n!make\n!make install\n!python setup.py install","metadata":{"_kg_hide-output":true,"papermill":{"duration":18.265912,"end_time":"2021-12-05T21:46:22.056603","exception":false,"start_time":"2021-12-05T21:46:03.790691","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-30T12:56:21.390707Z","iopub.execute_input":"2021-12-30T12:56:21.390973Z","iopub.status.idle":"2021-12-30T12:56:40.417991Z","shell.execute_reply.started":"2021-12-30T12:56:21.390938Z","shell.execute_reply":"2021-12-30T12:56:40.417181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pycocotools","metadata":{"papermill":{"duration":0.045381,"end_time":"2021-12-05T21:46:22.141156","exception":false,"start_time":"2021-12-05T21:46:22.095775","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-30T12:56:40.419573Z","iopub.execute_input":"2021-12-30T12:56:40.420358Z","iopub.status.idle":"2021-12-30T12:56:40.425860Z","shell.execute_reply.started":"2021-12-30T12:56:40.420317Z","shell.execute_reply":"2021-12-30T12:56:40.425184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Test Model","metadata":{"papermill":{"duration":0.036659,"end_time":"2021-12-05T21:46:22.214827","exception":false,"start_time":"2021-12-05T21:46:22.178168","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%cd /kaggle/working/yolox-cots-models/YOLOX\n\nCHECKPOINT_FILE = '/kaggle/working/yolox-cots-models/yx_l_003.pth'","metadata":{"papermill":{"duration":0.046709,"end_time":"2021-12-05T21:46:22.298652","exception":false,"start_time":"2021-12-05T21:46:22.251943","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-30T12:56:40.427293Z","iopub.execute_input":"2021-12-30T12:56:40.427552Z","iopub.status.idle":"2021-12-30T12:56:40.446617Z","shell.execute_reply.started":"2021-12-30T12:56:40.427518Z","shell.execute_reply":"2021-12-30T12:56:40.445728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"config_file_template = '''\n\n#!/usr/bin/env python3\n# -*- coding:utf-8 -*-\n# Copyright (c) Megvii, Inc. and its affiliates.\n\nimport os\n\nfrom yolox.exp import Exp as MyExp\n\n\nclass Exp(MyExp):\n def __init__(self):\n super(Exp, self).__init__()\n self.depth = 1\n self.width = 1\n self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(\".\")[0]\n self.num_classes = 1\n\n'''\n\nwith open('cots_config.py', 'w') as f:\n f.write(config_file_template)","metadata":{"papermill":{"duration":0.044275,"end_time":"2021-12-05T21:46:22.380801","exception":false,"start_time":"2021-12-05T21:46:22.336526","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-30T12:56:40.447680Z","iopub.execute_input":"2021-12-30T12:56:40.449331Z","iopub.status.idle":"2021-12-30T12:56:40.455269Z","shell.execute_reply.started":"2021-12-30T12:56:40.449298Z","shell.execute_reply":"2021-12-30T12:56:40.454460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from yolox.utils import postprocess\nfrom yolox.data.data_augment import ValTransform\n\nCOCO_CLASSES = (\n \"starfish\",\n)\n\n# get YOLOX experiment\ncurrent_exp = importlib.import_module('cots_config')\nexp = current_exp.Exp()\n\n# set inference parameters\ntest_size = (800, 1280)\nnum_classes = 1\nconfthre = 0.1\nnmsthre = 0.4\n\n\n# get YOLOX model\nyolox_model = exp.get_model()\nyolox_model.cuda()\nyolox_model.eval()\n\n# get custom trained checkpoint\nckpt_file = CHECKPOINT_FILE\nckpt = torch.load(ckpt_file, map_location=\"cpu\")\nyolox_model.load_state_dict(ckpt[\"model\"])","metadata":{"papermill":{"duration":4.184097,"end_time":"2021-12-05T21:46:26.601756","exception":false,"start_time":"2021-12-05T21:46:22.417659","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-30T12:56:40.456636Z","iopub.execute_input":"2021-12-30T12:56:40.457512Z","iopub.status.idle":"2021-12-30T12:56:44.749421Z","shell.execute_reply.started":"2021-12-30T12:56:40.457477Z","shell.execute_reply":"2021-12-30T12:56:44.748644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def yolox_inference(img, model, test_size): \n bboxes = []\n bbclasses = []\n scores = []\n \n preproc = ValTransform(legacy = False)\n\n tensor_img, _ = preproc(img, None, test_size)\n tensor_img = torch.from_numpy(tensor_img).unsqueeze(0)\n tensor_img = tensor_img.float()\n tensor_img = tensor_img.cuda()\n\n with torch.no_grad():\n outputs = model(tensor_img)\n outputs = postprocess(\n outputs, num_classes, confthre,\n nmsthre, class_agnostic=True\n )\n\n if outputs[0] is None:\n return [], [], []\n \n outputs = outputs[0].cpu()\n bboxes = outputs[:, 0:4]\n\n bboxes /= min(test_size[0] / img.shape[0], test_size[1] / img.shape[1])\n bbclasses = outputs[:, 6]\n scores = outputs[:, 4] * outputs[:, 5]\n \n return bboxes, bbclasses, scores","metadata":{"papermill":{"duration":0.051006,"end_time":"2021-12-05T21:46:26.690736","exception":false,"start_time":"2021-12-05T21:46:26.63973","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-30T12:56:44.752725Z","iopub.execute_input":"2021-12-30T12:56:44.752933Z","iopub.status.idle":"2021-12-30T12:56:44.761641Z","shell.execute_reply.started":"2021-12-30T12:56:44.752909Z","shell.execute_reply":"2021-12-30T12:56:44.760729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def draw_yolox_predictions(img, bboxes, scores, bbclasses, confthre, classes_dict):\n for i in range(len(bboxes)):\n box = bboxes[i]\n cls_id = int(bbclasses[i])\n score = scores[i]\n if score < confthre:\n continue\n x0 = int(box[0])\n y0 = int(box[1])\n x1 = int(box[2])\n y1 = int(box[3])\n\n cv2.rectangle(img, (x0, y0), (x1, y1), (0, 255, 0), 2)\n cv2.putText(img, '{}:{:.1f}%'.format(classes_dict[cls_id], score * 100), (x0, y0 - 3), cv2.FONT_HERSHEY_PLAIN, 0.8, (0,255,0), thickness = 1)\n return img","metadata":{"papermill":{"duration":0.046822,"end_time":"2021-12-05T21:46:26.774811","exception":false,"start_time":"2021-12-05T21:46:26.727989","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-30T12:56:44.762963Z","iopub.execute_input":"2021-12-30T12:56:44.763542Z","iopub.status.idle":"2021-12-30T12:56:44.773200Z","shell.execute_reply.started":"2021-12-30T12:56:44.763504Z","shell.execute_reply":"2021-12-30T12:56:44.772429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# YOLOv5 Model","metadata":{}},{"cell_type":"code","source":"ROOT_DIR = '/kaggle/input/tensorflow-great-barrier-reef/'\nCKPT_PATH = '/kaggle/input/greatbarrierreef-yolov5-train-ds/yolov5/runs/train/exp/weights/best.pt'\nIMG_SIZE = 1280\nCONF = 0.15\nIOU = 0.50\nAUGMENT = False","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-30T12:56:44.774625Z","iopub.execute_input":"2021-12-30T12:56:44.774940Z","iopub.status.idle":"2021-12-30T12:56:44.786939Z","shell.execute_reply.started":"2021-12-30T12:56:44.774904Z","shell.execute_reply":"2021-12-30T12:56:44.785922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_path(row):\n row['image_path'] = f'{ROOT_DIR}/train_images/video_{row.video_id}/{row.video_frame}.jpg'\n return row","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-30T12:56:44.789054Z","iopub.execute_input":"2021-12-30T12:56:44.789806Z","iopub.status.idle":"2021-12-30T12:56:44.796667Z","shell.execute_reply.started":"2021-12-30T12:56:44.789761Z","shell.execute_reply":"2021-12-30T12:56:44.795922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train Data\ndf = pd.read_csv(f'{ROOT_DIR}/train.csv')\ndf = df.progress_apply(get_path, axis=1)\ndf['annotations'] = df['annotations'].progress_apply(lambda x: ast.literal_eval(x))","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-30T12:56:44.798172Z","iopub.execute_input":"2021-12-30T12:56:44.798427Z","iopub.status.idle":"2021-12-30T12:57:00.775776Z","shell.execute_reply.started":"2021-12-30T12:56:44.798393Z","shell.execute_reply":"2021-12-30T12:57:00.775121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['num_bbox'] = df['annotations'].progress_apply(lambda x: len(x))\ndata = (df.num_bbox>0).value_counts()/len(df)*100","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-30T12:57:00.781307Z","iopub.execute_input":"2021-12-30T12:57:00.783512Z","iopub.status.idle":"2021-12-30T12:57:00.881042Z","shell.execute_reply.started":"2021-12-30T12:57:00.783426Z","shell.execute_reply":"2021-12-30T12:57:00.880358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def voc2yolo(bboxes, image_height=720, image_width=1280):\n \"\"\"\n voc => [x1, y1, x2, y1]\n yolo => [xmid, ymid, w, h] (normalized)\n \"\"\"\n bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n bboxes[..., [0, 2]] = bboxes[..., [0, 2]]/ image_width\n bboxes[..., [1, 3]] = bboxes[..., [1, 3]]/ image_height\n w = bboxes[..., 2] - bboxes[..., 0]\n h = bboxes[..., 3] - bboxes[..., 1]\n bboxes[..., 0] = bboxes[..., 0] + w/2\n bboxes[..., 1] = bboxes[..., 1] + h/2\n bboxes[..., 2] = w\n bboxes[..., 3] = h\n return bboxes\n\ndef yolo2voc(bboxes, image_height=720, image_width=1280):\n \"\"\"\n yolo => [xmid, ymid, w, h] (normalized)\n voc => [x1, y1, x2, y1]\n\n \"\"\"\n bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n bboxes[..., [0, 2]] = bboxes[..., [0, 2]]* image_width\n bboxes[..., [1, 3]] = bboxes[..., [1, 3]]* image_height\n bboxes[..., [0, 1]] = bboxes[..., [0, 1]] - bboxes[..., [2, 3]]/2\n bboxes[..., [2, 3]] = bboxes[..., [0, 1]] + bboxes[..., [2, 3]]\n return bboxes\n\ndef coco2yolo(bboxes, image_height=720, image_width=1280):\n \"\"\"\n coco => [xmin, ymin, w, h]\n yolo => [xmid, ymid, w, h] (normalized)\n \"\"\"\n bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n # normolizinig\n bboxes[..., [0, 2]]= bboxes[..., [0, 2]]/ image_width\n bboxes[..., [1, 3]]= bboxes[..., [1, 3]]/ image_height\n # converstion (xmin, ymin) => (xmid, ymid)\n bboxes[..., [0, 1]] = bboxes[..., [0, 1]] + bboxes[..., [2, 3]]/2\n return bboxes\n\ndef yolo2coco(bboxes, image_height=720, image_width=1280):\n \"\"\"\n yolo => [xmid, ymid, w, h] (normalized)\n coco => [xmin, ymin, w, h]\n \"\"\"\n bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n # denormalizing\n bboxes[..., [0, 2]]= bboxes[..., [0, 2]]* image_width\n bboxes[..., [1, 3]]= bboxes[..., [1, 3]]* image_height\n # converstion (xmid, ymid) => (xmin, ymin)\n bboxes[..., [0, 1]] = bboxes[..., [0, 1]] - bboxes[..., [2, 3]]/2\n return bboxes\n\ndef voc2coco(bboxes, image_height=720, image_width=1280):\n bboxes = voc2yolo(bboxes, image_height, image_width)\n bboxes = yolo2coco(bboxes, image_height, image_width)\n return bboxes\n\ndef load_image(image_path):\n return cv2.cvtColor(cv2.imread(image_path), cv2.COLOR_BGR2RGB)\n\ndef plot_one_box(x, img, color=None, label=None, line_thickness=None):\n # Plots one bounding box on image img\n tl = line_thickness or round(0.002 * (img.shape[0] + img.shape[1]) / 2) + 1 # line/font thickness\n color = color or [random.randint(0, 255) for _ in range(3)]\n c1, c2 = (int(x[0]), int(x[1])), (int(x[2]), int(x[3]))\n cv2.rectangle(img, c1, c2, color, thickness=tl, lineType=cv2.LINE_AA)\n if label:\n tf = max(tl - 1, 1) # font thickness\n t_size = cv2.getTextSize(label, 0, fontScale=tl / 3, thickness=tf)[0]\n c2 = c1[0] + t_size[0], c1[1] - t_size[1] - 3\n cv2.rectangle(img, c1, c2, color, -1, cv2.LINE_AA) # filled\n cv2.putText(img, label, (c1[0], c1[1] - 2), 0, tl / 3, [225, 255, 255], thickness=tf, lineType=cv2.LINE_AA)\n\ndef draw_bboxes(img, bboxes, classes, class_ids, colors = None, show_classes = None, bbox_format = 'yolo', class_name = False, line_thickness = 2):\n\n image = img.copy()\n show_classes = classes if show_classes is None else show_classes\n colors = (0, 255 ,0) if colors is None else colors\n\n if bbox_format == 'yolo':\n\n for idx in range(len(bboxes)):\n\n bbox = bboxes[idx]\n cls = classes[idx]\n cls_id = class_ids[idx]\n color = colors[cls_id] if type(colors) is list else colors\n\n if cls in show_classes:\n\n x1 = round(float(bbox[0])*image.shape[1])\n y1 = round(float(bbox[1])*image.shape[0])\n w = round(float(bbox[2])*image.shape[1]/2) #w/2\n h = round(float(bbox[3])*image.shape[0]/2)\n\n voc_bbox = (x1-w, y1-h, x1+w, y1+h)\n plot_one_box(voc_bbox,\n image,\n color = color,\n label = cls if class_name else str(get_label(cls)),\n line_thickness = line_thickness)\n\n elif bbox_format == 'coco':\n\n for idx in range(len(bboxes)):\n\n bbox = bboxes[idx]\n cls = classes[idx]\n cls_id = class_ids[idx]\n color = colors[cls_id] if type(colors) is list else colors\n\n if cls in show_classes:\n x1 = int(round(bbox[0]))\n y1 = int(round(bbox[1]))\n w = int(round(bbox[2]))\n h = int(round(bbox[3]))\n\n voc_bbox = (x1, y1, x1+w, y1+h)\n plot_one_box(voc_bbox,\n image,\n color = color,\n label = cls if class_name else str(cls_id),\n line_thickness = line_thickness)\n\n elif bbox_format == 'voc_pascal':\n\n for idx in range(len(bboxes)):\n\n bbox = bboxes[idx]\n cls = classes[idx]\n cls_id = class_ids[idx]\n color = colors[cls_id] if type(colors) is list else colors\n\n if cls in show_classes:\n x1 = int(round(bbox[0]))\n y1 = int(round(bbox[1]))\n x2 = int(round(bbox[2]))\n y2 = int(round(bbox[3]))\n voc_bbox = (x1, y1, x2, y2)\n plot_one_box(voc_bbox,\n image,\n color = color,\n label = cls if class_name else str(cls_id),\n line_thickness = line_thickness)\n else:\n raise ValueError('wrong bbox format')\n\n return image\n\ndef get_bbox(annots):\n bboxes = [list(annot.values()) for annot in annots]\n return bboxes\n\ndef get_imgsize(row):\n row['width'], row['height'] = imagesize.get(row['image_path'])\n return row\n\nnp.random.seed(32)\ncolors = [(np.random.randint(255), np.random.randint(255), np.random.randint(255))\\\n for idx in range(1)]","metadata":{"pycharm":{"name":"#%%\n"},"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-30T12:57:00.882498Z","iopub.execute_input":"2021-12-30T12:57:00.882879Z","iopub.status.idle":"2021-12-30T12:57:00.920196Z","shell.execute_reply.started":"2021-12-30T12:57:00.882839Z","shell.execute_reply":"2021-12-30T12:57:00.919317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!mkdir -p /root/.config/Ultralytics\n!cp /kaggle/input/yolov5-font/Arial.ttf /root/.config/Ultralytics/","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-30T12:57:00.921738Z","iopub.execute_input":"2021-12-30T12:57:00.922066Z","iopub.status.idle":"2021-12-30T12:57:01.254097Z","shell.execute_reply.started":"2021-12-30T12:57:00.922029Z","shell.execute_reply":"2021-12-30T12:57:01.253146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_model(ckpt_path, conf=0.25, iou=0.50):\n model = torch.hub.load('/kaggle/input/yolov5-lib-ds',\n 'custom',\n path=ckpt_path,\n source='local',\n force_reload=True) # local repo\n model.conf = conf # NMS confidence threshold\n model.iou = iou # NMS IoU threshold\n model.classes = None # (optional list) filter by class, i.e. = [0, 15, 16] for persons, cats and dogs\n model.multi_label = False # NMS multiple labels per box\n model.max_det = 1000 # maximum number of detections per image\n return model","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-30T12:57:01.255783Z","iopub.execute_input":"2021-12-30T12:57:01.256061Z","iopub.status.idle":"2021-12-30T12:57:01.262982Z","shell.execute_reply.started":"2021-12-30T12:57:01.256025Z","shell.execute_reply":"2021-12-30T12:57:01.262284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def predict(model, img, size=768, augment=False):\n height, width = img.shape[:2]\n results = model(img, size=size, augment=augment) # custom inference size\n preds = results.pandas().xyxy[0]\n bboxes = preds[['xmin','ymin','xmax','ymax']].values\n if len(bboxes):\n bboxes = voc2coco(bboxes,height,width).astype(int)\n confs = preds.confidence.values\n return bboxes, confs\n else:\n return [],[]\n\ndef format_prediction(bboxes, confs):\n annot = ''\n if len(bboxes)>0:\n for idx in range(len(bboxes)):\n xmin, ymin, w, h = bboxes[idx]\n conf = confs[idx]\n annot += f'{conf} {xmin} {ymin} {w} {h}'\n annot +=' '\n annot = annot.strip(' ')\n return annot\n\ndef show_img(img, bboxes, bbox_format='yolo', bbox_colors = None):\n names = ['starfish']*len(bboxes)\n labels = [0]*len(bboxes)\n img = draw_bboxes(img = img,\n bboxes = bboxes,\n classes = names,\n class_ids = labels,\n class_name = True,\n colors = colors if bbox_colors is None else bbox_colors,\n bbox_format = bbox_format,\n line_thickness = 2)\n return Image.fromarray(img).resize((800, 400))","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-30T12:57:01.264737Z","iopub.execute_input":"2021-12-30T12:57:01.265352Z","iopub.status.idle":"2021-12-30T12:57:01.277503Z","shell.execute_reply.started":"2021-12-30T12:57:01.265313Z","shell.execute_reply":"2021-12-30T12:57:01.276736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Ensembling","metadata":{}},{"cell_type":"markdown","source":"## Let's look at some train images!","metadata":{}},{"cell_type":"code","source":"import sys; sys.path.append('/kaggle/input/weightedboxesfusion/')\n# !pip install --no-deps '/kaggle/input/weightedboxesfusion/' > /dev/null","metadata":{"execution":{"iopub.status.busy":"2021-12-30T12:57:01.278969Z","iopub.execute_input":"2021-12-30T12:57:01.279343Z","iopub.status.idle":"2021-12-30T12:57:01.287239Z","shell.execute_reply.started":"2021-12-30T12:57:01.279207Z","shell.execute_reply":"2021-12-30T12:57:01.286364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TEST_IMAGE_PATH = \"/kaggle/input/tensorflow-great-barrier-reef/train_images/video_0/9674.jpg\"\n\nfrom ensemble_boxes import *\n\ndef run_wbf(bboxes, confs, image_size=512, iou_thr=0.55, skip_box_thr=0.7, weights=None):\n boxes = [bbox/(image_size-1) for bbox in bboxes]\n scores = [conf for conf in confs] \n labels = [np.ones(conf.shape[0]) for conf in confs]\n boxes, scores, labels = weighted_boxes_fusion(boxes, scores, labels, weights=None, iou_thr=iou_thr, skip_box_thr=skip_box_thr)\n boxes = boxes*(image_size-1)\n return boxes, scores, labels\n\nmodel = load_model(CKPT_PATH, conf=CONF, iou=IOU)\nimage_paths = df[df.num_bbox>1].sample(100).image_path.tolist()\nfor idx, path in enumerate(image_paths):\n img = cv2.imread(path)[...,::-1]\n \n bboxes_1, bbclasses, scores = yolox_inference(img[...,::-1], yolox_model, test_size) \n bboxes_2, confis = predict(model, img, size=IMG_SIZE, augment=AUGMENT) \n pred_1, pred_2 = voc2coco(bboxes_1.detach().numpy(),img.shape[1],img.shape[2]).astype(int), bboxes_2\n if len(pred_1) > 0 and len(pred_2) > 0: boxes, scores, labels = run_wbf([pred_1, pred_2], [scores, confis], image_size = IMG_SIZE)\n elif len(pred_1) > 0: boxes, scores = bboxes_1.detach().numpy(), scores\n elif len(pred_2) > 0: boxes, scores = bboxes_2.detach().numpy(), confis\n \n \n if len(pred_1) > 0: \n print('\\n\\nYOLOX Predictions: ')\n display(show_img(img, voc2coco(bboxes_1.detach().numpy(),img.shape[1],img.shape[2]).astype(int), bbox_format='coco'))\n if len(pred_2) > 0: \n print('\\n\\nYoloV5 Predictions: ')\n display(show_img(img, bboxes_2, bbox_format='coco')) \n print('\\n\\nEnsemble (WBF) Predictions: ')\n display(show_img(img, boxes, bbox_format='coco'))\n \n if idx>5:\n break","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-30T12:57:01.288362Z","iopub.execute_input":"2021-12-30T12:57:01.288708Z","iopub.status.idle":"2021-12-30T12:57:13.364881Z","shell.execute_reply.started":"2021-12-30T12:57:01.288668Z","shell.execute_reply":"2021-12-30T12:57:13.364164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SUBMIT PREDICTION TO COMPETITION","metadata":{"papermill":{"duration":0.095501,"end_time":"2021-12-05T21:46:32.303063","exception":false,"start_time":"2021-12-05T21:46:32.207562","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%cd /kaggle/working/","metadata":{"papermill":{"duration":0.104486,"end_time":"2021-12-05T21:46:32.501117","exception":false,"start_time":"2021-12-05T21:46:32.396631","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-30T12:57:13.366427Z","iopub.execute_input":"2021-12-30T12:57:13.366841Z","iopub.status.idle":"2021-12-30T12:57:13.374881Z","shell.execute_reply.started":"2021-12-30T12:57:13.366802Z","shell.execute_reply":"2021-12-30T12:57:13.374210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import greatbarrierreef\n\nenv = greatbarrierreef.make_env() # initialize the environment\niter_test = env.iter_test() ","metadata":{"papermill":{"duration":0.133428,"end_time":"2021-12-05T21:46:32.729017","exception":false,"start_time":"2021-12-05T21:46:32.595589","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-30T12:57:13.376254Z","iopub.execute_input":"2021-12-30T12:57:13.376932Z","iopub.status.idle":"2021-12-30T12:57:13.405852Z","shell.execute_reply.started":"2021-12-30T12:57:13.376896Z","shell.execute_reply":"2021-12-30T12:57:13.405075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_dict = {\n 'id': [],\n 'prediction_string': [],\n}\n\nfor (image_np, sample_prediction_df) in iter_test:\n try: \n bboxes_1, bbclasses, scores = yolox_inference(image_np[...,::-1], yolox_model, test_size)\n bboxes_2, confis = predict(model, image_np, size=IMG_SIZE, augment=AUGMENT) \n\n if len(bboxes_1) > 0 and len(bboxes_2) > 0: \n pred_1, pred_2 = voc2coco(bboxes_1.detach().numpy(), image_np.shape[1],image_np.shape[2]).astype(int), bboxes_2\n bboxes, scores, labels = run_wbf([pred_1, pred_2], [scores, confis], image_size = IMG_SIZE)\n elif len(bboxes_1) > 0: bboxes, scores = voc2coco(bboxes_1.detach().numpy(), image_np.shape[1], image_np.shape[2]).astype(int), scores \n elif len(bboxes_2) > 0: bboxes, scores = bboxes_2, confis \n else: bboxes = []\n # display(show_img(image_np, bboxes, bbox_format='coco')) \n\n predictions = []\n for i in range(len(bboxes)):\n box = bboxes[i] \n score = scores[i]\n \n if score > 0.1:\n \n x_min = int(box[0])\n y_min = int(box[1])\n bbox_width = int(box[2])\n bbox_height = int(box[3])\n\n predictions.append('{:.2f} {} {} {} {}'.format(score, x_min, y_min, bbox_width, bbox_height))\n\n prediction_str = ' '.join(predictions)\n except:\n traceback.print_exc()\n predictions = []\n sample_prediction_df['annotations'] = prediction_str\n env.predict(sample_prediction_df)\n\n print('Prediction:', prediction_str)","metadata":{"papermill":{"duration":0.674293,"end_time":"2021-12-05T21:46:33.496659","exception":false,"start_time":"2021-12-05T21:46:32.822366","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-30T12:57:13.407057Z","iopub.execute_input":"2021-12-30T12:57:13.407772Z","iopub.status.idle":"2021-12-30T12:57:14.058884Z","shell.execute_reply.started":"2021-12-30T12:57:13.407735Z","shell.execute_reply":"2021-12-30T12:57:14.058187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df = pd.read_csv('submission.csv')\nsub_df.head()","metadata":{"papermill":{"duration":0.1231,"end_time":"2021-12-05T21:46:33.726446","exception":false,"start_time":"2021-12-05T21:46:33.603346","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-30T12:57:14.060378Z","iopub.execute_input":"2021-12-30T12:57:14.060659Z","iopub.status.idle":"2021-12-30T12:57:14.076830Z","shell.execute_reply.started":"2021-12-30T12:57:14.060615Z","shell.execute_reply":"2021-12-30T12:57:14.075989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n Find this notebook helpful? :) Upvoting is FREE ;) \n
","metadata":{"papermill":{"duration":0.103554,"end_time":"2021-12-05T21:46:33.93281","exception":false,"start_time":"2021-12-05T21:46:33.829256","status":"completed"},"tags":[]}}]}