{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# YOLOX detections + TRACKING submission made on COTS dataset (PART 2 - DETECTION)\n\nThis notebook based on the great work [YoloX inference on Kaggle for COTS [LB 0.507] !!!](https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507), please upvote it also! Weights from here [YoloX inference on Kaggle for COTS [LB 0.520+]](https://www.kaggle.com/dragonzhang/yolox-inference-on-kaggle-for-cots-lb-0-520). \n\nLB without tracking: 0.53   \nLB with tracking: 0.539\n\nIf you like this work, please upvote :)\n\nI've added online tracking algorithm based on previous frames detections using [norfair](https://github.com/tryolabs/norfair) library to increase recall and F2 score.   \nYou can find tracking code at inference loop cell. I've added comments with description how it works on inference loop, feel free to ask any questions at comments.   \nIt's just baseline, you can tune parameters and achieve better score. Good luck!  \n \n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"This notebook shows how to detect starfish objects (COTS dataset) using YOLOX ON  Kaggle. First part - Building Cutom Model on Kaggle using YOLOX I implmeneted in notebook called [YoloX full training pipeline for COTS dataset](https://www.kaggle.com/remekkinas/yolox-full-training-pipeline-for-cots-dataset). It could be good starting point for build own custom model based on YOLOX detector. Full github repository you can find here - [YOLOX](https://github.com/Megvii-BaseDetection/YOLOX)\n\n<div align = 'center'><img src='https://github.com/Megvii-BaseDetection/YOLOX/raw/main/assets/logo.png'/></div>\n\n<div class=\"alert alert-success\" role=\"alert\">\nThis work consists of two parts:     \n    <ul>\n        <li> <a href=\"https://www.kaggle.com/remekkinas/yolox-full-training-pipeline-for-cots-dataset\">YoloX full training pipeline for COTS dataset</a></li>\n        <li> YOLOX detections submission made on COTS dataset</li>\n    </ul>\n    \n</div>\n\n<div class=\"alert alert-warning\" role=\"alert\"><strong><ul><li>This is DEOMO only! What does it mean? Inference is made so far on weak model - trained only on 20 epochs.</li><li>I really appreciate if you <u>vote on both of these notebooks</u> - thank you! I just share my work to make competition fun and more interesting.</li></ul> </strong></div>\n","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport os\nimport torch\nimport importlib\nimport cv2 \nimport pandas as pd\n\nfrom PIL import Image\nfrom IPython.display import display","metadata":{"execution":{"iopub.status.busy":"2021-12-22T17:41:22.56693Z","iopub.execute_input":"2021-12-22T17:41:22.569677Z","iopub.status.idle":"2021-12-22T17:41:24.331715Z","shell.execute_reply.started":"2021-12-22T17:41:22.569551Z","shell.execute_reply":"2021-12-22T17:41:24.330978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### INSTALL YOLOX \n<div class=\"alert alert-warning\" role=\"alert\"><strong>It unfortunately requires a lot of Kaggle enviroment hacking :) due to competition limitation - no internet access during submission.</strong></div>","metadata":{}},{"cell_type":"code","source":"# download required packages - first time when I created database (https://www.kaggle.com/remekkinas/yolox-cots-models) with required moduls for YOLOX\n# don't use this section of code until Kaggle doesn't change something in the environment (!!)\n\n\n#%mkdir /kaggle/working/yolox-dep\n#!pip download pip -d \"/kaggle/working/yolox-dep\"\n#!pip download loguru -d \"/kaggle/working/yolox-dep\"\n#!pip download ninja -d \"/kaggle/working/yolox-dep\"\n#!pip download onnx==\"1.8.1\" -d \"/kaggle/working/yolox-dep\"\n#!pip download onnxruntime==\"1.8.0\" -d \"/kaggle/working/yolox-dep\"\n#!pip download onnxoptimizer>=\"0.2.5\" -d \"/kaggle/working/yolox-dep\"\n#!pip download thop -d \"/kaggle/working/yolox-dep\"\n#!pip download tabulate -d \"/kaggle/working/yolox-dep\"\n#!pip download onnx-simplifier==0.3.5 -d \"/kaggle/working/yolox-dep\"","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-22T17:41:24.333469Z","iopub.execute_input":"2021-12-22T17:41:24.333706Z","iopub.status.idle":"2021-12-22T17:41:24.338383Z","shell.execute_reply.started":"2021-12-22T17:41:24.333673Z","shell.execute_reply":"2021-12-22T17:41:24.336974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Copy YOLOX and required modules from local repository (Kaggle dataset -> https://www.kaggle.com/remekkinas/yolox-cots-models)\n%cp -r /kaggle/input/yolox-cots-models /kaggle/working/\n%cd /kaggle/working/yolox-cots-models/yolox-dep","metadata":{"execution":{"iopub.status.busy":"2021-12-22T17:41:24.339691Z","iopub.execute_input":"2021-12-22T17:41:24.339936Z","iopub.status.idle":"2021-12-22T17:41:43.946986Z","shell.execute_reply.started":"2021-12-22T17:41:24.339904Z","shell.execute_reply":"2021-12-22T17:41:43.946174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Install YOLOX required modules\n\n!pip install pip-21.3.1-py3-none-any.whl -f ./ --no-index\n!pip install loguru-0.5.3-py3-none-any.whl -f ./ --no-index\n!pip install ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl -f ./ --no-index\n!pip install onnx-1.8.1-cp37-cp37m-manylinux2010_x86_64.whl -f ./ --no-index\n!pip install onnxruntime-1.8.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl -f ./ --no-index\n!pip install onnxoptimizer-0.2.6-cp37-cp37m-manylinux2014_x86_64.whl -f ./ --no-index\n!pip install thop-0.0.31.post2005241907-py3-none-any.whl -f ./ --no-index\n!pip install tabulate-0.8.9-py3-none-any.whl -f ./ --no-index\n#!pip install onnx-simplifier-0.3.6.tar.gz -f ./ --no-index","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-22T17:41:43.949766Z","iopub.execute_input":"2021-12-22T17:41:43.950062Z","iopub.status.idle":"2021-12-22T17:42:57.891849Z","shell.execute_reply.started":"2021-12-22T17:41:43.950025Z","shell.execute_reply":"2021-12-22T17:42:57.891037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Install YOLOX\n%cd /kaggle/working/yolox-cots-models/YOLOX\n!pip install -r requirements.txt\n!pip install -v -e . ","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-22T17:42:57.893403Z","iopub.execute_input":"2021-12-22T17:42:57.894065Z","iopub.status.idle":"2021-12-22T17:44:11.747825Z","shell.execute_reply.started":"2021-12-22T17:42:57.894028Z","shell.execute_reply":"2021-12-22T17:44:11.746992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Install CocoAPI tool\n%cd /kaggle/working/yolox-cots-models/yolox-dep/cocoapi/PythonAPI\n\n!make\n!make install\n!python setup.py install","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-22T17:44:11.749676Z","iopub.execute_input":"2021-12-22T17:44:11.749953Z","iopub.status.idle":"2021-12-22T17:44:29.969314Z","shell.execute_reply.started":"2021-12-22T17:44:11.749913Z","shell.execute_reply":"2021-12-22T17:44:29.968395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pycocotools","metadata":{"execution":{"iopub.status.busy":"2021-12-22T17:44:29.971583Z","iopub.execute_input":"2021-12-22T17:44:29.972004Z","iopub.status.idle":"2021-12-22T17:44:29.978144Z","shell.execute_reply.started":"2021-12-22T17:44:29.971964Z","shell.execute_reply":"2021-12-22T17:44:29.977425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# norfair dependencies\n%cd /kaggle/input/norfair031py3/\n!pip install commonmark-0.9.1-py2.py3-none-any.whl -f ./ --no-index\n!pip install rich-9.13.0-py3-none-any.whl\n\n!mkdir /kaggle/working/tmp\n!cp -r /kaggle/input/norfair031py3/filterpy-1.4.5/filterpy-1.4.5/ /kaggle/working/tmp/\n%cd /kaggle/working/tmp/filterpy-1.4.5/\n!pip install .\n!rm -rf /kaggle/working/tmp\n\n# norfair\n%cd /kaggle/input/norfair031py3/\n!pip install norfair-0.3.1-py3-none-any.whl -f ./ --no-index","metadata":{"execution":{"iopub.status.busy":"2021-12-22T17:44:29.97956Z","iopub.execute_input":"2021-12-22T17:44:29.979866Z","iopub.status.idle":"2021-12-22T17:45:43.699103Z","shell.execute_reply.started":"2021-12-22T17:44:29.979818Z","shell.execute_reply":"2021-12-22T17:45:43.698253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### TEST MODEL - MAKE INFERENCE ON SAMPLE DATA\n","metadata":{}},{"cell_type":"code","source":"%cd /kaggle/working/yolox-cots-models/YOLOX\n\nCHECKPOINT_FILE = '/kaggle/input/f0-25-yolox-pth/YOLOX_outputs/cots_config/best_ckpt.pth'","metadata":{"execution":{"iopub.status.busy":"2021-12-22T17:45:43.702705Z","iopub.execute_input":"2021-12-22T17:45:43.702933Z","iopub.status.idle":"2021-12-22T17:45:43.711875Z","shell.execute_reply.started":"2021-12-22T17:45:43.702902Z","shell.execute_reply":"2021-12-22T17:45:43.711104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"config_file_template = '''\n\n#!/usr/bin/env python3\n# -*- coding:utf-8 -*-\n# Copyright (c) Megvii, Inc. and its affiliates.\n\nimport os\n\nfrom yolox.exp import Exp as MyExp\n\n\nclass Exp(MyExp):\n    def __init__(self):\n        super(Exp, self).__init__()\n        self.depth = 1\n        self.width = 1\n        self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(\".\")[0]\n        self.num_classes = 1\n\n'''\n\nwith open('cots_config.py', 'w') as f:\n    f.write(config_file_template)","metadata":{"execution":{"iopub.status.busy":"2021-12-22T17:45:43.71605Z","iopub.execute_input":"2021-12-22T17:45:43.716277Z","iopub.status.idle":"2021-12-22T17:45:43.723341Z","shell.execute_reply.started":"2021-12-22T17:45:43.716253Z","shell.execute_reply":"2021-12-22T17:45:43.722558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from yolox.utils import postprocess\nfrom yolox.data.data_augment import ValTransform\n\nCOCO_CLASSES = (\n  \"starfish\",\n)\n\n# get YOLOX experiment\ncurrent_exp = importlib.import_module('cots_config')\nexp = current_exp.Exp()\n\n# set inference parameters\ntest_size = (800, 1280)\nnum_classes = 1\nconfthre = 0.35\nnmsthre = 0.4\n\n\n# get YOLOX model\nmodel = exp.get_model()\nmodel.cuda()\nmodel.eval()\n\n# get custom trained checkpoint\nckpt_file = CHECKPOINT_FILE\nckpt = torch.load(ckpt_file, map_location=\"cpu\")\nmodel.load_state_dict(ckpt[\"model\"])","metadata":{"execution":{"iopub.status.busy":"2021-12-22T17:45:43.72466Z","iopub.execute_input":"2021-12-22T17:45:43.724915Z","iopub.status.idle":"2021-12-22T17:45:53.42862Z","shell.execute_reply.started":"2021-12-22T17:45:43.724881Z","shell.execute_reply":"2021-12-22T17:45:53.42792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def yolox_inference(img, model, test_size): \n    bboxes = []\n    bbclasses = []\n    scores = []\n    \n    preproc = ValTransform(legacy = False)\n\n    tensor_img, _ = preproc(img, None, test_size)\n    tensor_img = torch.from_numpy(tensor_img).unsqueeze(0)\n    tensor_img = tensor_img.float()\n    tensor_img = tensor_img.cuda()\n\n    with torch.no_grad():\n        outputs = model(tensor_img)\n        outputs = postprocess(\n                    outputs, num_classes, confthre,\n                    nmsthre, class_agnostic=True\n                )\n\n    if outputs[0] is None:\n        return [], [], []\n    \n    outputs = outputs[0].cpu()\n    bboxes = outputs[:, 0:4]\n\n    bboxes /= min(test_size[0] / img.shape[0], test_size[1] / img.shape[1])\n    bbclasses = outputs[:, 6]\n    scores = outputs[:, 4] * outputs[:, 5]\n    \n    return bboxes, bbclasses, scores","metadata":{"execution":{"iopub.status.busy":"2021-12-22T17:45:53.429793Z","iopub.execute_input":"2021-12-22T17:45:53.430029Z","iopub.status.idle":"2021-12-22T17:45:53.440507Z","shell.execute_reply.started":"2021-12-22T17:45:53.429994Z","shell.execute_reply":"2021-12-22T17:45:53.43975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def draw_yolox_predictions(img, bboxes, scores, bbclasses, confthre, classes_dict):\n    for i in range(len(bboxes)):\n            box = bboxes[i]\n            cls_id = int(bbclasses[i])\n            score = scores[i]\n            if score < confthre:\n                continue\n            x0 = int(box[0])\n            y0 = int(box[1])\n            x1 = int(box[2])\n            y1 = int(box[3])\n\n            cv2.rectangle(img, (x0, y0), (x1, y1), (0, 255, 0), 2)\n            cv2.putText(img, '{}:{:.1f}%'.format(classes_dict[cls_id], score * 100), (x0, y0 - 3), cv2.FONT_HERSHEY_PLAIN, 0.8, (0,255,0), thickness = 1)\n    return img","metadata":{"execution":{"iopub.status.busy":"2021-12-22T17:45:53.441979Z","iopub.execute_input":"2021-12-22T17:45:53.442271Z","iopub.status.idle":"2021-12-22T17:45:53.452085Z","shell.execute_reply.started":"2021-12-22T17:45:53.442235Z","shell.execute_reply":"2021-12-22T17:45:53.451357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TEST_IMAGE_PATH = \"/kaggle/input/tensorflow-great-barrier-reef/train_images/video_0/9674.jpg\"\nimg = cv2.imread(TEST_IMAGE_PATH)\n\n# Get predictions\nbboxes, bbclasses, scores = yolox_inference(img, model, test_size)\n\n# Draw predictions\nout_image = draw_yolox_predictions(img, bboxes, scores, bbclasses, confthre, COCO_CLASSES)\n\n# Since we load image using OpenCV we have to convert it \nout_image = cv2.cvtColor(out_image, cv2.COLOR_BGR2RGB)\ndisplay(Image.fromarray(out_image))","metadata":{"execution":{"iopub.status.busy":"2021-12-22T17:45:53.454461Z","iopub.execute_input":"2021-12-22T17:45:53.45465Z","iopub.status.idle":"2021-12-22T17:45:58.725629Z","shell.execute_reply.started":"2021-12-22T17:45:53.454627Z","shell.execute_reply":"2021-12-22T17:45:58.722155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SUBMIT PREDICTION TO COMPETITION","metadata":{}},{"cell_type":"code","source":"%cd /kaggle/working/","metadata":{"execution":{"iopub.status.busy":"2021-12-22T17:46:10.870337Z","iopub.execute_input":"2021-12-22T17:46:10.870896Z","iopub.status.idle":"2021-12-22T17:46:10.876683Z","shell.execute_reply.started":"2021-12-22T17:46:10.870859Z","shell.execute_reply":"2021-12-22T17:46:10.875567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import greatbarrierreef\n\nenv = greatbarrierreef.make_env()   # initialize the environment\niter_test = env.iter_test()  ","metadata":{"execution":{"iopub.status.busy":"2021-12-22T17:46:12.173653Z","iopub.execute_input":"2021-12-22T17:46:12.174264Z","iopub.status.idle":"2021-12-22T17:46:12.204098Z","shell.execute_reply.started":"2021-12-22T17:46:12.174227Z","shell.execute_reply":"2021-12-22T17:46:12.20323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##############################################################\n#                      Tracking helpers                      #\n##############################################################\n\nimport numpy as np\nfrom norfair import Detection, Tracker\n\n# Helper to convert bbox in format [x_min, y_min, x_max, y_max, score] to norfair.Detection class\ndef to_norfair(detects, frame_id):\n    result = []\n    for x_min, y_min, x_max, y_max, score in detects:\n        xc, yc = (x_min + x_max) / 2, (y_min + y_max) / 2\n        w, h = x_max - x_min, y_max - y_min\n        result.append(Detection(points=np.array([xc, yc]), scores=np.array([score]), data=np.array([w, h, frame_id])))\n        \n    return result\n\n# Euclidean distance function to match detections on this frame with tracked_objects from previous frames\ndef euclidean_distance(detection, tracked_object):\n    return np.linalg.norm(detection.points - tracked_object.estimate)\n        ","metadata":{"execution":{"iopub.status.busy":"2021-12-22T17:46:13.825846Z","iopub.execute_input":"2021-12-22T17:46:13.826417Z","iopub.status.idle":"2021-12-22T17:46:14.450431Z","shell.execute_reply.started":"2021-12-22T17:46:13.826377Z","shell.execute_reply":"2021-12-22T17:46:14.449711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_dict = {\n    'id': [],\n    'prediction_string': [],\n}\n\n\n#######################################################\n#                      Tracking                       #\n#######################################################\n\n# Tracker will update tracks based on detections from current frame\n# Matching based on euclidean distance between bbox centers of detections \n# from current frame and tracked_objects based on previous frames\n# You can check it's parameters in norfair docs\n# https://github.com/tryolabs/norfair/blob/master/docs/README.md\ntracker = Tracker(\n    distance_function=euclidean_distance, \n    distance_threshold=30,\n    hit_inertia_min=3,\n    hit_inertia_max=6,\n    initialization_delay=1,\n)\n\n# Save frame_id into detection to know which tracks have no detections on current frame\nframe_id = 0\n#######################################################\n\n\nfor (image_np, sample_prediction_df) in iter_test:\n \n    bboxes, bbclasses, scores = yolox_inference(image_np[:,:,::-1], model, test_size)\n    \n    predictions = []\n    detects = []\n    for i in range(len(bboxes)):\n        box = bboxes[i]\n        cls_id = int(bbclasses[i])\n        score = scores[i]\n        if score < confthre:\n            continue\n        x_min = int(box[0])\n        y_min = int(box[1])\n        x_max = int(box[2])\n        y_max = int(box[3])\n        detects.append([x_min, y_min, x_max, y_max, score])\n        \n        bbox_width = x_max - x_min\n        bbox_height = y_max - y_min\n        \n        predictions.append('{:.2f} {} {} {} {}'.format(score, x_min, y_min, bbox_width, bbox_height))\n    \n    #######################################################\n    #                      Tracking                       #\n    #######################################################\n    \n    # Update tracks using detects from current frame\n    tracked_objects = tracker.update(detections=to_norfair(detects, frame_id))\n    for tobj in tracked_objects:\n        bbox_width, bbox_height, last_detected_frame_id = tobj.last_detection.data\n        if last_detected_frame_id == frame_id:  # Skip objects that were detected on current frame\n            continue\n            \n        if last_detected_frame_id + 1 != frame_id:  # Track only one frame forward\n            continue\n\n        # Add objects that have no detections on current frame to predictions\n        xc, yc = tobj.estimate[0]\n        x_min, y_min = int(round(xc - bbox_width / 2)), int(round(yc - bbox_height / 2))\n        score = tobj.last_detection.scores[0]\n\n        predictions.append('{:.2f} {} {} {} {}'.format(score, x_min, y_min, bbox_width, bbox_height))\n    #######################################################\n    \n    prediction_str = ' '.join(predictions)\n    sample_prediction_df['annotations'] = prediction_str\n    env.predict(sample_prediction_df)\n\n    print('Prediction:', prediction_str)\n    frame_id += 1","metadata":{"execution":{"iopub.status.busy":"2021-12-22T16:56:15.182898Z","iopub.execute_input":"2021-12-22T16:56:15.183628Z","iopub.status.idle":"2021-12-22T16:56:15.824912Z","shell.execute_reply.started":"2021-12-22T16:56:15.18359Z","shell.execute_reply":"2021-12-22T16:56:15.82427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df = pd.read_csv('submission.csv')\nsub_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-22T16:56:23.337735Z","iopub.execute_input":"2021-12-22T16:56:23.338372Z","iopub.status.idle":"2021-12-22T16:56:23.354867Z","shell.execute_reply.started":"2021-12-22T16:56:23.338334Z","shell.execute_reply":"2021-12-22T16:56:23.353138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<div class=\"alert alert-success\" role=\"alert\">\n    Find this notebook helpful? :) Please give me a vote ;) Thank you\n </div>","metadata":{}}]}