{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Intro and refs","metadata":{}},{"cell_type":"markdown","source":"This is my first object detection project and I used the YOLOX framework and pretrained model downloaded from [this](https://github.com/Megvii-BaseDetection/YOLOX) gitub repo.  I used Remek Kinas's [excellent notebook](https://www.kaggle.com/remekkinas/yolox-training-pipeline-cots-dataset-lb-0-507) as an inspiration and reference for some parts, though I wrote most of what's here from scratch to encourage learning (anything copied is referenced inline).  Coming into this with no object detection experience there was a lot to learn, and hopefully this notebook might be helpful to someone else in the same boat.  \n\nI used the the YOLOX small model setting with COCO pretrained weights to perform detection, with a (very low) 320 x 320 image size (to fit in my computers 12gb vram).  Later I experimented with 1280 and 2560 resolutions on a remote server with an A6000, which greatly increased performance (the 320 resolution model was almost useless on the LB dataset).  So far I've only done limited hyperparameter tuning to get to 4.81 on the LB datset, but generally overfitting the train and validation set is an issue (given we are only training on 3 videos of data), so # of epochs is important.  \n\n**Full notebook with training**\n\nThis version of the notebook is for inference and competition only to avoid cluttering the output directory with extra files created during cv fold generation.  See the [full notebook](https://www.kaggle.com/max237/getting-started-with-yolox-training-and-inference) for the complete pipeline.  \n\n\n**Some Terminology:**    \n\nIoU - Intersection over union, a measure of how close the predicted bounding box overlaps with the actual box NMS - non maximum suppression, a technique to filter and dedup prediction boxes that overlap. Uses IoU to measure the confidence of each box.\nconf - Confidence level/threshold for the prediction - experimenting with the threshold for this is important.","metadata":{}},{"cell_type":"markdown","source":"## Install and load dependencies  \n\nInstall YOLOX and any other dependencies.  Some of this can be skipped if running on a kaggle notebook, I set this up to be used on a remote ssh server for training as well.  \n\nSome of these are commented out to allow the notebook to run without internet, and i'm using Remek Kinas's [yolox-cots-models](https://www.kaggle.com/remekkinas/yolox-cots-models) dataset.  ","metadata":{}},{"cell_type":"code","source":"# Download the model repo\n#! git clone https://github.com/Megvii-BaseDetection/YOLOX -q\n    \n#! cp -r /kaggle/input/yolox-cots-models/YOLOX/ /kaggle/working/","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:53:59.129168Z","iopub.execute_input":"2022-02-08T16:53:59.129664Z","iopub.status.idle":"2022-02-08T16:53:59.148601Z","shell.execute_reply.started":"2022-02-08T16:53:59.129581Z","shell.execute_reply":"2022-02-08T16:53:59.14796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Install the model\n#%cd YOLOX\n\n# Install yolox  \n#!pip install -v -e .\n\n# Reset filepath\n#%cd ..","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:53:59.149823Z","iopub.execute_input":"2022-02-08T16:53:59.152253Z","iopub.status.idle":"2022-02-08T16:53:59.156587Z","shell.execute_reply.started":"2022-02-08T16:53:59.152213Z","shell.execute_reply":"2022-02-08T16:53:59.155745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load pretrained weights to yolox_s.pth\n#! wget https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_s.pth","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:53:59.161126Z","iopub.execute_input":"2022-02-08T16:53:59.161659Z","iopub.status.idle":"2022-02-08T16:53:59.1658Z","shell.execute_reply.started":"2022-02-08T16:53:59.161621Z","shell.execute_reply":"2022-02-08T16:53:59.164967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport torch\nfrom torchvision import transforms\nimport math\nimport time\nimport os\nimport shutil\nfrom skimage import io, transform\nimport PIL\nimport cv2\nimport IPython.display as display\nimport ast","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:53:59.167661Z","iopub.execute_input":"2022-02-08T16:53:59.168089Z","iopub.status.idle":"2022-02-08T16:54:02.136388Z","shell.execute_reply.started":"2022-02-08T16:53:59.168017Z","shell.execute_reply":"2022-02-08T16:54:02.135601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from shutil import copyfile\nfrom sklearn.model_selection import KFold\nimport random\nfrom collections import defaultdict\nimport json","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:54:02.138536Z","iopub.execute_input":"2022-02-08T16:54:02.139058Z","iopub.status.idle":"2022-02-08T16:54:02.380015Z","shell.execute_reply.started":"2022-02-08T16:54:02.139019Z","shell.execute_reply":"2022-02-08T16:54:02.379287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Add yolox path and load dependencies from their file structure\nimport sys\nsys.path.append(\"/kaggle/input/yolox-cots-models/YOLOX\")\nsys.path.append(\"./pycocotools-2.0.4\")","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:54:02.381384Z","iopub.execute_input":"2022-02-08T16:54:02.381668Z","iopub.status.idle":"2022-02-08T16:54:02.38717Z","shell.execute_reply.started":"2022-02-08T16:54:02.381633Z","shell.execute_reply":"2022-02-08T16:54:02.386309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Unzip and Install pycocotools from a file\n# This was necessary for inference using some of the YOLOX modules\n! tar -xf ../input/pycocotools/pycocotools-2.0.4.tar","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:54:02.389148Z","iopub.execute_input":"2022-02-08T16:54:02.389547Z","iopub.status.idle":"2022-02-08T16:54:03.114732Z","shell.execute_reply.started":"2022-02-08T16:54:02.389512Z","shell.execute_reply":"2022-02-08T16:54:03.113795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%cd pycocotools-2.0.4/\n!python setup.py build_ext --inplace\n%cd ..","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:54:03.116543Z","iopub.execute_input":"2022-02-08T16:54:03.117062Z","iopub.status.idle":"2022-02-08T16:54:10.967025Z","shell.execute_reply.started":"2022-02-08T16:54:03.117016Z","shell.execute_reply":"2022-02-08T16:54:10.966148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Add yolox path and pycocotools paths\nimport sys\nsys.path.append(\"../input/yolox-cots-models/YOLOX\")\nsys.path.append(\"./pycocotools-2.0.4/\")","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:54:27.702118Z","iopub.execute_input":"2022-02-08T16:54:27.702605Z","iopub.status.idle":"2022-02-08T16:54:27.706963Z","shell.execute_reply.started":"2022-02-08T16:54:27.702568Z","shell.execute_reply":"2022-02-08T16:54:27.705986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Install other YOLOX dependencies from Ramek's dataset\n! pip install loguru --no-index --find-links=file:///kaggle/input/yolox-cots-models/yolox-dep/\n! pip install thop --no-index --find-links=file:///kaggle/input/yolox-cots-models/yolox-dep/","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:54:10.976251Z","iopub.execute_input":"2022-02-08T16:54:10.976723Z","iopub.status.idle":"2022-02-08T16:54:27.531742Z","shell.execute_reply.started":"2022-02-08T16:54:10.976688Z","shell.execute_reply":"2022-02-08T16:54:27.530859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load yolox dependencies for inference later\nfrom yolox.data.data_augment import ValTransform\nfrom yolox.utils import postprocess","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:54:27.534249Z","iopub.execute_input":"2022-02-08T16:54:27.534532Z","iopub.status.idle":"2022-02-08T16:54:27.700771Z","shell.execute_reply.started":"2022-02-08T16:54:27.534495Z","shell.execute_reply":"2022-02-08T16:54:27.700139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load and preprocess data","metadata":{}},{"cell_type":"markdown","source":"### Format annotations and get cv folds\n\nI create 5 folders with COCO formated data, which makes it easy to feed into the YOLOX training script (which requires this format).  ","metadata":{}},{"cell_type":"code","source":"data_dir = '../input/tensorflow-great-barrier-reef'","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:54:33.063535Z","iopub.execute_input":"2022-02-08T16:54:33.064075Z","iopub.status.idle":"2022-02-08T16:54:33.067739Z","shell.execute_reply.started":"2022-02-08T16:54:33.064035Z","shell.execute_reply":"2022-02-08T16:54:33.067061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(f'{data_dir}/train.csv')\ndf.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:54:33.525433Z","iopub.execute_input":"2022-02-08T16:54:33.525995Z","iopub.status.idle":"2022-02-08T16:54:33.598698Z","shell.execute_reply.started":"2022-02-08T16:54:33.525963Z","shell.execute_reply":"2022-02-08T16:54:33.597949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Limit to annotated points only (a majority don't have annotations)\ndf_train = df[df['annotations'] != '[]'].copy(deep=True).reset_index(drop=True)\n# Convert from string \ndf_train['annotations'] = df_train['annotations'].apply(lambda x: ast.literal_eval(x))\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:54:34.333632Z","iopub.execute_input":"2022-02-08T16:54:34.334467Z","iopub.status.idle":"2022-02-08T16:54:34.574751Z","shell.execute_reply.started":"2022-02-08T16:54:34.334422Z","shell.execute_reply":"2022-02-08T16:54:34.573977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create experiment config\n\nThis matches the config used in training and allows us to load the model in preparation for adding the weights from training.  I upload my training weights from the [full notebook](https://www.kaggle.com/max237/getting-started-with-yolox-training-and-inference) as a dataset to use for inference.  ","metadata":{}},{"cell_type":"code","source":"# Keep for inference\n\nexp_file_path = './barrier_reef_exp.py'\n\nwith open(exp_file_path, mode='w') as outfile:\n    outfile.write(\"\"\"import os\nfrom yolox.exp import Exp as MyExp\n\nclass Exp320(MyExp):\n    def __init__(self):\n        super(Exp320, self).__init__()\n        self.depth = 0.33 # values for the yolox_s\n        self.width = 0.50 # values for the yolox_s\n        self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(\".\")[0]\n\n        # Define yourself dataset path\n        self.data_dir = \"./fold_0\"\n        self.train_ann = \"train.json\"\n        self.val_ann = \"valid.json\"\n\n        self.num_classes = 1\n\n        self.warmup_epochs = 4\n        self.max_epoch = 15\n        self.data_num_workers = 8\n\n        self.print_interval = 40\n        self.eval_interval = 1\n\n        self.input_size = (320, 320)\n        self.test_size = (320, 320)\n\nclass Exp1280(MyExp):\n    def __init__(self):\n        super(Exp1280, self).__init__()\n        self.depth = 0.33 # values for the yolox_s\n        self.width = 0.50 # values for the yolox_s\n        self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(\".\")[0]\n\n        # Define yourself dataset path\n        self.data_dir = \"./fold_0\"\n        self.train_ann = \"train.json\"\n        self.val_ann = \"valid.json\"\n\n        self.num_classes = 1\n\n        self.warmup_epochs = 4\n        self.max_epoch = 15\n        self.data_num_workers = 8\n\n        self.print_interval = 40\n        self.eval_interval = 1\n\n        self.input_size = (1280, 1280)\n        self.test_size = (1280, 1280)\n        \nclass Exp2560(MyExp):\n    def __init__(self):\n        super(Exp2560, self).__init__()\n        self.depth = 0.33 # values for the yolox_s\n        self.width = 0.50 # values for the yolox_s\n        self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(\".\")[0]\n\n        # Define yourself dataset path\n        self.data_dir = \"./fold_0\"\n        self.train_ann = \"train.json\"\n        self.val_ann = \"valid.json\"\n\n        self.num_classes = 1\n\n        self.warmup_epochs = 4\n        self.max_epoch = 15\n        self.data_num_workers = 8\n\n        self.print_interval = 40\n        self.eval_interval = 1\n\n        self.input_size = (2560, 2560)\n        self.test_size = (2560, 2560)\n\"\"\")","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:54:37.790183Z","iopub.execute_input":"2022-02-08T16:54:37.792819Z","iopub.status.idle":"2022-02-08T16:54:37.807858Z","shell.execute_reply.started":"2022-02-08T16:54:37.792767Z","shell.execute_reply":"2022-02-08T16:54:37.807126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inference and results","metadata":{}},{"cell_type":"markdown","source":"### Visualize test predictions\n\nTo start, we can usse the YOLOX demo script to run the model with the latest trained weights on some sample images, and then plot the bboxes on top of the images.  The script also helpfully draws the box predictions onto the images and saves them, and i've implemented some code to draw the true boxes to compare.  \n\nExample of how to run the script here: https://github.com/Megvii-BaseDetection/YOLOX/blob/main/docs/quick_run.md  ","metadata":{}},{"cell_type":"code","source":"from barrier_reef_exp import Exp320, Exp1280, Exp2560\nexperiment = Exp2560()\ntest_image_path = '../input/tensorflow-great-barrier-reef/train_images/video_0/9653.jpg'\n# model_weights_path = 'YOLOX_outputs/barrier_reef_exp/best_ckpt.pth' # Local\nmodel_weights_path = '../input/yolox-s-trained-weights/best_ckpt_2560_15.pth' # kaggle dataset","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:18.560256Z","iopub.execute_input":"2022-02-08T16:55:18.560532Z","iopub.status.idle":"2022-02-08T16:55:18.570666Z","shell.execute_reply.started":"2022-02-08T16:55:18.560501Z","shell.execute_reply":"2022-02-08T16:55:18.569805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use the demo.py tool to run inference\n# Currently not running this\n\n#! python3 YOLOX/tools/demo.py image \\\n#    -f barrier_reef_exp.py \\\n#    -c {model_weights_path} \\\n#    --path {test_image_path} \\\n#    --conf 0.1 \\\n#    --nms 0.3 \\\n#    --tsize 320 \\\n#    --device gpu \\\n#    --save_result\n    ","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:20.733177Z","iopub.execute_input":"2022-02-08T16:55:20.733718Z","iopub.status.idle":"2022-02-08T16:55:20.738916Z","shell.execute_reply.started":"2022-02-08T16:55:20.733678Z","shell.execute_reply":"2022-02-08T16:55:20.738074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize predicted boxes\n# Copy the image path output from the previous step\n\n#img_path = './YOLOX_outputs/barrier_reef_exp/vis_res/2022_02_02_00_24_37/0-9653.jpg'\n#test_img = PIL.Image.open(img_path)\n#display.display(test_img)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:26.499657Z","iopub.execute_input":"2022-02-08T16:55:26.500125Z","iopub.status.idle":"2022-02-08T16:55:26.50399Z","shell.execute_reply.started":"2022-02-08T16:55:26.500071Z","shell.execute_reply":"2022-02-08T16:55:26.502995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the actual ground truth locations (use cv2 to draw in the boxes)\n\ntest_img = cv2.imread(test_image_path)\nboxes = df_train[df_train['image_id'] == '0-' + test_image_path.split('/')[-1][:-4]]['annotations'].tolist()[0]\n\nfor box in boxes:\n    upper_left = (int(box['x']), int(box['y']))\n    lower_right = (int(box['x'] + box['width']), int(box['y'] + box['height']))\n    color = (255, 0, 0)\n    test_img = cv2.rectangle(test_img, upper_left, lower_right, color=color, thickness = 2)\n\ntest_img = cv2.cvtColor(test_img, cv2.COLOR_BGR2RGB)\ntest_img_pil = PIL.Image.fromarray(test_img)\ndisplay.display(test_img_pil)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:28.344903Z","iopub.execute_input":"2022-02-08T16:55:28.345641Z","iopub.status.idle":"2022-02-08T16:55:28.777713Z","shell.execute_reply.started":"2022-02-08T16:55:28.345603Z","shell.execute_reply":"2022-02-08T16:55:28.776896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Run on the test dataset for submission\n\n**Inference in notebook**\n\nSince we are getting images directly from the api and not from a file, we unfortunately can't just use the YOLOX demo tool to get our box predictions.  Instead, we have to adapt pieces of that tool to fit our needs.  I adapt the [inference function from demo.py](https://github.com/Megvii-BaseDetection/YOLOX/blob/main/tools/demo.py#L132) script for this purpose.  \n\nUsing this also requires installing some new python libaries, which I do offline from Remek's dataset.  Instructions [here](https://www.kaggle.com/samuelepino/pip-installing-packages-with-no-internet).","metadata":{}},{"cell_type":"markdown","source":"**Load the model object**\n\nUses the get_model function from the [experiment base class] (https://github.com/Megvii-BaseDetection/YOLOX/blob/main/yolox/exp/yolox_base.py)","metadata":{}},{"cell_type":"code","source":"def get_trained_model(experiment, weights):\n    \n    # Use the experiment built in function to generate the same model we trained with\n    model = experiment.get_model()\n    \n    # Inference on gpu\n    model.cuda()\n    \n    # Turn off training mode so the model so it won't try to calculate loss\n    model.eval()\n    model.head.training=False\n    model.training=False\n    \n    # Load in the weights from training\n    best_weights = torch.load(weights)\n    model.load_state_dict(best_weights['model'])\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:35.869938Z","iopub.execute_input":"2022-02-08T16:55:35.870219Z","iopub.status.idle":"2022-02-08T16:55:35.875616Z","shell.execute_reply.started":"2022-02-08T16:55:35.870189Z","shell.execute_reply":"2022-02-08T16:55:35.874595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Extract the boxes from the processed predictions \n\ndef get_boxes(outputs):\n    output = outputs[0][0]\n    \n    if output == None:\n        return {'bboxes': [], 'scores': []}\n    # move to cpu\n    output = output.cpu()\n    \n    img_info = outputs[1]\n    \n    bboxes = output[:, 0:4]/img_info['ratio']\n    scores = output[:, 4] * output[:, 5]\n    \n    return {'bboxes': bboxes, 'scores': scores}","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:37.551065Z","iopub.execute_input":"2022-02-08T16:55:37.551336Z","iopub.status.idle":"2022-02-08T16:55:37.556958Z","shell.execute_reply.started":"2022-02-08T16:55:37.551307Z","shell.execute_reply":"2022-02-08T16:55:37.556132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Custom implementation of the inference function from demo.py \n# Takes in an image object instead of a filepath\n\ndef inference(img, model, experiment, device):\n    \n        test_size = experiment.test_size\n        confthre = experiment.test_conf\n        nmsthre = experiment.nmsthre\n    \n        img_info = {\"id\": 0}\n        img_info[\"file_name\"] = None\n\n        height, width = img.shape[:2]\n        img_info[\"height\"] = height\n        img_info[\"width\"] = width\n        img_info[\"raw_img\"] = img\n\n        ratio = min(test_size[0] / img.shape[0], test_size[1] / img.shape[1])\n        img_info[\"ratio\"] = ratio\n        \n        preproc = ValTransform(legacy=False)\n        \n        img, _ = preproc(img, None, test_size)\n        img = torch.from_numpy(img).unsqueeze(0)\n        img = img.float()\n        if device == \"gpu\":\n            img = img.cuda()\n\n        with torch.no_grad():\n            t0 = time.time()\n            outputs = model(img)\n\n            outputs = postprocess(\n                outputs, 1, confthre,\n                nmsthre, class_agnostic=True\n            )\n        return outputs, img_info","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:37.950228Z","iopub.execute_input":"2022-02-08T16:55:37.950902Z","iopub.status.idle":"2022-02-08T16:55:37.960251Z","shell.execute_reply.started":"2022-02-08T16:55:37.950864Z","shell.execute_reply":"2022-02-08T16:55:37.959497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get box predictions for a single image and given thresholds\n\ndef barrier_reef_inference(exp_file, weights, test_image, \n                           conf_threshold=0.1, nms_threshold=0.3,\n                           device='gpu'):\n    \n    # Load the experiment file\n    experiment = exp_file\n    \n    # Set up the model and weights\n    model = get_trained_model(experiment, weights)\n    \n    # Set custom thresholds for inference\n    experiment.test_conf = conf_threshold\n    experiment.nmsthre = nms_threshold\n    \n    test_size = experiment.test_size\n    \n    # Run the image through the model\n    outputs = inference(test_image, model, experiment, device)\n    \n    return get_boxes(outputs)\n","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:38.312133Z","iopub.execute_input":"2022-02-08T16:55:38.312375Z","iopub.status.idle":"2022-02-08T16:55:38.319052Z","shell.execute_reply.started":"2022-02-08T16:55:38.312348Z","shell.execute_reply":"2022-02-08T16:55:38.318268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Try it out on the test image \n\n# Load as a array first\ntest_image = cv2.imread(test_image_path)\nprint(test_image.shape)\n\nbox_preds = barrier_reef_inference(experiment, model_weights_path, test_image,\n                                   conf_threshold=0.1, nms_threshold=0.3)\nprint(box_preds)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:38.700135Z","iopub.execute_input":"2022-02-08T16:55:38.700539Z","iopub.status.idle":"2022-02-08T16:55:49.596331Z","shell.execute_reply.started":"2022-02-08T16:55:38.700505Z","shell.execute_reply":"2022-02-08T16:55:49.594561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Sample submission string for this example\ntest_image = cv2.imread(test_image_path)\n\noutputs = barrier_reef_inference(experiment, model_weights_path, test_image,\n                                 conf_threshold=0.1, nms_threshold=0.3)\n\nbboxes = outputs['bboxes']\nscores = outputs['scores']\n\npredictions = []\n\nfor i in range(len(bboxes)):\n    box = bboxes[i]\n    score = scores[i]\n\n    x_min = int(box[0])\n    y_min = int(box[1])\n    x_max = int(box[2])\n    y_max = int(box[3])\n\n    bbox_width = x_max - x_min\n    bbox_height = y_max - y_min\n\n    predictions.append('{:.2f} {} {} {} {}'.format(score, x_min, y_min, bbox_width, bbox_height))\n\nprediction_str = ' '.join(predictions)\n\nprint('Prediction:', prediction_str)\n","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:49.598201Z","iopub.execute_input":"2022-02-08T16:55:49.598699Z","iopub.status.idle":"2022-02-08T16:55:49.936001Z","shell.execute_reply.started":"2022-02-08T16:55:49.598656Z","shell.execute_reply":"2022-02-08T16:55:49.935189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def visualize_boxes(test_image, boxes, scores):\n    \n    test_image_boxed = test_image\n\n    for box in boxes:\n        \n        x0 = int(box[0])\n        y0 = int(box[1])\n        x1 = int(box[2])\n        y1 = int(box[3])\n\n        color = (255, 0, 0)\n        test_image_boxed = cv2.rectangle(test_image_boxed, (x0, y0), (x1, y1), color=color, thickness = 2)\n    \n    test_image_recolored = cv2.cvtColor(test_image_boxed, cv2.COLOR_BGR2RGB)\n    test_image_pil = PIL.Image.fromarray(test_image_recolored)\n    \n    return test_image_pil\n\ntest_image = cv2.imread(test_image_path)\ntest_image_pil = visualize_boxes(test_image, box_preds['bboxes'], box_preds['scores'])\ndisplay.display(test_image_pil)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:49.937328Z","iopub.execute_input":"2022-02-08T16:55:49.937991Z","iopub.status.idle":"2022-02-08T16:55:50.307848Z","shell.execute_reply.started":"2022-02-08T16:55:49.937952Z","shell.execute_reply":"2022-02-08T16:55:50.307089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The boxes from our custom inference implementation match that of the YOLOX demo tool, which means we are good to go and inference is working as expected.  ","metadata":{}},{"cell_type":"markdown","source":"### Call the barrier reef api and send predictions\n\nThis last section is also adapted from [Ramek's notebook](https://www.kaggle.com/remekkinas/yolox-training-pipeline-cots-dataset-lb-0-507).  ","metadata":{}},{"cell_type":"code","source":"import greatbarrierreef\n\nenv = greatbarrierreef.make_env()  # initialize the environment\niter_test = env.iter_test() ","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:50.309634Z","iopub.execute_input":"2022-02-08T16:55:50.31002Z","iopub.status.idle":"2022-02-08T16:55:50.341861Z","shell.execute_reply.started":"2022-02-08T16:55:50.309984Z","shell.execute_reply":"2022-02-08T16:55:50.340966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get predicted boxes for each image returned by the api\n\nfor (image_np, sample_prediction_df) in iter_test:\n    \n    \n    outputs = barrier_reef_inference(experiment, model_weights_path, image_np[:,:,::-1],\n                                     conf_threshold=0.15, nms_threshold=0.3)\n\n    bboxes = outputs['bboxes']\n    scores = outputs['scores']\n\n    predictions = []\n\n    for i in range(len(bboxes)):\n        box = bboxes[i]\n        score = scores[i]\n\n        x_min = int(box[0])\n        y_min = int(box[1])\n        x_max = int(box[2])\n        y_max = int(box[3])\n\n        bbox_width = x_max - x_min\n        bbox_height = y_max - y_min\n\n        predictions.append('{:.2f} {} {} {} {}'.format(score, x_min, y_min, bbox_width, bbox_height))\n\n    prediction_str = ' '.join(predictions)\n    sample_prediction_df['annotations'] = prediction_str\n    env.predict(sample_prediction_df)\n\n    print('Prediction:', prediction_str)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:50.343326Z","iopub.execute_input":"2022-02-08T16:55:50.343705Z","iopub.status.idle":"2022-02-08T16:55:51.663196Z","shell.execute_reply.started":"2022-02-08T16:55:50.34367Z","shell.execute_reply":"2022-02-08T16:55:51.661356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check the dataframe output from the previous cell (this is what's used for submission)\nsub_df = pd.read_csv('submission.csv')\nsub_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:51.664748Z","iopub.execute_input":"2022-02-08T16:55:51.665045Z","iopub.status.idle":"2022-02-08T16:55:51.678274Z","shell.execute_reply.started":"2022-02-08T16:55:51.665005Z","shell.execute_reply":"2022-02-08T16:55:51.677174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Test out image color transforms\ntest_image_recolored = cv2.cvtColor(image_np[:,:,::-1], cv2.COLOR_BGR2RGB)\ntest_image_pil = PIL.Image.fromarray(test_image_recolored)\ndisplay.display(test_image_pil)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T16:55:51.679762Z","iopub.execute_input":"2022-02-08T16:55:51.680053Z","iopub.status.idle":"2022-02-08T16:55:52.170502Z","shell.execute_reply.started":"2022-02-08T16:55:51.680016Z","shell.execute_reply":"2022-02-08T16:55:52.168199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}