{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# ScaledYOLOv4 Inference Notebook for COTS\n\nPaper: https://arxiv.org/pdf/2011.08036.pdf\n\nWang et al's ScaledYOLOv4 is a YOLOv4-based architecture that can scale up for better accuracy. There are 3 'large' models:\n* P5: contains P3, P4 and P5 detection heads\n* P6: P3, P4, P5, and P6 detection heads\n* P7: P3, P4, P5, P6 and P7 detection heads\n\n<img src=\"https://blog.roboflow.com/content/images/2020/12/image-2.png\">\n\ncourtesy https://blog.roboflow.com/scaled-yolov4-tops-efficientdet/","metadata":{}},{"cell_type":"markdown","source":"## Notebook Control Panel","metadata":{}},{"cell_type":"code","source":"# CONTROL PANEL\n\nCHECKPOINT_FILE = '/kaggle/input/scaledyolov4-cots-p6/e12_FT_e8_freeze/last_020.pt'\n\nSIZE = 1920\n\n# SY4's augment does not work well. Use manual ensemble where model is inferenced at several sizes\nAUGMENT = False\nAUGMENTED_SIZES = [1920, 2560, 2880] # must be multiples of 32. Scaling by [1.0, 1.333, 1.5]\n\n# Whether to use single resolution or multi-resolution WBF for inference\nUSE_AUGMENTED_INFERENCE = True\n\nCONF = 0.12\nNMS_THRESHOLD = 0.4\nWBF_IOU = NMS_THRESHOLD\nUSE_HALF_PRECISION = True\n\ndevice_type = '0' # (either 'cpu' for CPU, or '0' for GPU)","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:25:55.665997Z","iopub.execute_input":"2022-02-21T22:25:55.666251Z","iopub.status.idle":"2022-02-21T22:25:55.671069Z","shell.execute_reply.started":"2022-02-21T22:25:55.666224Z","shell.execute_reply":"2022-02-21T22:25:55.670216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Results\n\n### Rationale for training\n* This model was trained using a concept similar to that used by YOLOX\n  * First, train a model with heavy mixup. The model will achieve modest mAP, but its backbone learns a lot because of the mixup.\n  * On fine-tuning, shut down the mixup and reduce the augmentations, in order to maximise mAP. Also, learn at reduced learning rate, so that knowledge gained from the previous mixup training is not lost. \n  * To ensure mixup knowledge is not lost, CSPDark layers are frozen during fine-tuning. This is implemented via my custom fork of ScaledYOLOv4: https://github.com/alexchwong/ScaledYOLOv4\n\n### Model Hyperparameters\n* Important training parameters:\n  * Model type: yolov4-p6\n  * Scratch (epochs 0-12):\n    * Res: 1920p\n    * mixup: 1.0\n    * degrees: 30.0\n    * translate: 0.5\n    * scale: 0.2\n    * shear: 0.0\n  * then Fine-tine (epochs 12-20)\n    * Res: 1920p\n    * mixup: 0.0\n    * degrees: 0.0\n    * translate: 0.2\n    * scale: 0.2\n    * lr0: 0.002 (default 0.01)\n    * frozen layers: 1 to 12 (i.e. CSPDark backbone)","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.figure(figsize=(30,15))\nplt.axis('off')\nplt.imshow(plt.imread('/kaggle/input/scaledyolov4-cots-p6/e12_FT_e8_freeze/results.png'));","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-21T22:23:40.367537Z","iopub.execute_input":"2022-02-21T22:23:40.367812Z","iopub.status.idle":"2022-02-21T22:23:41.498678Z","shell.execute_reply.started":"2022-02-21T22:23:40.367782Z","shell.execute_reply":"2022-02-21T22:23:41.497987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ScaledYOLOv4 installation","metadata":{}},{"cell_type":"code","source":"!cp -r /kaggle/input/scaledyolov4-installation/ScaledYOLOv4 /kaggle/working/","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:23:46.268539Z","iopub.execute_input":"2022-02-21T22:23:46.268802Z","iopub.status.idle":"2022-02-21T22:23:47.362712Z","shell.execute_reply.started":"2022-02-21T22:23:46.268773Z","shell.execute_reply":"2022-02-21T22:23:47.361688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## FFMPEG for videos","metadata":{}},{"cell_type":"code","source":"# Install ffmpeg for video compression\n%cd /kaggle/working\n\n! tar xvf ../input/ffmpeg-static-build/ffmpeg-git-amd64-static.tar.xz\n\nimport subprocess\n\nFFMPEG_BIN = \"/kaggle/working/ffmpeg-git-20191209-amd64-static/ffmpeg\"","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-21T22:23:59.286971Z","iopub.execute_input":"2022-02-21T22:23:59.287246Z","iopub.status.idle":"2022-02-21T22:24:03.343441Z","shell.execute_reply.started":"2022-02-21T22:23:59.287217Z","shell.execute_reply":"2022-02-21T22:24:03.34253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Required Modules","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport time\n\nimport os\nimport torch\nimport importlib\nimport cv2 \nimport pandas as pd\nimport numpy as np\n\nimport ast\nimport shutil\nimport sys\n\nfrom tqdm.notebook import tqdm\ntqdm.pandas()\n\nfrom PIL import Image\nfrom IPython.display import display","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:24:26.133571Z","iopub.execute_input":"2022-02-21T22:24:26.133859Z","iopub.status.idle":"2022-02-21T22:24:27.77307Z","shell.execute_reply.started":"2022-02-21T22:24:26.133828Z","shell.execute_reply":"2022-02-21T22:24:27.7723Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load the ScaledYOLOv4 Model","metadata":{}},{"cell_type":"code","source":"%cd /kaggle/working/ScaledYOLOv4\n\nimgsz = SIZE\nweights = CHECKPOINT_FILE\n\nfrom models.experimental import attempt_load\nfrom utils.datasets import LoadImages\nfrom utils.general import check_img_size, non_max_suppression, scale_coords, xyxy2xywh\nfrom utils.torch_utils import select_device\n\ndevice = select_device(device_type)\nhalf = (USE_HALF_PRECISION and (device_type != 'cpu'))\n\nmodel = attempt_load(weights, map_location=device)  # load FP32 model\nimgsz = check_img_size(imgsz, s=model.stride.max())  # check img_size\nif half:\n    model.half()  # to FP16\n    \nimg = torch.zeros((1, 3, imgsz, imgsz), device=device)  # init img\n_ = model(img.half() if half else img) if device.type != 'cpu' else None  # run once\n\nCOCO_CLASSES = ['cots']","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:24:29.668508Z","iopub.execute_input":"2022-02-21T22:24:29.669171Z","iopub.status.idle":"2022-02-21T22:24:46.014518Z","shell.execute_reply.started":"2022-02-21T22:24:29.669131Z","shell.execute_reply":"2022-02-21T22:24:46.013782Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ScaledYOLOv4 utility functions","metadata":{}},{"cell_type":"code","source":"def yolo2coco(bboxes, image_height=720, image_width=1280):\n    \"\"\"\n    yolo => [xmid, ymid, w, h] (normalized)\n    coco => [xmin, ymin, w, h]\n    \"\"\" \n    bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n    \n    # denormalizing\n    bboxes[..., [0, 2]]= bboxes[..., [0, 2]]* image_width\n    bboxes[..., [1, 3]]= bboxes[..., [1, 3]]* image_height\n    \n    # converstion (xmid, ymid) => (xmin, ymin) \n    bboxes[..., [0, 1]] = bboxes[..., [0, 1]] - bboxes[..., [2, 3]]/2\n    \n    return bboxes","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:24:27.774621Z","iopub.execute_input":"2022-02-21T22:24:27.774893Z","iopub.status.idle":"2022-02-21T22:24:27.781824Z","shell.execute_reply.started":"2022-02-21T22:24:27.774858Z","shell.execute_reply":"2022-02-21T22:24:27.781117Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from utils.datasets import letterbox\n\ndef preprocess_image(im0, new_shape = None):\n    if new_shape is None:\n        new_shape = im0.shape\n    img = letterbox(im0, new_shape=new_shape)[0]\n    img = img[:, :, ::-1].transpose(2, 0, 1)  # BGR to RGB, to 3x416x416\n    img = np.ascontiguousarray(img)\n    return(img)","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:24:46.016218Z","iopub.execute_input":"2022-02-21T22:24:46.016516Z","iopub.status.idle":"2022-02-21T22:24:46.024101Z","shell.execute_reply.started":"2022-02-21T22:24:46.016489Z","shell.execute_reply":"2022-02-21T22:24:46.023419Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def sy4_inference(img0, model, test_size, conf_threshold = 0.4, verbose = True): \n    bboxes = []\n    bbclasses = []\n    scores = []\n    ts = time.perf_counter()\n\n    img = preprocess_image(img0, new_shape = test_size)\n    img = torch.from_numpy(img).to(device)\n    img = img.half() if half else img.float()  # uint8 to fp16/32\n    img /= 255.0  # 0 - 255 to 0.0 - 1.0\n    if img.ndimension() == 3:\n        img = img.unsqueeze(0)\n\n    # Inference\n    pred = model(img, augment=AUGMENT)[0]\n    # Apply NMS\n    pred = non_max_suppression(pred, conf_threshold, NMS_THRESHOLD, classes=[0], agnostic=False)\n\n    det = pred[0]\n    bboxes = []\n    bbclasses = []\n    scores = []\n    gn = torch.tensor(img0.shape)[[1, 0, 1, 0]]\n    \n    if det is not None and len(det):\n        det[:, :4] = scale_coords(img.shape[2:], det[:, :4], img0.shape).round()\n        for *xyxy, conf, cls in det:\n            bbclasses.append(int(cls.item()))\n            scores.append(conf.item())\n            \n            xywh = (xyxy2xywh(torch.tensor(xyxy).view(1, 4)) / gn).view(-1).tolist() # normalized xywh\n            bboxes.append(xywh)                       \n                            \n    if len(bboxes) == 0:\n            if verbose:\n                te = time.perf_counter()\n                print(f\"ScaledYOLOv4 inference time {round(te - ts, 4)} s\")\n            return [], [], []\n    \n    bboxes_coco = yolo2coco(np.array(bboxes), img0.shape[0], img0.shape[1])\n    \n    if verbose:\n        te = time.perf_counter()\n        print(f\"ScaledYOLOv4 inference time {round(te - ts, 4)} s\")\n    \n    return bboxes_coco, bbclasses, scores","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:24:46.037466Z","iopub.execute_input":"2022-02-21T22:24:46.03779Z","iopub.status.idle":"2022-02-21T22:24:46.05228Z","shell.execute_reply.started":"2022-02-21T22:24:46.037754Z","shell.execute_reply":"2022-02-21T22:24:46.051439Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from copy import deepcopy\n\ndef sy4_inference_augmented(img0, model, test_sizes, conf_threshold = 0.4, verbose = True): \n    bboxes_list = []\n    bbclasses_list = []\n    scores_list = []\n    ts = time.perf_counter()\n    \n    for test_size in test_sizes:\n        bboxes, bbclasses, scores = sy4_inference(img0, model, test_size, conf_threshold, verbose = False)\n        if len(bboxes):\n            bboxes_list.append(deepcopy(bboxes))\n            bbclasses_list.append(deepcopy(bbclasses))\n            scores_list.append(deepcopy(scores))\n\n    bboxes_final, scores_final, bbclasses_final = wbf_coco(bboxes_list, scores_list, bbclasses_list, img0, conf_type = 'max', verbose = False)\n    if len(bboxes_final) == 0:\n        if verbose:\n            te = time.perf_counter()\n            print(f\"Augmented ScaledYOLOv4 inference time {round(te - ts, 4)} s\")\n        return [], [], []\n    if verbose:\n        te = time.perf_counter()\n        print(f\"Augmented ScaledYOLOv4 inference time {round(te - ts, 4)} s\")\n    return bboxes_final, bbclasses_final, scores_final","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:24:46.025198Z","iopub.execute_input":"2022-02-21T22:24:46.025988Z","iopub.status.idle":"2022-02-21T22:24:46.035937Z","shell.execute_reply.started":"2022-02-21T22:24:46.025948Z","shell.execute_reply":"2022-02-21T22:24:46.035155Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Modified from https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507\n\ndef draw_yolox_predictions(img, bboxes, scores, bbclasses, classes_dict, boxcolor = (0,0,255)):\n    outimg = img.copy()\n    for i in range(len(bboxes)):\n        box = bboxes[i]\n        cls_id = int(bbclasses[i])\n        score = scores[i]\n        x0 = int(box[0])\n        y0 = int(box[1])\n        x1 = x0 + int(box[2])\n        y1 = y0 + int(box[3])\n\n        cv2.rectangle(outimg, (x0, y0), (x1, y1), boxcolor, 2)\n        cv2.putText(outimg, '{}:{:.1f}%'.format(classes_dict[cls_id], score * 100), (x0, y0 - 3), cv2.FONT_HERSHEY_PLAIN, 0.8, boxcolor, thickness = 1)\n    return outimg","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:24:48.230556Z","iopub.execute_input":"2022-02-21T22:24:48.231118Z","iopub.status.idle":"2022-02-21T22:24:48.239226Z","shell.execute_reply.started":"2022-02-21T22:24:48.231081Z","shell.execute_reply":"2022-02-21T22:24:48.238448Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Weighted Box fusion for Model Ensemble","metadata":{}},{"cell_type":"code","source":"sys.path.append('/kaggle/input/packages-wbf/packages')\nfrom ensemble_boxes.ensemble_boxes_wbf import *\n\ndef wbf(boxes_list, scores_list, labels_list):\n    \"\"\"\n    vocnorm => [x1, y1, x2, y2] (normalized)\n    \"\"\"\n    iou_thr = WBF_IOU\n    skip_box_thr = 0.0001\n    return(weighted_boxes_fusion(boxes_list, scores_list, labels_list, iou_thr=iou_thr, skip_box_thr=skip_box_thr))\n\ndef wbf_coco(boxes_list, scores_list, labels_list, img, conf_type='max', verbose = True):\n    \"\"\"\n    coco => [xmin, ymin, w, h]\n    \"\"\"\n    ts = time.perf_counter()\n    height = img.shape[0]\n    width = img.shape[1]\n    new_bboxes_list = []\n    new_scores_list = []\n    new_labels_list = []\n    for i in range(len(boxes_list)):\n        if len(boxes_list[i]):\n            new_bboxes_list.append(coco2vocnorm(boxes_list[i], height, width))\n            new_scores_list.append(scores_list[i])\n            new_labels_list.append(labels_list[i])\n    iou_thr = WBF_IOU\n    skip_box_thr = 0.0001\n    bboxes, confs, labels = weighted_boxes_fusion(new_bboxes_list, new_scores_list, new_labels_list, iou_thr=iou_thr, skip_box_thr=skip_box_thr, conf_type=conf_type)\n    bboxes  = vocdenorm(bboxes,height,width)\n    bboxes  = voc2coco(bboxes).astype(int)\n    te = time.perf_counter()\n    if verbose:\n        print(f\"WBF inference time {te - ts}\")\n    return bboxes, confs, labels\n\ndef coco2vocnorm(bboxes, image_height=720, image_width=1280):\n    \"\"\"\n    coco => [xmin, ymin, w, h]\n    vocnorm => [x1, y1, x2, y2] (normalized)\n    \"\"\"\n    bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n    bboxes[..., [0, 2]] = bboxes[..., [0, 2]]/ image_width\n    bboxes[..., [1, 3]] = bboxes[..., [1, 3]]/ image_height\n    \n    bboxes[..., [2, 3]] = bboxes[..., [2, 3]] + bboxes[..., [0, 1]]\n    \n    return bboxes\n\ndef voc2coco(bboxes):\n    bboxes = bboxes.copy()\n    bboxes[..., [2, 3]] = bboxes[..., [2, 3]] - bboxes[..., [0, 1]]\n    return bboxes\n\ndef vocnorm(bboxes, image_height=720, image_width=1280):\n    \"\"\"\n    voc  => [x1, y1, x2, y2]\n    vocnorm => [x1, y1, x2, y2] (normalized)\n    \"\"\"\n    bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n    bboxes[..., [0, 2]] = bboxes[..., [0, 2]]/ image_width\n    bboxes[..., [1, 3]] = bboxes[..., [1, 3]]/ image_height\n    \n    return bboxes\n\ndef vocdenorm(bboxes, image_height=720, image_width=1280):\n    \"\"\"\n    vocnorm  => [x1, y1, x2, y2] (normalized)\n    voc => [x1, y1, x2, y2] \n    \"\"\"\n    bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n    bboxes[..., [0, 2]] = bboxes[..., [0, 2]] * image_width\n    bboxes[..., [1, 3]] = bboxes[..., [1, 3]] * image_height\n    \n    return bboxes","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:24:50.080176Z","iopub.execute_input":"2022-02-21T22:24:50.080628Z","iopub.status.idle":"2022-02-21T22:24:50.868435Z","shell.execute_reply.started":"2022-02-21T22:24:50.080591Z","shell.execute_reply":"2022-02-21T22:24:50.867519Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loading the validation dataset","metadata":{}},{"cell_type":"code","source":"%cd /kaggle/working\n\nfrom sklearn.model_selection import GroupKFold\n\ndef get_bbox(annots):\n    bboxes = [list(annot.values()) for annot in annots]\n    return bboxes\n\ndef get_path(row):\n    row['image_path'] = f'{ROOT_DIR}/train_images/video_{row.video_id}/{row.video_frame}.jpg'\n    return row\n\nROOT_DIR  = '/kaggle/input/tensorflow-great-barrier-reef/'\n\ndf = pd.read_csv(\"/kaggle/input/tensorflow-great-barrier-reef/train.csv\")\n\n\ndf[\"num_bbox\"] = df['annotations'].apply(lambda x: str.count(x, 'x'))\ndf_train = df\n\n#Annotations \ndf_train['annotations'] = df_train['annotations'].progress_apply(lambda x: ast.literal_eval(x))\ndf_train['bboxes'] = df_train.annotations.progress_apply(get_bbox)\n\ndf_train = df_train.progress_apply(get_path, axis=1)\n\nkf = GroupKFold(n_splits = 5) \ndf_train = df_train.reset_index(drop=True)\ndf_train['fold'] = -1\nfor fold, (train_idx, val_idx) in enumerate(kf.split(df_train, y = df_train.video_id.tolist(), groups=df_train.sequence)):\n    df_train.loc[val_idx, 'fold'] = fold\n\ndf_train.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:24:53.071628Z","iopub.execute_input":"2022-02-21T22:24:53.072135Z","iopub.status.idle":"2022-02-21T22:25:08.937211Z","shell.execute_reply.started":"2022-02-21T22:24:53.072097Z","shell.execute_reply":"2022-02-21T22:25:08.936379Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test = df_train[df_train.fold == 4]","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:25:08.938819Z","iopub.execute_input":"2022-02-21T22:25:08.939143Z","iopub.status.idle":"2022-02-21T22:25:08.946786Z","shell.execute_reply.started":"2022-02-21T22:25:08.939106Z","shell.execute_reply":"2022-02-21T22:25:08.94609Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_paths = df_test.image_path.tolist()\ngt = df_test.bboxes.tolist()","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:25:08.947881Z","iopub.execute_input":"2022-02-21T22:25:08.948194Z","iopub.status.idle":"2022-02-21T22:25:08.955109Z","shell.execute_reply.started":"2022-02-21T22:25:08.948157Z","shell.execute_reply":"2022-02-21T22:25:08.954358Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Test Inference","metadata":{}},{"cell_type":"code","source":"i = 1350\nTEST_IMAGE_PATH = image_paths[i]\nimg = cv2.imread(TEST_IMAGE_PATH)\n\n# Draw Green ground truth box\nout_image0 = draw_yolox_predictions(img, gt[i], [1.0] * len(gt[i]), [0] * len(gt[i]), COCO_CLASSES, (0,255,0))\n\n# Base model inference\nbboxes, bbclasses, scores = sy4_inference(img, model, SIZE, CONF)\nout_image = draw_yolox_predictions(out_image0, bboxes, scores, bbclasses, COCO_CLASSES, (0,0,255))\n# Convert BGR to RGB\nout_image = cv2.cvtColor(out_image, cv2.COLOR_BGR2RGB)\ndisplay(Image.fromarray(out_image))\n\n# Augmented model inference\nbboxes, bbclasses, scores = sy4_inference_augmented(img, model, AUGMENTED_SIZES, CONF)\nout_image = draw_yolox_predictions(out_image0, bboxes, scores, bbclasses, COCO_CLASSES, (0,0,255))\n# Convert BGR to RGB\nout_image = cv2.cvtColor(out_image, cv2.COLOR_BGR2RGB)\ndisplay(Image.fromarray(out_image))","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:25:59.662274Z","iopub.execute_input":"2022-02-21T22:25:59.663065Z","iopub.status.idle":"2022-02-21T22:26:01.474099Z","shell.execute_reply.started":"2022-02-21T22:25:59.663026Z","shell.execute_reply":"2022-02-21T22:26:01.472426Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Make Videos","metadata":{}},{"cell_type":"code","source":"%cd /kaggle/working\n\nvideo_size = (1280, 720)\n\nout1 = cv2.VideoWriter('ScaledYOLOv4.avi',cv2.VideoWriter_fourcc(*'DIVX'), 15, video_size)\nout2 = cv2.VideoWriter('ScaledYOLOv4_augmented.avi',cv2.VideoWriter_fourcc(*'DIVX'), 15, video_size)\n\nfor i in tqdm(range(1250, 1600)):\n# for i in tqdm(range(len(image_paths))):\n    TEST_IMAGE_PATH = image_paths[i]\n    img = cv2.imread(TEST_IMAGE_PATH)\n    out_image0 = draw_yolox_predictions(img, gt[i], [1.0] * len(gt[i]), [0] * len(gt[i]), COCO_CLASSES, (0,255,0))\n    \n    # Draw Base model predictions\n    bboxes, bbclasses, scores = sy4_inference(img, model, SIZE, CONF, verbose = False)\n    out_image = draw_yolox_predictions(out_image0, bboxes, scores, bbclasses, COCO_CLASSES, (0,0,255))\n    out1.write(out_image)\n\n    # Draw augmented model predictions\n    bboxes, bbclasses, scores = sy4_inference_augmented(img, model, AUGMENTED_SIZES, CONF, verbose = False)\n    out_image = draw_yolox_predictions(out_image0, bboxes, scores, bbclasses, COCO_CLASSES, (0,0,255))\n    out2.write(out_image)\n    \nout1.release()\n\n# Compress video files\n\nAVI2MP4 = \"-ac 2 -b:v 2000k -c:a aac -c:v libx264 -b:a 160k -vprofile high -bf 0 -strict experimental -f mp4\"\n\ncommand = f\"{FFMPEG_BIN} -i ScaledYOLOv4.avi {AVI2MP4} ScaledYOLOv4.mp4\"\nsubprocess.call(command, shell=True)\n\ncommand = f\"{FFMPEG_BIN} -i ScaledYOLOv4_augmented.avi {AVI2MP4} ScaledYOLOv4_augmented.mp4\"\nsubprocess.call(command, shell=True)","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-21T22:26:09.15792Z","iopub.execute_input":"2022-02-21T22:26:09.158656Z","iopub.status.idle":"2022-02-21T22:32:43.902539Z","shell.execute_reply.started":"2022-02-21T22:26:09.158617Z","shell.execute_reply":"2022-02-21T22:32:43.901674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Display Video","metadata":{}},{"cell_type":"code","source":"from IPython.display import HTML\nfrom base64 import b64encode\n\ndef play(filename):\n    html = ''\n    video = open(filename,'rb').read()\n    src = 'data:video/mp4;base64,' + b64encode(video).decode()\n    html += '<video width=800 controls autoplay loop><source src=\"%s\" type=\"video/mp4\"></video>' % src \n    return HTML(html)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-21T22:32:59.277863Z","iopub.execute_input":"2022-02-21T22:32:59.278507Z","iopub.status.idle":"2022-02-21T22:32:59.283956Z","shell.execute_reply.started":"2022-02-21T22:32:59.278466Z","shell.execute_reply":"2022-02-21T22:32:59.283089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Scaled YOLOv4 without size TTA","metadata":{}},{"cell_type":"code","source":"play('ScaledYOLOv4.mp4')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-21T22:33:00.557153Z","iopub.execute_input":"2022-02-21T22:33:00.557746Z","iopub.status.idle":"2022-02-21T22:33:00.682219Z","shell.execute_reply.started":"2022-02-21T22:33:00.557693Z","shell.execute_reply":"2022-02-21T22:33:00.679822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Scaled YOLOv4 with size TTA","metadata":{}},{"cell_type":"code","source":"play('ScaledYOLOv4_augmented.mp4')","metadata":{"execution":{"iopub.status.busy":"2022-02-21T22:33:02.215702Z","iopub.execute_input":"2022-02-21T22:33:02.216202Z","iopub.status.idle":"2022-02-21T22:33:02.340454Z","shell.execute_reply.started":"2022-02-21T22:33:02.216163Z","shell.execute_reply":"2022-02-21T22:33:02.339639Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# SUBMIT PREDICTION TO COMPETITION","metadata":{}},{"cell_type":"code","source":"%cd /kaggle/working/","metadata":{"execution":{"iopub.status.busy":"2022-02-21T02:57:03.911795Z","iopub.execute_input":"2022-02-21T02:57:03.912347Z","iopub.status.idle":"2022-02-21T02:57:03.91822Z","shell.execute_reply.started":"2022-02-21T02:57:03.912308Z","shell.execute_reply":"2022-02-21T02:57:03.917528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import greatbarrierreef\n\nenv = greatbarrierreef.make_env()   # initialize the environment\niter_test = env.iter_test()  ","metadata":{"execution":{"iopub.status.busy":"2022-02-21T02:57:04.746521Z","iopub.execute_input":"2022-02-21T02:57:04.747051Z","iopub.status.idle":"2022-02-21T02:57:04.781257Z","shell.execute_reply.started":"2022-02-21T02:57:04.747014Z","shell.execute_reply":"2022-02-21T02:57:04.780537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for (image_np, sample_prediction_df) in iter_test:\n    img0 = image_np[:,:,::-1]\n    \n    if USE_AUGMENTED_INFERENCE:\n        bboxes, bbclasses, scores = sy4_inference_augmented(img0, model, AUGMENTED_SIZES, CONF, verbose = False)\n    else:\n        bboxes, bbclasses, scores = sy4_inference(img0, model, SIZE, CONF, verbose = False)\n\n    predictions = []\n    for i in range(len(bboxes)):\n        box = bboxes[i]\n        cls_id = int(bbclasses[i])\n        score = scores[i]\n        if score < CONF:\n            continue\n        x_min = int(box[0])\n        y_min = int(box[1])\n        \n        bbox_width = int(box[2])\n        bbox_height = int(box[3])\n        \n        predictions.append('{:.2f} {} {} {} {}'.format(score, x_min, y_min, bbox_width, bbox_height))\n    \n    prediction_str = ' '.join(predictions)\n    sample_prediction_df['annotations'] = prediction_str\n    env.predict(sample_prediction_df)\n\n    print('Prediction:', prediction_str)","metadata":{"execution":{"iopub.status.busy":"2022-02-21T02:57:06.611573Z","iopub.execute_input":"2022-02-21T02:57:06.611929Z","iopub.status.idle":"2022-02-21T02:57:07.852957Z","shell.execute_reply.started":"2022-02-21T02:57:06.611893Z","shell.execute_reply":"2022-02-21T02:57:07.851844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df = pd.read_csv('submission.csv')\nsub_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-21T02:57:10.379784Z","iopub.execute_input":"2022-02-21T02:57:10.380353Z","iopub.status.idle":"2022-02-21T02:57:10.393206Z","shell.execute_reply.started":"2022-02-21T02:57:10.380315Z","shell.execute_reply":"2022-02-21T02:57:10.392518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Cleanup","metadata":{}},{"cell_type":"code","source":"%cd /kaggle/working\n!rm *.avi\n!rm -r ffmpeg*\n!rm -r ScaledYOLOv4","metadata":{},"execution_count":null,"outputs":[]}]}