{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# [Tensorflow - Help Protect the Great Barrier Reef](https://www.kaggle.com/c/tensorflow-great-barrier-reef)\n> Detect crown-of-thorns starfish in underwater image data\n\n<img src=\"https://storage.googleapis.com/kaggle-competitions/kaggle/31703/logos/header.png?t=2021-10-29-00-30-04\">","metadata":{}},{"cell_type":"markdown","source":"# Save your upvote for:\n * OG Notebook: [Stop guessing CONF - systematically evaluate F2](https://www.kaggle.com/alexchwong/stop-guessing-conf-systematically-evaluate-f2)\n * Training and Infer Yolov5 Notebook: [Awsaf](https://www.kaggle.com/awsaf49)\n > * [TRAIN](https://www.kaggle.com/awsaf49/great-barrier-reef-yolov5-train), \n > * [INFER](https://www.kaggle.com/awsaf49/great-barrier-reef-yolov5-infer/notebook)\n \n I'm just modified [Alex Wong](https://www.kaggle.com/alexchwong) notebook for Yolov5 b/c lots of people are experimenting on this model. \n\n# Why this notebook\n* Each model is different in how the CONF parameter decides PRECISION and RECALL\n* Finding which CONF level is often a matter of guesswork, and you are limited by 5 submissions per day.\n* By systematically evaluating F2 score at each level of CONF, you can have a good idea which CONF will give you the best competition F2 metric.\n\n* This assumes:\n  * Your validation dataset is similar to the hidden test set\n  * Your model has not seen your validation dataset\n\n### What's new in this notebook?\n* F2 is evaluated at every CONF level on validation dataset, similar to F1_curve.png for YoloV5\n* Define 'eval_IOU' for which IOU level to evaluate F2 at\n  * (Competition metric tests IOU @ 0.3 to 0.8 with step of 0.05)\n\n### Example model used in this notebook\n\n* This notebook uses the YOLOV5 model kindly provided byAwsaf (https://www.kaggle.com/awsaf49/great-barrier-reef-yolov5-train)\n  * And assumes it is validate on FOLD=1 of 5-fold split performed in: https://www.kaggle.com/awsaf49/greatbarrierreef-yolov5-train-ds (well, idk what fold he's trained on on this dataset, but you can get the idea)\n\n### Warning\n* Note the iteration time as an estimate to how long this notebook takes to run.\n* Minimise wasting precious GPU minutes running at large resolutions. Test at small resolutions! The CONF peak should theoretically be the same.","metadata":{}},{"cell_type":"markdown","source":"# 🛠 Install Libraries","metadata":{}},{"cell_type":"code","source":"# bbox-utility, check https://github.com/awsaf49/bbox for source code\n# !pip install -q ../input/loguru-lib-ds/loguru-0.5.3-py3-none-any.whl\n# !pip install -q ../input/bbox-lib-ds","metadata":{"execution":{"iopub.status.busy":"2022-01-23T00:28:53.046404Z","iopub.execute_input":"2022-01-23T00:28:53.046963Z","iopub.status.idle":"2022-01-23T00:28:53.068053Z","shell.execute_reply.started":"2022-01-23T00:28:53.04685Z","shell.execute_reply":"2022-01-23T00:28:53.067361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 📚 Import Libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np\nfrom tqdm.notebook import tqdm\ntqdm.pandas()\nimport pandas as pd\nimport os\nimport cv2\nimport matplotlib.pyplot as plt\nimport glob\nimport shutil\nimport sys\nsys.path.append('../input/tensorflow-great-barrier-reef')\nimport torch\nfrom PIL import Image\nimport ast\nimport albumentations as albu","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-01-23T00:28:53.06973Z","iopub.execute_input":"2022-01-23T00:28:53.070189Z","iopub.status.idle":"2022-01-23T00:28:56.597704Z","shell.execute_reply.started":"2022-01-23T00:28:53.070154Z","shell.execute_reply":"2022-01-23T00:28:56.596893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 📌 Key-Points\n* One have to submit prediction using the provided **python time-series API**, which makes this competition different from previous Object Detection Competitions.\n* Each prediction row needs to include all bounding boxes for the image. Submission is format seems also **COCO** which means `[x_min, y_min, width, height]`\n* Copmetition metric `F2` tolerates some false positives(FP) in order to ensure very few starfish are missed. Which means tackling **false negatives(FN)** is more important than false positives(FP). \n$$F2 = 5 \\cdot \\frac{precision \\cdot recall}{4\\cdot precision + recall}$$","metadata":{}},{"cell_type":"markdown","source":"## Please Upvote if you find this Helpful","metadata":{}},{"cell_type":"markdown","source":"# 📖 Meta Data\n* `train_images/` - Folder containing training set photos of the form `video_{video_id}/{video_frame}.jpg`.\n\n* `[train/test].csv` - Metadata for the images. As with other test files, most of the test metadata data is only available to your notebook upon submission. Just the first few rows available for download.\n\n* `video_id` - ID number of the video the image was part of. The video ids are not meaningfully ordered.\n* `video_frame` - The frame number of the image within the video. Expect to see occasional gaps in the frame number from when the diver surfaced.\n* `sequence` - ID of a gap-free subset of a given video. The sequence ids are not meaningfully ordered.\n* `sequence_frame` - The frame number within a given sequence.\n* `image_id` - ID code for the image, in the format `{video_id}-{video_frame}`\n* `annotations` - The bounding boxes of any starfish detections in a string format that can be evaluated directly with Python. Does not use the same format as the predictions you will submit. Not available in test.csv. A bounding box is described by the pixel coordinate `(x_min, y_min)` of its lower left corner within the image together with its `width` and `height` in pixels --> (COCO format).","metadata":{}},{"cell_type":"code","source":"ROOT_DIR  = '/kaggle/input/tensorflow-great-barrier-reef/'\nCKPT_PATH = '/kaggle/input/greatbarrierreef-yolov5-train-ds/yolov5/runs/train/exp/weights/best.pt'\nIMG_SIZE  = 1280\nCONF      = 0.01\nIOU       = 0.65\nAUGMENT   = True","metadata":{"execution":{"iopub.status.busy":"2022-01-23T00:28:56.600059Z","iopub.execute_input":"2022-01-23T00:28:56.600587Z","iopub.status.idle":"2022-01-23T00:28:56.605899Z","shell.execute_reply.started":"2022-01-23T00:28:56.600548Z","shell.execute_reply":"2022-01-23T00:28:56.604914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%cd /kaggle/working\n\nfrom sklearn.model_selection import GroupKFold\n\ndef get_bbox(annots):\n    bboxes = [list(annot.values()) for annot in annots]\n    return bboxes\n\ndef get_path(row):\n    row['image_path'] = f'{ROOT_DIR}/train_images/video_{row.video_id}/{row.video_frame}.jpg'\n    return row\n\nROOT_DIR  = '/kaggle/input/tensorflow-great-barrier-reef/'\n\ndf = pd.read_csv(\"/kaggle/input/tensorflow-great-barrier-reef/train.csv\")\n\n# Don't filter for annotated frames. Include frames with no bboxes as well!\ndf[\"num_bbox\"] = df['annotations'].apply(lambda x: str.count(x, 'x'))\ndf_train = df\n\n# Annotations \ndf_train['annotations'] = df_train['annotations'].progress_apply(lambda x: ast.literal_eval(x))\ndf_train['bboxes'] = df_train.annotations.progress_apply(get_bbox)\n\ndf_train = df_train.progress_apply(get_path, axis=1)\n\nkf = GroupKFold(n_splits = 5) \ndf_train = df_train.reset_index(drop=True)\ndf_train['fold'] = -1\nfor fold, (train_idx, val_idx) in enumerate(kf.split(df_train, y = df_train.video_id.tolist(), groups=df_train.sequence)):\n    df_train.loc[val_idx, 'fold'] = fold\n\ndf_train.head(5)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-23T00:28:56.607936Z","iopub.execute_input":"2022-01-23T00:28:56.608187Z","iopub.status.idle":"2022-01-23T00:29:12.339242Z","shell.execute_reply.started":"2022-01-23T00:28:56.608153Z","shell.execute_reply":"2022-01-23T00:29:12.338495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-23T00:29:12.34054Z","iopub.execute_input":"2022-01-23T00:29:12.340988Z","iopub.status.idle":"2022-01-23T00:29:12.356473Z","shell.execute_reply.started":"2022-01-23T00:29:12.34095Z","shell.execute_reply":"2022-01-23T00:29:12.355694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Number of BBoxes","metadata":{}},{"cell_type":"code","source":"FDA_reference = df_train[df_train['annotations']!='[]']\nFDA_trans = albu.FDA(FDA_reference['image_path'].values)","metadata":{"execution":{"iopub.status.busy":"2022-01-23T00:29:12.357879Z","iopub.execute_input":"2022-01-23T00:29:12.358146Z","iopub.status.idle":"2022-01-23T00:29:12.372328Z","shell.execute_reply.started":"2022-01-23T00:29:12.358109Z","shell.execute_reply":"2022-01-23T00:29:12.371439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = (df.num_bbox>0).value_counts()/len(df)*100\nprint(f\"No BBox: {data[0]:0.2f}% | With BBox: {data[1]:0.2f}%\")","metadata":{"execution":{"iopub.status.busy":"2022-01-23T00:29:12.373924Z","iopub.execute_input":"2022-01-23T00:29:12.374502Z","iopub.status.idle":"2022-01-23T00:29:12.383288Z","shell.execute_reply.started":"2022-01-23T00:29:12.374466Z","shell.execute_reply":"2022-01-23T00:29:12.382488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🔨 Helper","metadata":{}},{"cell_type":"code","source":"# Modified from https://www.kaggle.com/awsaf49/great-barrier-reef-yolov5-infer\n# Additions: \n#     confidence scores\ndef voc2yolo(bboxes, image_height=720, image_width=1280):\n    \"\"\"\n    voc  => [x1, y1, x2, y1]\n    yolo => [xmid, ymid, w, h] (normalized)\n    \"\"\"\n    \n    bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n    \n    bboxes[..., [0, 2]] = bboxes[..., [0, 2]]/ image_width\n    bboxes[..., [1, 3]] = bboxes[..., [1, 3]]/ image_height\n    \n    w = bboxes[..., 2] - bboxes[..., 0]\n    h = bboxes[..., 3] - bboxes[..., 1]\n    \n    bboxes[..., 0] = bboxes[..., 0] + w/2\n    bboxes[..., 1] = bboxes[..., 1] + h/2\n    bboxes[..., 2] = w\n    bboxes[..., 3] = h\n    \n    return bboxes\n\ndef yolo2voc(bboxes, image_height=720, image_width=1280):\n    \"\"\"\n    yolo => [xmid, ymid, w, h] (normalized)\n    voc  => [x1, y1, x2, y1]\n    \n    \"\"\" \n    bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n    \n    bboxes[..., [0, 2]] = bboxes[..., [0, 2]]* image_width\n    bboxes[..., [1, 3]] = bboxes[..., [1, 3]]* image_height\n    \n    bboxes[..., [0, 1]] = bboxes[..., [0, 1]] - bboxes[..., [2, 3]]/2\n    bboxes[..., [2, 3]] = bboxes[..., [0, 1]] + bboxes[..., [2, 3]]\n    \n    return bboxes\n\ndef coco2yolo(bboxes, image_height=720, image_width=1280):\n    \"\"\"\n    coco => [xmin, ymin, w, h]\n    yolo => [xmid, ymid, w, h] (normalized)\n    \"\"\"\n    \n    bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n    \n    # normolizinig\n    bboxes[..., [0, 2]]= bboxes[..., [0, 2]]/ image_width\n    bboxes[..., [1, 3]]= bboxes[..., [1, 3]]/ image_height\n    \n    # converstion (xmin, ymin) => (xmid, ymid)\n    bboxes[..., [0, 1]] = bboxes[..., [0, 1]] + bboxes[..., [2, 3]]/2\n    \n    return bboxes\n\ndef yolo2coco(bboxes, image_height=720, image_width=1280):\n    \"\"\"\n    yolo => [xmid, ymid, w, h] (normalized)\n    coco => [xmin, ymin, w, h]\n    \n    \"\"\" \n    bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n    \n    # denormalizing\n    bboxes[..., [0, 2]]= bboxes[..., [0, 2]]* image_width\n    bboxes[..., [1, 3]]= bboxes[..., [1, 3]]* image_height\n    \n    # converstion (xmid, ymid) => (xmin, ymin) \n    bboxes[..., [0, 1]] = bboxes[..., [0, 1]] - bboxes[..., [2, 3]]/2\n    \n    return bboxes\n\ndef voc2coco(bboxes, image_height=720, image_width=1280):\n    bboxes  = voc2yolo(bboxes, image_height, image_width)\n    bboxes  = yolo2coco(bboxes, image_height, image_width)\n    return bboxes\n\n\ndef load_image(image_path):\n    return cv2.cvtColor(cv2.imread(image_path), cv2.COLOR_BGR2RGB)\n\n\ndef plot_one_box(x, img, score=None, color=None, label=None, line_thickness=None):\n    # Plots one bounding box on image img\n    tl = line_thickness or round(0.002 * (img.shape[0] + img.shape[1]) / 2) + 1  # line/font thickness\n    color = color or [random.randint(0, 255) for _ in range(3)]\n    c1, c2 = (int(x[0]), int(x[1])), (int(x[2]), int(x[3]))\n    cv2.rectangle(img, c1, c2, color, thickness=tl, lineType=cv2.LINE_AA)\n    if label:\n        tf = max(tl - 1, 1)  # font thickness\n        t_size = cv2.getTextSize(label, 0, fontScale=tl / 3, thickness=tf)[0]\n        c2 = c1[0] + t_size[0], c1[1] - t_size[1] - 3\n        cv2.rectangle(img, c1, c2, color, -1, cv2.LINE_AA)  # filled\n        cv2.putText(img, \"{}:{:.2f}\".format(label, score), (c1[0], c1[1] - 2), 0, tl / 3, [225, 255, 255], thickness=tf, lineType=cv2.LINE_AA)\n\ndef draw_bboxes(img, bboxes, scores, classes, class_ids, colors = None, show_classes = None, bbox_format = 'yolo', class_name = False, line_thickness = 2):  \n     \n    image = img.copy()\n    show_classes = classes if show_classes is None else show_classes\n    colors = (0, 255 ,0) if colors is None else colors\n    \n    if bbox_format == 'yolo':\n        \n        for idx in range(len(bboxes)):  \n            \n            bbox  = bboxes[idx]\n            cls   = classes[idx]\n            cls_id = class_ids[idx]\n            try:\n                score = scores[idx]\n            except:\n                score = None\n            color = colors[cls_id] if type(colors) is list else colors\n            \n            if cls in show_classes:\n            \n                x1 = round(float(bbox[0])*image.shape[1])\n                y1 = round(float(bbox[1])*image.shape[0])\n                w  = round(float(bbox[2])*image.shape[1]/2) #w/2 \n                h  = round(float(bbox[3])*image.shape[0]/2)\n\n                voc_bbox = (x1-w, y1-h, x1+w, y1+h)\n                plot_one_box(voc_bbox, \n                             image,\n                             score= score if score else None,\n                             color = color,\n                             label = cls if class_name else str(get_label(cls)),\n                             line_thickness = line_thickness)\n            \n    elif bbox_format == 'coco':\n        \n        for idx in range(len(bboxes)):  \n            \n            bbox  = bboxes[idx]\n            cls   = classes[idx]\n            cls_id = class_ids[idx]\n            try:\n                score = scores[idx]\n            except:\n                score = None\n            color = colors[cls_id] if type(colors) is list else colors\n            \n            if cls in show_classes:            \n                x1 = int(round(bbox[0]))\n                y1 = int(round(bbox[1]))\n                w  = int(round(bbox[2]))\n                h  = int(round(bbox[3]))\n\n                voc_bbox = (x1, y1, x1+w, y1+h)\n                plot_one_box(voc_bbox, \n                             image,\n                             score = score if score else None,\n                             color = color,\n                             label = cls if class_name else str(cls_id),\n                             line_thickness = line_thickness)\n                \n    elif bbox_format == 'voc_pascal':\n        \n        for idx in range(len(bboxes)):  \n            \n            bbox  = bboxes[idx]\n            cls   = classes[idx]\n            cls_id = class_ids[idx]\n            try:\n                score = scores[idx]\n            except:\n                score = None\n            color = colors[cls_id] if type(colors) is list else colors\n            \n            if cls in show_classes: \n                x1 = int(round(bbox[0]))\n                y1 = int(round(bbox[1]))\n                x2 = int(round(bbox[2]))\n                y2 = int(round(bbox[3]))\n                voc_bbox = (x1, y1, x2, y2)\n                plot_one_box(voc_bbox, \n                             image,\n                             score = score if score else None,\n                             color = color,\n                             label = cls if class_name else str(cls_id),\n                             line_thickness = line_thickness)\n    else:\n        raise ValueError('wrong bbox format')\n\n    return image\n\ndef get_bbox(annots):\n    bboxes = [list(annot.values()) for annot in annots]\n    return bboxes\n\ndef get_imgsize(row):\n    row['width'], row['height'] = imagesize.get(row['image_path'])\n    return row\n\nnp.random.seed(32)\ncolors = [(np.random.randint(255), np.random.randint(255), np.random.randint(255))\\\n          for idx in range(1)]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-23T00:29:12.384758Z","iopub.execute_input":"2022-01-23T00:29:12.385397Z","iopub.status.idle":"2022-01-23T00:29:12.424702Z","shell.execute_reply.started":"2022-01-23T00:29:12.385361Z","shell.execute_reply":"2022-01-23T00:29:12.4239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def predict(model, img, size=768, augment=False):\n    height, width = img.shape[:2]\n    results = model(img, size=size, augment=augment)  # custom inference size\n    preds   = results.pandas().xyxy[0]\n    bboxes  = preds[['xmin','ymin','xmax','ymax']].values\n    if len(bboxes):\n        bboxes  = voc2coco(bboxes,height,width).astype(int)\n        confs   = preds.confidence.values\n        return bboxes, confs\n    else:\n        return [],[]\n    \ndef format_prediction(bboxes, confs):\n    annot = ''\n    if len(bboxes)>0:\n        for idx in range(len(bboxes)):\n            xmin, ymin, w, h = bboxes[idx]\n            conf             = confs[idx]\n            annot += f'{conf} {xmin} {ymin} {w} {h}'\n            annot +=' '\n        annot = annot.strip(' ')\n    return annot\n\ndef show_img(img, bboxes, confis, bbox_format='yolo', colors=colors):\n    names  = ['starfish']*len(bboxes)\n    labels = [0]*len(bboxes)\n    img    = draw_bboxes(img = img,\n                           bboxes = bboxes,\n                           scores = confis,\n                           classes = names,\n                           class_ids = labels,\n                           class_name = True, \n                           colors = colors, \n                           bbox_format = bbox_format,\n                           line_thickness = 2)\n    return Image.fromarray(img)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-23T00:29:12.426124Z","iopub.execute_input":"2022-01-23T00:29:12.426607Z","iopub.status.idle":"2022-01-23T00:29:12.439098Z","shell.execute_reply.started":"2022-01-23T00:29:12.426569Z","shell.execute_reply":"2022-01-23T00:29:12.438244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def IOU_coco(bbox1, bbox2):\n    '''\n        adapted from https://stackoverflow.com/questions/25349178/calculating-percentage-of-bounding-box-overlap-for-image-detector-evaluation\n    '''\n    x_left = max(bbox1[0], bbox2[0])\n    y_top = max(bbox1[1], bbox2[1])\n    x_right = min(bbox1[0] + bbox1[2], bbox2[0] + bbox2[2])\n    y_bottom = min(bbox1[1] + bbox1[3], bbox2[1] + bbox2[3])\n    if x_right < x_left or y_bottom < y_top:\n        return 0.0\n    intersection_area = (x_right - x_left) * (y_bottom - y_top)\n    bb1_area = bbox1[2] * bbox1[3]\n    bb2_area = bbox2[2] * bbox2[3]\n    iou = intersection_area / float(bb1_area + bb2_area - intersection_area)\n\n    assert iou >= 0.0\n    assert iou <= 1.0\n    return iou","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-23T00:29:12.442836Z","iopub.execute_input":"2022-01-23T00:29:12.44352Z","iopub.status.idle":"2022-01-23T00:29:12.453219Z","shell.execute_reply.started":"2022-01-23T00:29:12.443456Z","shell.execute_reply":"2022-01-23T00:29:12.452422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!mkdir -p /root/.config/Ultralytics\n!cp /kaggle/input/yolov5-font/Arial.ttf /root/.config/Ultralytics/","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-23T00:29:12.454546Z","iopub.execute_input":"2022-01-23T00:29:12.455048Z","iopub.status.idle":"2022-01-23T00:29:13.778115Z","shell.execute_reply.started":"2022-01-23T00:29:12.455008Z","shell.execute_reply":"2022-01-23T00:29:13.777261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_model(ckpt_path, conf=0.01, iou=0.50):\n    model = torch.hub.load('/kaggle/input/yolov5-lib-ds',\n                           'custom',\n                           path=ckpt_path,\n                           source='local',\n                           force_reload=True)  # local repo\n    model.conf = conf  # NMS confidence threshold\n    model.iou  = iou  # NMS IoU threshold\n    model.classes = None   # (optional list) filter by class, i.e. = [0, 15, 16] for persons, cats and dogs\n    model.multi_label = False  # NMS multiple labels per box\n    model.max_det = 1000  # maximum number of detections per image\n    return model","metadata":{"execution":{"iopub.status.busy":"2022-01-23T00:29:13.780284Z","iopub.execute_input":"2022-01-23T00:29:13.780587Z","iopub.status.idle":"2022-01-23T00:29:13.787058Z","shell.execute_reply.started":"2022-01-23T00:29:13.780548Z","shell.execute_reply":"2022-01-23T00:29:13.78628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Modified from https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507\n# Additions: \n#     auto converts to xywh format\n#     converts tensors to list of floats\n# Updates:\n# i changed it to yolov5 version. code is still dirty\ndef yolov5_inference(img, model, test_size, conf_threshold = 0.4):\n    bboxes = []\n    bbclasses = []\n    scores = []\n    \n    preproc = ValTransform(legacy = False)\n    \n    tensor_img, _ = preproc(img, None, test_size)\n    tensor_img = torch.from_numpy(tensor_img).unsqueeze(0)\n    tensor_img = tensor_img.float()\n    tensor_img = tensor_img.cuda()\n    \n    with torch.no_grad():\n        outputs = model(tensor_img)\n        outputs = postprocess(\n                    outputs, num_classes, conf_threshold,\n                    nmsthre, class_agnostic=True\n                )\n    \n    if outputs[0] is None:\n        return [], [], []\n    \n    outputs = outputs[0].cpu()\n    bboxes = outputs[:, 0:4]\n    \n    bboxes /= min(test_size[0] / img.shape[0], test_size[1] / img.shape[1])\n    bbclasses = outputs[:, 6]\n    scores = outputs[:, 4] * outputs[:, 5]\n    \n    if len(bboxes) == 0:\n        return [], [], []\n    \n    bboxes = bboxes.numpy()\n    \n    # format to coco\n    bboxes[:, 2] = bboxes[:, 2] - bboxes[:, 0]\n    bboxes[:, 3] = bboxes[:, 3] - bboxes[:, 1]    \n    \n    # Converts tensors to lists\n    return bboxes, bbclasses.tolist(), scores.tolist()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-23T00:29:13.788612Z","iopub.execute_input":"2022-01-23T00:29:13.789175Z","iopub.status.idle":"2022-01-23T00:29:13.801353Z","shell.execute_reply.started":"2022-01-23T00:29:13.789136Z","shell.execute_reply":"2022-01-23T00:29:13.800582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Modified from https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507\n# Additions: \n#     allows customized box color (BGR)\n# Updates:\n# i changed it to yolov5 version. code is still dirty\ndef draw_yolov5_predictions(img, bboxes, scores, bbclasses, classes_dict, boxcolor = (0,0,255)):\n    outimg = img.copy()\n    for i in range(len(bboxes)):\n        box = bboxes[i]\n        cls_id = int(bbclasses[i])\n        score = scores[i]\n        x0 = int(box[0])\n        y0 = int(box[1])\n        x1 = x0 + int(box[2])\n        y1 = y0 + int(box[3])\n\n        cv2.rectangle(outimg, (x0, y0), (x1, y1), boxcolor, 2)\n        cv2.putText(outimg, '{}:{:.1f}%'.format(classes_dict[cls_id], score * 100), (x0, y0 - 3), cv2.FONT_HERSHEY_PLAIN, 0.8, boxcolor, thickness = 1)\n    return outimg","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-23T00:29:13.803592Z","iopub.execute_input":"2022-01-23T00:29:13.804239Z","iopub.status.idle":"2022-01-23T00:29:13.812914Z","shell.execute_reply.started":"2022-01-23T00:29:13.804199Z","shell.execute_reply":"2022-01-23T00:29:13.812225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🔭 Inference","metadata":{}},{"cell_type":"markdown","source":"## Helper","metadata":{}},{"cell_type":"code","source":"COCO_CLASSES = (\n  \"starfish\",\n)\n\n# YOLOv5 Inference Size \nIMG_SIZE = 1280\nAUGMENT  = True\nCONF     = 0.01\nIOU = 0.65\n\n# Which IOU level to evaluate (Competition metric tests 0.3 to 0.8 with step of 0.05)\neval_IOU = 0.65\n\nMODEL_PATH = '../input/greatbarrierreef-yolov5-train-ds/yolov5/great-barrier-reef-public/yolov5m-dim1280-fold4/weights/best.pt'","metadata":{"execution":{"iopub.status.busy":"2022-01-23T00:29:13.814312Z","iopub.execute_input":"2022-01-23T00:29:13.814875Z","iopub.status.idle":"2022-01-23T00:29:13.824006Z","shell.execute_reply.started":"2022-01-23T00:29:13.814825Z","shell.execute_reply":"2022-01-23T00:29:13.823249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# load model\nmodel = load_model(MODEL_PATH, conf=CONF, iou=IOU)\n# print(model)","metadata":{"execution":{"iopub.status.busy":"2022-01-23T00:29:13.82534Z","iopub.execute_input":"2022-01-23T00:29:13.825824Z","iopub.status.idle":"2022-01-23T00:29:20.649281Z","shell.execute_reply.started":"2022-01-23T00:29:13.825791Z","shell.execute_reply":"2022-01-23T00:29:20.648508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Run Inference on **Not seen Data**","metadata":{}},{"cell_type":"code","source":"SELECTED_FOLD = 1\ndf_test = df_train[df_train.fold == SELECTED_FOLD]","metadata":{"execution":{"iopub.status.busy":"2022-01-23T00:29:20.650772Z","iopub.execute_input":"2022-01-23T00:29:20.651353Z","iopub.status.idle":"2022-01-23T00:29:20.658994Z","shell.execute_reply.started":"2022-01-23T00:29:20.651312Z","shell.execute_reply":"2022-01-23T00:29:20.658304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Get image paths and ground truth BB's","metadata":{}},{"cell_type":"code","source":"import copy\n\n# deepcopy is required to avoid \n\ndf_sample = df_test\nimage_paths = df_sample.image_path.tolist()\ngt = copy.deepcopy(df_sample.bboxes.tolist())\ngtmem = copy.deepcopy(df_sample.bboxes.tolist())","metadata":{"execution":{"iopub.status.busy":"2022-01-23T00:29:20.660057Z","iopub.execute_input":"2022-01-23T00:29:20.660325Z","iopub.status.idle":"2022-01-23T00:29:20.718651Z","shell.execute_reply.started":"2022-01-23T00:29:20.66029Z","shell.execute_reply":"2022-01-23T00:29:20.718014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Test your model is working","metadata":{}},{"cell_type":"code","source":"df_test[df_test.num_bbox > 3].image_path.to_list()[0]","metadata":{"execution":{"iopub.status.busy":"2022-01-23T00:29:20.719827Z","iopub.execute_input":"2022-01-23T00:29:20.720216Z","iopub.status.idle":"2022-01-23T00:29:20.729476Z","shell.execute_reply.started":"2022-01-23T00:29:20.720179Z","shell.execute_reply":"2022-01-23T00:29:20.728641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_paths[2135]","metadata":{"execution":{"iopub.status.busy":"2022-01-23T00:29:20.731015Z","iopub.execute_input":"2022-01-23T00:29:20.731275Z","iopub.status.idle":"2022-01-23T00:29:20.738544Z","shell.execute_reply.started":"2022-01-23T00:29:20.731241Z","shell.execute_reply":"2022-01-23T00:29:20.73775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"i = 2135\nTEST_IMAGE_PATH = image_paths[i]\nimg = cv2.imread(TEST_IMAGE_PATH)[...,::-1]\n\n# img = TEST_IMAGE_PATH\n\n# bboxes, bbclasses, scores, img = tempt_yolov5_inference(model, img)\nbboxes, confis = predict(model, img, size=IMG_SIZE, augment=AUGMENT)\n\nprint(gt[i])\nprint(bboxes)\n\n# # Draw Green ground truth box\n# out_image = draw_yolov5_predictions(img, gt[i], [1.0] * len(gt[i]), [0] * len(gt[i]), COCO_CLASSES, (0,255,0))\nout_image = show_img(img, gt[i],[1.0]*len(gt[i]), bbox_format='coco', colors=[(0, 255, 0)]*(len(bboxes)))\n\n# # Draw Red inference box\nout_image = show_img(np.array(out_image), bboxes,confis, bbox_format='coco', colors=[(0,0,255)]*(len(bboxes)))\n\ndisplay(out_image)","metadata":{"execution":{"iopub.status.busy":"2022-01-23T00:29:20.739701Z","iopub.execute_input":"2022-01-23T00:29:20.740392Z","iopub.status.idle":"2022-01-23T00:29:25.896829Z","shell.execute_reply.started":"2022-01-23T00:29:20.740358Z","shell.execute_reply":"2022-01-23T00:29:25.895959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Assess Model Performance","metadata":{}},{"cell_type":"code","source":"%cd /kaggle/working\n\n# Confidence scores of true positives, false positives and count false negatives\nTP = [] # Confidence scores of true positives\nFP = [] # Confidence scores of true positives\nFN = 0  # Count of false negative boxes\n\nfor i in tqdm(range(len(image_paths))):\n    TEST_IMAGE_PATH = image_paths[i]\n    img = cv2.imread(TEST_IMAGE_PATH)\n    img = cv2.imread(TEST_IMAGE_PATH)[...,::-1]\n    bboxes, scores = predict(model, img, size=IMG_SIZE, augment=AUGMENT)\n\n    # Test YOLOV5\n    gt0 = gt[i]\n    if len(bboxes) == 0:\n        # all gt are false negative\n        FN += len(gt0)\n    else:\n        bb = bboxes.copy().tolist()\n        for idx, b in enumerate(bb):\n            b.append(scores[idx])\n        bb.sort(key = lambda x: x[4], reverse = True)\n        \n        if len(gt0) == 0:\n            # all bboxes are false positives\n            for b in bb:\n                FP.append(b[4])\n        else:\n            # match bbox with gt\n            for b in bb:\n                matched = False\n                for g in gt0:\n                    # check whether gt box is already matched to an inference bb\n                    if len(g) == 4:\n                        # g bbox is unmatched\n                        if IOU_coco(b, g) >= eval_IOU:\n                            g.append(b[4]) # assign confidence values to g; marks g as matched\n                            matched = True\n                            TP.append(b[4])\n                            break\n                if not matched:\n                    FP.append(b[4])\n            for g in gt0:\n                if len(g) == 4:\n                    FN += 1","metadata":{"execution":{"iopub.status.busy":"2022-01-23T00:29:25.897955Z","iopub.execute_input":"2022-01-23T00:29:25.898221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Display your model's Metrics","metadata":{}},{"cell_type":"code","source":"from matplotlib import pyplot as plt\n%matplotlib inline\n\nplt.hist(TP, 100)\nplt.title(\"CONF of true positives, base YOLOV5\")\nplt.xlabel('CONF')\nplt.ylabel('TP count')\nplt.show()\n\nprint(f'True positives = {len(TP)}')\nprint(f'False negatives = {FN}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist(FP, 100)\nplt.title(\"CONF of false positives, base YOLOX\")\nplt.xlabel('CONF')\nplt.ylabel('FP count')\nplt.show()\n\nprint(f'False positives = {len(FP)}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"F2list = []\nF2max = 0.0\nF2maxat = -1.0\n\nfor c in np.arange(0.0, 1.0, 0.01):\n    FNcount = FN + sum(1 for i in TP if i < c)\n    TPcount = sum(1 for i in TP if i >= c)\n    FPcount = sum(1 for i in FP if i >= c)\n    R = TPcount / (TPcount + FNcount + 0.0001)\n    P = TPcount / (TPcount + FPcount + 0.0001)\n    F2 = (5 * P * R) / (4 * P + R + 0.0001)\n    F2list.append((c, F2))\n    if F2max < F2:\n        F2max = F2\n        F2maxat = c\n\nplt.scatter(*zip(*F2list))\nplt.title(\"CONF vs F2 score\")\nplt.xlabel('CONF')\nplt.ylabel('F2')\nplt.show()\n\nprint(f'F2 max is {F2max} at CONF = {F2maxat}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Please Upvote if you find this Helpful","metadata":{}},{"cell_type":"markdown","source":"<img src=\"https://www.pngall.com/wp-content/uploads/2018/04/Under-Construction-PNG-File.png\">","metadata":{}}]}