{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Why this notebook?\n* Thanks to everyone who upvoted my previous notebook which is now gold medal status: https://www.kaggle.com/alexchwong/stop-guessing-conf-systematically-evaluate-f2\n* The popularity of this and its cousin notebook by @locbaop: https://www.kaggle.com/locbaop/systematic-evaluate-f2-yolov5 suggest it is useful.\n* The highest scoring public notebooks use yolov5 with Norfair tracking\n* So is Norfair actually increasing F2 score, or is it just changing the F2/CONF curve?\n\n# Why evaluate F2 score systematically?\n* Each model is different in how the CONF parameter decides PRECISION and RECALL\n* Finding which CONF level is often a matter of guesswork, and you are limited by 5 submissions per day.\n* By systematically evaluating F2 score at each level of CONF, you can have a good idea which CONF will give you the best competition F2 metric.\n\n* This assumes:\n  * Your validation dataset is similar to the hidden test set\n  * Your model has not seen your validation dataset\n\n### Example model used in this notebook\n\n* This notebook uses the YOLOX model kindly provided by @awsaf (https://www.kaggle.com/awsaf49/greatbarrierreef-yolov5-train-ds)\n  * And assumes it is trained FOLD-4 as the model name implies\n* Norfair tracking is implemented and code modified from @parapapapam (https://www.kaggle.com/parapapapam/yolox-inference-tracking-on-cots-lb-0-539)\n\n### Warning\n* Note the iteration time as an estimate to how long this notebook takes to run.\n* Minimise wasting precious GPU minutes running at large resolutions. Test at small resolutions! The CONF peak should theoretically be the same.","metadata":{}},{"cell_type":"markdown","source":"### By popular demand, this notebook now tests all IOUs in the Competition metric F2","metadata":{}},{"cell_type":"code","source":"import numpy as np\n\ntest_IOU = np.linspace(0.3, 0.8, 11)\ntest_IOU","metadata":{"execution":{"iopub.status.busy":"2022-02-03T01:00:30.923962Z","iopub.execute_input":"2022-02-03T01:00:30.924518Z","iopub.status.idle":"2022-02-03T01:00:30.953599Z","shell.execute_reply.started":"2022-02-03T01:00:30.924424Z","shell.execute_reply":"2022-02-03T01:00:30.952900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Which IOU should be used to evaluate histograms?\n* NB the F2 scores are now calculated as the means at each tested IOU, for each level of confidence\n* Histograms of TPs and FPs are only drawn for the given eval_IOU","metadata":{}},{"cell_type":"code","source":"eval_IOU = 0.5","metadata":{"execution":{"iopub.status.busy":"2022-02-03T01:00:30.954775Z","iopub.execute_input":"2022-02-03T01:00:30.956212Z","iopub.status.idle":"2022-02-03T01:00:30.960700Z","shell.execute_reply.started":"2022-02-03T01:00:30.956173Z","shell.execute_reply":"2022-02-03T01:00:30.959976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Control Panel (change your settings here)","metadata":{}},{"cell_type":"code","source":"# CONTROL PANEL (change your settings here instead)\n\n# The pato to your yolov5 model\nMODEL_PATH = '../input/greatbarrierreef-yolov5-train-ds/yolov5/great-barrier-reef-public/yolov5m-dim1280-fold4/weights/best.pt'\n\n# Confidence cutoff\nINFER_CONF = 0.01\n\n# Inference size\nINFER_SIZE = 1280\n\n# Whether to use yolov5 TTA\nINFER_TTA = True","metadata":{"execution":{"iopub.status.busy":"2022-02-03T01:04:01.536793Z","iopub.execute_input":"2022-02-03T01:04:01.537067Z","iopub.status.idle":"2022-02-03T01:04:01.541780Z","shell.execute_reply.started":"2022-02-03T01:04:01.537024Z","shell.execute_reply":"2022-02-03T01:04:01.540623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Installation and Code","metadata":{}},{"cell_type":"markdown","source":"## Import Modules","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport os\nimport torch\nimport importlib\nimport cv2 \nimport pandas as pd\nimport numpy as np\n\nimport ast\nimport shutil\nimport sys\n\nfrom tqdm.notebook import tqdm\ntqdm.pandas()\n\nfrom PIL import Image\nfrom IPython.display import display\n\nimport copy # for deepcopy nested lists","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-03T01:00:36.769898Z","iopub.execute_input":"2022-02-03T01:00:36.770235Z","iopub.status.idle":"2022-02-03T01:00:38.423601Z","shell.execute_reply.started":"2022-02-03T01:00:36.770195Z","shell.execute_reply":"2022-02-03T01:00:38.422845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## YOLOV5 Installation","metadata":{}},{"cell_type":"code","source":"!mkdir -p /root/.config/Ultralytics\n!cp /kaggle/input/yolov5-font/Arial.ttf /root/.config/Ultralytics/","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-03T01:00:40.505034Z","iopub.execute_input":"2022-02-03T01:00:40.505837Z","iopub.status.idle":"2022-02-03T01:00:41.849559Z","shell.execute_reply.started":"2022-02-03T01:00:40.505798Z","shell.execute_reply":"2022-02-03T01:00:41.848512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define the model here","metadata":{}},{"cell_type":"code","source":"model = torch.hub.load('../input/yolov5-lib-ds', \n                       'custom', \n                       path=MODEL_PATH,\n                       source='local',\n                       force_reload=True)  # local repo\nmodel.conf = 0.01","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-03T01:00:42.681224Z","iopub.execute_input":"2022-02-03T01:00:42.681496Z","iopub.status.idle":"2022-02-03T01:00:49.253011Z","shell.execute_reply.started":"2022-02-03T01:00:42.681466Z","shell.execute_reply":"2022-02-03T01:00:49.252255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## IOU Calculation","metadata":{}},{"cell_type":"code","source":"def IOU_coco(bbox1, bbox2):\n    '''\n        adapted from https://stackoverflow.com/questions/25349178/calculating-percentage-of-bounding-box-overlap-for-image-detector-evaluation\n    '''\n    x_left = max(bbox1[0], bbox2[0])\n    y_top = max(bbox1[1], bbox2[1])\n    x_right = min(bbox1[0] + bbox1[2], bbox2[0] + bbox2[2])\n    y_bottom = min(bbox1[1] + bbox1[3], bbox2[1] + bbox2[3])\n    if x_right < x_left or y_bottom < y_top:\n        return 0.0\n    intersection_area = (x_right - x_left) * (y_bottom - y_top)\n    bb1_area = bbox1[2] * bbox1[3]\n    bb2_area = bbox2[2] * bbox2[3]\n    iou = intersection_area / float(bb1_area + bb2_area - intersection_area)\n\n    assert iou >= 0.0\n    assert iou <= 1.0\n    return iou","metadata":{"execution":{"iopub.status.busy":"2022-02-03T01:00:49.255112Z","iopub.execute_input":"2022-02-03T01:00:49.255388Z","iopub.status.idle":"2022-02-03T01:00:49.262910Z","shell.execute_reply.started":"2022-02-03T01:00:49.255349Z","shell.execute_reply":"2022-02-03T01:00:49.262074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualization function","metadata":{}},{"cell_type":"code","source":"# Modified from https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507\n\ndef draw_yolox_predictions(img, bboxes, scores, bbclasses, classes_dict, boxcolor = (0,0,255)):\n    outimg = img.copy()\n    for i in range(len(bboxes)):\n        box = bboxes[i]\n        cls_id = int(bbclasses[i])\n        score = scores[i]\n        x0 = int(box[0])\n        y0 = int(box[1])\n        x1 = x0 + int(box[2])\n        y1 = y0 + int(box[3])\n\n        cv2.rectangle(outimg, (x0, y0), (x1, y1), boxcolor, 2)\n        cv2.putText(outimg, '{}:{:.1f}%'.format(classes_dict[cls_id], score * 100), (x0, y0 - 3), cv2.FONT_HERSHEY_PLAIN, 0.8, boxcolor, thickness = 1)\n    return outimg\n\nCOCO_CLASSES = (\"starfish\")","metadata":{"execution":{"iopub.status.busy":"2022-02-03T01:00:51.868826Z","iopub.execute_input":"2022-02-03T01:00:51.869107Z","iopub.status.idle":"2022-02-03T01:00:51.876545Z","shell.execute_reply.started":"2022-02-03T01:00:51.869070Z","shell.execute_reply":"2022-02-03T01:00:51.875748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Norfair","metadata":{}},{"cell_type":"markdown","source":"### Installation","metadata":{}},{"cell_type":"code","source":"# norfair dependencies\n%cd /kaggle/input/norfair031py3/\n!pip install commonmark-0.9.1-py2.py3-none-any.whl -f ./ --no-index\n!pip install rich-9.13.0-py3-none-any.whl\n\n!mkdir /kaggle/working/tmp\n!cp -r /kaggle/input/norfair031py3/filterpy-1.4.5/filterpy-1.4.5/ /kaggle/working/tmp/\n%cd /kaggle/working/tmp/filterpy-1.4.5/\n!pip install .\n!rm -rf /kaggle/working/tmp\n\n# norfair\n%cd /kaggle/input/norfair031py3/\n!pip install norfair-0.3.1-py3-none-any.whl -f ./ --no-index","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-03T01:00:54.640716Z","iopub.execute_input":"2022-02-03T01:00:54.641399Z","iopub.status.idle":"2022-02-03T01:02:09.294696Z","shell.execute_reply.started":"2022-02-03T01:00:54.641362Z","shell.execute_reply":"2022-02-03T01:02:09.293858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Functions","metadata":{}},{"cell_type":"code","source":"from norfair import Detection, Tracker\n\n# Helper to convert bbox in format [x_min, y_min, w, h] and scores to norfair.Detection class\ndef to_norfair(bboxes, scores, frame_id):\n    result = []\n    for bb, score in zip(bboxes, scores):\n        xc, yc = bb[0] + int(round(bb[2] / 2.0)), bb[1] + int(round(bb[3] / 2.0))\n        w, h = bb[2], bb[3]\n        result.append(Detection(points=np.array([xc, yc]), scores=np.array([score]), data=np.array([w, h, frame_id])))\n        \n    return result\n\n# Euclidean distance function to match detections on this frame with tracked_objects from previous frames\ndef euclidean_distance(detection, tracked_object):\n    return np.linalg.norm(detection.points - tracked_object.estimate)","metadata":{"execution":{"iopub.status.busy":"2022-02-03T01:02:09.296815Z","iopub.execute_input":"2022-02-03T01:02:09.297108Z","iopub.status.idle":"2022-02-03T01:02:09.359389Z","shell.execute_reply.started":"2022-02-03T01:02:09.297069Z","shell.execute_reply":"2022-02-03T01:02:09.358682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Tracker list initialization","metadata":{}},{"cell_type":"code","source":"tracker = Tracker(\n    distance_function=euclidean_distance, \n    distance_threshold=30,\n    hit_inertia_min=3,\n    hit_inertia_max=6,\n    initialization_delay=1,\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-03T01:02:09.360764Z","iopub.execute_input":"2022-02-03T01:02:09.361006Z","iopub.status.idle":"2022-02-03T01:02:09.365878Z","shell.execute_reply.started":"2022-02-03T01:02:09.360972Z","shell.execute_reply":"2022-02-03T01:02:09.365120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Function to parse YOLOV5 output to tracker","metadata":{}},{"cell_type":"code","source":"def norfair_track(tracker, frame_id, bboxes, bbclasses, scores):\n    \"\"\"\n        inputs:\n        * tracker: the tracker object to process\n        * bboxes, bbclasses, scores: processed model output\n    \"\"\"\n    bboxes_out = []\n    bbclasses_out = []\n    scores_out = []\n    \n    if len(bboxes):\n        bboxes_out = copy.deepcopy(bboxes)\n        bbclasses_out = copy.deepcopy(bbclasses)\n        scores_out = copy.deepcopy(scores)\n    \n    tracked_objects = tracker.update(detections=to_norfair(bboxes, scores, frame_id))\n    \n    for tobj in tracked_objects:\n        bbox_width, bbox_height, last_detected_frame_id = tobj.last_detection.data\n        if last_detected_frame_id == frame_id:  # Skip objects that were detected on current frame\n            continue\n        \n        # print(\"Adding new bbox\")\n        # Add objects that have no detections on current frame to predictions\n        xc, yc = tobj.estimate[0]\n        x_min, y_min = int(round(xc - bbox_width / 2)), int(round(yc - bbox_height / 2))\n        score = tobj.last_detection.scores[0]\n        \n        bboxes_out.append([x_min, y_min, bbox_width, bbox_height])\n        bbclasses_out.append(0)\n        scores_out.append(score)\n    \n    return bboxes_out, bbclasses_out, scores_out","metadata":{"execution":{"iopub.status.busy":"2022-02-03T01:02:09.368214Z","iopub.execute_input":"2022-02-03T01:02:09.368700Z","iopub.status.idle":"2022-02-03T01:02:09.379673Z","shell.execute_reply.started":"2022-02-03T01:02:09.368663Z","shell.execute_reply":"2022-02-03T01:02:09.378935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Retrieve video frames not used in training","metadata":{}},{"cell_type":"code","source":"%cd /kaggle/working\n\nfrom sklearn.model_selection import GroupKFold\n\ndef get_bbox(annots):\n    bboxes = [list(annot.values()) for annot in annots]\n    return bboxes\n\ndef get_path(row):\n    row['image_path'] = f'{ROOT_DIR}/train_images/video_{row.video_id}/{row.video_frame}.jpg'\n    return row\n\nROOT_DIR  = '/kaggle/input/tensorflow-great-barrier-reef/'\n\ndf = pd.read_csv(\"/kaggle/input/tensorflow-great-barrier-reef/train.csv\")\n\n# Don't filter for annotated frames. Include frames with no bboxes as well!\ndf[\"num_bbox\"] = df['annotations'].apply(lambda x: str.count(x, 'x'))\ndf_train = df\n\n# Annotations \ndf_train['annotations'] = df_train['annotations'].progress_apply(lambda x: ast.literal_eval(x))\ndf_train['bboxes'] = df_train.annotations.progress_apply(get_bbox)\n\ndf_train = df_train.progress_apply(get_path, axis=1)\n\nkf = GroupKFold(n_splits = 5) \ndf_train = df_train.reset_index(drop=True)\ndf_train['fold'] = -1\nfor fold, (train_idx, val_idx) in enumerate(kf.split(df_train, y = df_train.video_id.tolist(), groups=df_train.sequence)):\n    df_train.loc[val_idx, 'fold'] = fold\n\ndf_train.head(5)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-03T01:02:09.380672Z","iopub.execute_input":"2022-02-03T01:02:09.382188Z","iopub.status.idle":"2022-02-03T01:02:25.677872Z","shell.execute_reply.started":"2022-02-03T01:02:09.382152Z","shell.execute_reply":"2022-02-03T01:02:25.677046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Select the dataset that your model hasn't seen!","metadata":{}},{"cell_type":"code","source":"df_test = df_train[df_train.fold == 4]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-03T01:02:25.679246Z","iopub.execute_input":"2022-02-03T01:02:25.679579Z","iopub.status.idle":"2022-02-03T01:02:25.688446Z","shell.execute_reply.started":"2022-02-03T01:02:25.679543Z","shell.execute_reply":"2022-02-03T01:02:25.687698Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Get image paths and ground truth BB's","metadata":{}},{"cell_type":"code","source":"# deepcopy is required to avoid contaminating input data\n\ndf_sample = df_test\nimage_paths = df_sample.image_path.tolist()\ngt = copy.deepcopy(df_sample.bboxes.tolist())\ngtmem = copy.deepcopy(df_sample.bboxes.tolist())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-03T01:02:25.689975Z","iopub.execute_input":"2022-02-03T01:02:25.690534Z","iopub.status.idle":"2022-02-03T01:02:25.739826Z","shell.execute_reply.started":"2022-02-03T01:02:25.690485Z","shell.execute_reply":"2022-02-03T01:02:25.739219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Test your model is working","metadata":{}},{"cell_type":"code","source":"def yolov5_infer(model, img, size, augment):\n    r = model(img, size=size, augment=augment)\n    \n    bboxes = []\n    bbclasses = []\n    scores = []\n    if r.pandas().xyxy[0].shape[0] > 0:\n        res = np.array(r.pandas().xyxy[0])\n        for r in res:\n            # Filter by INFER_CONF\n            if r[4] > INFER_CONF:\n                bb = r[0:4]\n                bb[2:4] -= bb[0:2] # Convert to xywh format\n                bboxes.append(bb.tolist()) # output bboxes as a nested list instead of numpy array\n                scores.append(r[4])\n                bbclasses.append(r[5])\n    \n    return bboxes, bbclasses, scores","metadata":{"execution":{"iopub.status.busy":"2022-02-03T01:02:25.741134Z","iopub.execute_input":"2022-02-03T01:02:25.741373Z","iopub.status.idle":"2022-02-03T01:02:25.748166Z","shell.execute_reply.started":"2022-02-03T01:02:25.741340Z","shell.execute_reply":"2022-02-03T01:02:25.747380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"i = 1400\nTEST_IMAGE_PATH = image_paths[i]\nimg = cv2.imread(TEST_IMAGE_PATH)\n\n# Insert your inference code here:\nbboxes, bbclasses, scores = yolov5_infer(model, img, INFER_SIZE, INFER_TTA)\n\n# Draw Green ground truth box\nout_image = draw_yolox_predictions(img, gt[i], [1.0] * len(gt[i]), [0] * len(gt[i]), COCO_CLASSES, (0,255,0))\n\n# Draw Red inference box\nout_image = draw_yolox_predictions(out_image, bboxes, scores, bbclasses, COCO_CLASSES, (0,0,255))\n\n# Convert BGR to RGB\nout_image = cv2.cvtColor(out_image, cv2.COLOR_BGR2RGB)\n\ndisplay(Image.fromarray(out_image))","metadata":{"execution":{"iopub.status.busy":"2022-02-03T01:04:05.823921Z","iopub.execute_input":"2022-02-03T01:04:05.824252Z","iopub.status.idle":"2022-02-03T01:04:06.303640Z","shell.execute_reply.started":"2022-02-03T01:04:05.824216Z","shell.execute_reply":"2022-02-03T01:04:06.302944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Assess Model Performance","metadata":{}},{"cell_type":"code","source":"def calc_pos_and_negs(bboxes, scores, gt_bboxes, IOU):\n    TP = []\n    FP = []\n    FN = 0\n    \n    gt0 = copy.deepcopy(gt_bboxes)\n    bb = copy.deepcopy(bboxes)\n    \n    if len(bboxes) == 0:\n        # all gt are false negative\n        FN += len(gt0)\n    else:\n        for idx, b in enumerate(bb):\n            b.append(scores[idx])\n        bb.sort(key = lambda x: x[4], reverse = True)\n\n        if len(gt0) == 0:\n            # all bboxes are false positives\n            for b in bb:\n                FP.append(b[4])\n        else:\n            # match bbox with gt\n            for b in bb:\n                matched = False\n                for g in gt0:\n                    # check whether gt box is already matched to an inference bb\n                    if len(g) == 4:\n                        # g bbox is unmatched\n                        if IOU_coco(b, g) >= IOU:\n                            g.append(b[4]) # assign confidence values to g; marks g as matched\n                            matched = True\n                            TP.append(b[4])\n                            break\n                if not matched:\n                    FP.append(b[4])\n            for g in gt0:\n                if len(g) == 4:\n                    FN += 1\n    return TP, FP, FN","metadata":{"execution":{"iopub.status.busy":"2022-02-03T01:04:17.554555Z","iopub.execute_input":"2022-02-03T01:04:17.554808Z","iopub.status.idle":"2022-02-03T01:04:17.564758Z","shell.execute_reply.started":"2022-02-03T01:04:17.554779Z","shell.execute_reply":"2022-02-03T01:04:17.563749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%cd /kaggle/working\n\n# Confidence scores of true positives, false positives and count false negatives\n\n# Yolov5\nTP_base = [[] for r in test_IOU] # Confidence scores of true positives\nFP_base = [[] for r in test_IOU] # Confidence scores of true positives\nFN_base = [0 for r in test_IOU]  # Count of false negative boxes\n\n# Norfair\nTP_nf = [[] for r in test_IOU] # Confidence scores of true positives\nFP_nf = [[] for r in test_IOU] # Confidence scores of true positives\nFN_nf = [0 for r in test_IOU]  # Count of false negative boxes\n\nfid = 0\n\nfor i in tqdm(range(len(image_paths))):\n# for i in tqdm(range(1250, 1450)):\n    TEST_IMAGE_PATH = image_paths[i]\n    img = cv2.imread(TEST_IMAGE_PATH)\n    \n    # Base yolov5\n    bboxes, bbclasses, scores = yolov5_infer(model, img, INFER_SIZE, INFER_TTA)\n\n    # Base norfair\n    bboxes_track, bbclasses_track, scores_track = norfair_track(tracker, fid, bboxes, bbclasses, scores)\n    \n    # Increment frame ID\n    fid += 1\n    \n    # yolov5 evaluation\n    for idx, IOU in enumerate(test_IOU):\n        TP, FP, FN = calc_pos_and_negs(bboxes, scores, gt[i], IOU)\n        if len(TP):\n            TP_base[idx].append(TP)\n        if len(FP):\n            FP_base[idx].append(FP)\n        FN_base[idx] += FN\n\n        # yolov5 norfair evaluation\n        TP, FP, FN = calc_pos_and_negs(bboxes_track, scores_track, gt[i], IOU)\n        if len(TP):\n            TP_nf[idx].append(TP)\n        if len(FP):\n            FP_nf[idx].append(FP)\n        FN_nf[idx] += FN","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-03T01:04:20.537928Z","iopub.execute_input":"2022-02-03T01:04:20.538471Z","iopub.status.idle":"2022-02-03T01:04:28.514529Z","shell.execute_reply.started":"2022-02-03T01:04:20.538434Z","shell.execute_reply":"2022-02-03T01:04:28.511828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Flatten nested lists\n\nfor idx, IOU in enumerate(test_IOU):\n    TP_base[idx] = [item for sublist in TP_base[idx] for item in sublist]\n    FP_base[idx] = [item for sublist in FP_base[idx] for item in sublist]\n    TP_nf[idx] = [item for sublist in TP_nf[idx] for item in sublist]\n    FP_nf[idx] = [item for sublist in FP_nf[idx] for item in sublist]","metadata":{"execution":{"iopub.status.busy":"2022-02-03T00:50:15.329689Z","iopub.execute_input":"2022-02-03T00:50:15.330013Z","iopub.status.idle":"2022-02-03T00:50:15.338321Z","shell.execute_reply.started":"2022-02-03T00:50:15.32997Z","shell.execute_reply":"2022-02-03T00:50:15.335599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for idx, IOU in enumerate(test_IOU):\n    print(IOU, len(TP_base[idx]), len(FP_base[idx]), FN_base[idx])","metadata":{"execution":{"iopub.status.busy":"2022-02-03T00:50:17.252976Z","iopub.execute_input":"2022-02-03T00:50:17.253762Z","iopub.status.idle":"2022-02-03T00:50:17.266181Z","shell.execute_reply.started":"2022-02-03T00:50:17.253722Z","shell.execute_reply":"2022-02-03T00:50:17.26545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Display your model's Metrics","metadata":{}},{"cell_type":"code","source":"# Which IOU to evaluate in histograms\neval_IOU_idx = np.where(np.isclose(test_IOU,eval_IOU))\neval_IOU_idx = eval_IOU_idx[0][0]","metadata":{"execution":{"iopub.status.busy":"2022-02-03T00:50:19.345536Z","iopub.execute_input":"2022-02-03T00:50:19.345794Z","iopub.status.idle":"2022-02-03T00:50:19.352067Z","shell.execute_reply.started":"2022-02-03T00:50:19.345763Z","shell.execute_reply":"2022-02-03T00:50:19.350509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Base YOLOV5","metadata":{}},{"cell_type":"code","source":"from matplotlib import pyplot as plt\n%matplotlib inline","metadata":{"execution":{"iopub.status.busy":"2022-02-03T00:50:21.319422Z","iopub.execute_input":"2022-02-03T00:50:21.319701Z","iopub.status.idle":"2022-02-03T00:50:21.325569Z","shell.execute_reply.started":"2022-02-03T00:50:21.319671Z","shell.execute_reply":"2022-02-03T00:50:21.324672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist(TP_base[eval_IOU_idx], 100)\nplt.title(f\"CONF of true positives, base YOLOV5 @ IOU {eval_IOU}\")\nplt.xlabel('CONF')\nplt.ylabel('TP count')\nplt.show()\n\nprint(f'True positives = {len(TP_base[eval_IOU_idx])}')\nprint(f'False negatives = {FN_base[eval_IOU_idx]}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-03T00:50:22.790853Z","iopub.execute_input":"2022-02-03T00:50:22.791136Z","iopub.status.idle":"2022-02-03T00:50:23.14197Z","shell.execute_reply.started":"2022-02-03T00:50:22.791103Z","shell.execute_reply":"2022-02-03T00:50:23.141236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist(FP_base[eval_IOU_idx], 100)\nplt.title(f\"CONF of false positives, base YOLOV5 @ IOU {eval_IOU}\")\nplt.xlabel('CONF')\nplt.ylabel('FP count')\nplt.show()\n\nprint(f'False positives = {len(FP_base[eval_IOU_idx])}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-03T00:50:25.437203Z","iopub.execute_input":"2022-02-03T00:50:25.437473Z","iopub.status.idle":"2022-02-03T00:50:25.776704Z","shell.execute_reply.started":"2022-02-03T00:50:25.437442Z","shell.execute_reply":"2022-02-03T00:50:25.775999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from statistics import mean\n\nF2list = []\nF2max = 0.0\nF2maxat = -1.0\nF2mean = 0.0\n\nfor c in np.arange(0.0, 1.0, 0.01):\n    F2temp = []\n    \n    for idx, IOU in enumerate(test_IOU):\n        FNcount = FN_base[idx] + sum(1 for i in TP_base[idx] if i < c)\n        TPcount = sum(1 for i in TP_base[idx] if i >= c)\n        FPcount = sum(1 for i in FP_base[idx] if i >= c)\n        R = TPcount / (TPcount + FNcount + 0.0001)\n        P = TPcount / (TPcount + FPcount + 0.0001)\n        F2 = (5 * P * R) / (4 * P + R + 0.0001)\n        F2temp.append(F2)\n    \n    F2mean = mean(F2temp)\n    F2list.append((c, F2mean))\n    if F2max < F2mean:\n        F2max = F2mean\n        F2maxat = c\n\nplt.scatter(*zip(*F2list))\nplt.title(\"CONF vs F2 score: Base YOLOV5\")\nplt.xlabel('CONF')\nplt.ylabel('F2')\nplt.show()\n\nprint(f'F2 max is {F2max} at CONF = {F2maxat}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-03T00:50:28.096881Z","iopub.execute_input":"2022-02-03T00:50:28.097465Z","iopub.status.idle":"2022-02-03T00:50:28.342139Z","shell.execute_reply.started":"2022-02-03T00:50:28.097423Z","shell.execute_reply":"2022-02-03T00:50:28.341081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### YOLOV5 Norfair","metadata":{}},{"cell_type":"code","source":"plt.hist(TP_nf[eval_IOU_idx], 100)\nplt.title(f\"CONF of true positives, YOLOV5 Norfair @ IOU {eval_IOU}\")\nplt.xlabel('CONF')\nplt.ylabel('TP count')\nplt.show()\n\nprint(f'True positives = {len(TP_nf[eval_IOU_idx])}')\nprint(f'False negatives = {FN_nf[eval_IOU_idx]}')","metadata":{"execution":{"iopub.status.busy":"2022-02-03T00:50:31.886522Z","iopub.execute_input":"2022-02-03T00:50:31.887288Z","iopub.status.idle":"2022-02-03T00:50:32.287056Z","shell.execute_reply.started":"2022-02-03T00:50:31.887246Z","shell.execute_reply":"2022-02-03T00:50:32.286337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist(FP_nf[eval_IOU_idx], 100)\nplt.title(f\"CONF of false positives, YOLOV5 Norfair @ IOU {eval_IOU}\")\nplt.xlabel('CONF')\nplt.ylabel('FP count')\nplt.show()\n\nprint(f'False positives = {len(FP_nf[eval_IOU_idx])}')","metadata":{"execution":{"iopub.status.busy":"2022-02-03T00:50:34.350249Z","iopub.execute_input":"2022-02-03T00:50:34.350752Z","iopub.status.idle":"2022-02-03T00:50:34.696808Z","shell.execute_reply.started":"2022-02-03T00:50:34.350711Z","shell.execute_reply":"2022-02-03T00:50:34.696079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"F2list = []\nF2max = 0.0\nF2maxat = -1.0\nF2mean = 0.0\n\nfor c in np.arange(0.0, 1.0, 0.01):\n    F2temp = []\n    \n    for idx, IOU in enumerate(test_IOU):\n        FNcount = FN_nf[idx] + sum(1 for i in TP_nf[idx] if i < c)\n        TPcount = sum(1 for i in TP_nf[idx] if i >= c)\n        FPcount = sum(1 for i in FP_nf[idx] if i >= c)\n        R = TPcount / (TPcount + FNcount + 0.0001)\n        P = TPcount / (TPcount + FPcount + 0.0001)\n        F2 = (5 * P * R) / (4 * P + R + 0.0001)\n        F2temp.append(F2)\n    \n    F2mean = mean(F2temp)\n    F2list.append((c, F2mean))\n    if F2max < F2mean:\n        F2max = F2mean\n        F2maxat = c\n\nplt.scatter(*zip(*F2list))\nplt.title(\"CONF vs F2 score: YOLOV5 Norfair\")\nplt.xlabel('CONF')\nplt.ylabel('F2')\nplt.show()\n\nprint(f'F2 max is {F2max} at CONF = {F2maxat}')","metadata":{"execution":{"iopub.status.busy":"2022-02-03T00:50:36.73553Z","iopub.execute_input":"2022-02-03T00:50:36.735829Z","iopub.status.idle":"2022-02-03T00:50:36.981033Z","shell.execute_reply.started":"2022-02-03T00:50:36.735781Z","shell.execute_reply":"2022-02-03T00:50:36.980282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Cleanup","metadata":{}},{"cell_type":"code","source":"!rm -rf /kaggle/working/*","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Credits","metadata":{}},{"cell_type":"markdown","source":"* https://www.kaggle.com/remekkinas/yolox-training-pipeline-cots-dataset-lb-0-507\n* https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507","metadata":{}}]}