{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# COTS","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"Inpiration from AWSAF, a great work introducing yolov5\nwebsite: https://www.kaggle.com/code/awsaf49/great-barrier-reef-yolov5-train/notebook","metadata":{}},{"cell_type":"markdown","source":"# Environment\n**It is recommended to run in Kaggle**\n\n**with best.pt as import file**\n\ntensorflow-great-barrier-reef data can be added easily in kaggle","metadata":{}},{"cell_type":"markdown","source":"# Run training\n**run all before \"prepare for inference\"**\n\n# Run inference\n**run library, run install yolov5 then run all after including\n\"prepare for inference\"**","metadata":{}},{"cell_type":"markdown","source":"# Download YOLOv5","metadata":{}},{"cell_type":"markdown","source":"**YOLO**(YOU ONLY LOOK ONCE) is quick and simple for object detection, it is also a one stage detector.\n\nFrom the high level object detection architecture, yolov5 comprises of input, backbone, neck and head which is dense prediction for yolo.\n\nIn yolov5, it uses CSPDarknet53 for its backbone and SPP block to seperate the critical features without sacrificing the network computational speed. PANet is utilised for its neck as parameter aggregation from various backbone levels. It uses YOLOv3 for its head.\n\nSelf-Adversarial Training (SAT) ,Mosaic data augmentation was introduced in yolov5, it mixes four training images. \n\nFurthermore, existing modules such as Spatial Attention Module (SAM), PAN and CBN have been adjusted to improve the overall performance of these modules.","metadata":{}},{"cell_type":"markdown","source":"# Intall yolov5","metadata":{}},{"cell_type":"code","source":"!git clone https://github.com/ultralytics/yolov5\n%cd yolov5\n!pip install -qr requirements.txt","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:42:57.863382Z","iopub.execute_input":"2022-11-16T06:42:57.864608Z","iopub.status.idle":"2022-11-16T06:43:14.964381Z","shell.execute_reply.started":"2022-11-16T06:42:57.864466Z","shell.execute_reply":"2022-11-16T06:43:14.963067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport sys\nimport cv2\nimport shutil\nimport wandb\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport pandas as pd\nimport torch\nimport ast\nimport albumentations as albu\nimport yaml\nimport copy\nfrom IPython.display import display\nfrom tqdm.notebook import tqdm\ntqdm = tqdm.pandas()\nfrom PIL import Image\n\nfrom sklearn.model_selection import GroupKFold\nfrom yolov5 import utils\ndisplay = utils.notebook_init()\n","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:43:14.967953Z","iopub.execute_input":"2022-11-16T06:43:14.968346Z","iopub.status.idle":"2022-11-16T06:43:19.522075Z","shell.execute_reply.started":"2022-11-16T06:43:14.968312Z","shell.execute_reply":"2022-11-16T06:43:19.520901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Weights and Biases\n**no W&B account skip this step**\n\nW&B is a powerful tool to retrieve the results such as F1 score, confusion matrix, P and R curves, validation images sample. It can track and compare each training.","metadata":{}},{"cell_type":"code","source":"# from kaggle_secrets import UserSecretsClient\n# user_secrets = UserSecretsClient()\n# secret_value_0 = user_secrets.get_secret(\"user_secret\")\n\n# ! wandb login $secret_value_0","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:43:19.523975Z","iopub.execute_input":"2022-11-16T06:43:19.525356Z","iopub.status.idle":"2022-11-16T06:43:19.531909Z","shell.execute_reply.started":"2022-11-16T06:43:19.525313Z","shell.execute_reply":"2022-11-16T06:43:19.529344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Preprocessing","metadata":{}},{"cell_type":"code","source":"# create directories\n!mkdir -p '/kaggle/working/images' # to store training images\n!mkdir -p '/kaggle/working/labels' # to store labels of images\n\n# for paths\nimg_path = '/kaggle/working/images'\nlabel_path = '/kaggle/working/labels'\ninput_path = '/kaggle/input/tensorflow-great-barrier-reef/train_images'\n\n# check\nprint('path:', os.listdir('/kaggle/working/'))","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:43:19.535267Z","iopub.execute_input":"2022-11-16T06:43:19.536306Z","iopub.status.idle":"2022-11-16T06:43:21.405855Z","shell.execute_reply.started":"2022-11-16T06:43:19.536269Z","shell.execute_reply":"2022-11-16T06:43:21.404755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import data\ndata = pd.read_csv('/kaggle/input/tensorflow-great-barrier-reef/train.csv')\n\n# get number of images\n# data['annotations'] = data['annotations'].progress_apply(eval)\n# data.sample(3)\ndata['bbox_nums'] = data['annotations'].progress_apply(lambda x: str.count(x, 'x'))\n\n# get number of images with box\ndata = data[data['bbox_nums']>0]\n# data.sample(2)\nprint('num of images with bbox:', (data['bbox_nums']>0).count())\n\n# get train data by removing images with no bbox\ntrain_data = data[data['bbox_nums']>0].reset_index(drop=True)\n\n# get path of input, new image path and new label path\ntrain_data['input_path'] = input_path + '/video_' + \\\n                            train_data['video_id'].astype(str) + \\\n                            '/' + train_data['video_frame'].astype(str) + \\\n                            '.jpg'\ntrain_data['image_path'] = '/kaggle/working/images/video_' + \\\n                            train_data['video_id'].astype(str) + \\\n                            '_' + train_data['video_frame'].astype(str) + \\\n                            '.jpg'\ntrain_data['label_path'] = '/kaggle/working/labels/video_' + \\\n                            train_data['video_id'].astype(str) + \\\n                            '_' + train_data['video_frame'].astype(str) + \\\n                            '.txt'\n\n# all images has same dim of 1280x720 (wid, height)\ntrain_data['width'] = 1280\ntrain_data['height'] = 720\n\n# get coco bbox\n# coco format: [x, y, wid, height]\n# to get later yolo format: [x_center, y_c, wid, height]\ntrain_data['bbox'] = train_data['annotations'].apply(lambda x: [list(annot.values()) \\\n                                                    for annot in eval(x)])\ntrain_data.sample(2)\n","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:43:21.407727Z","iopub.execute_input":"2022-11-16T06:43:21.408059Z","iopub.status.idle":"2022-11-16T06:43:21.712270Z","shell.execute_reply.started":"2022-11-16T06:43:21.408027Z","shell.execute_reply":"2022-11-16T06:43:21.711173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Write images and labels","metadata":{}},{"cell_type":"markdown","source":"**Write images in output files**\n\nImages are copied from ../input/tensorflow-great-barrier-reef/train_images to newly created images file.\n\nutilization of shutil library to copy","metadata":{}},{"cell_type":"code","source":"# store images in new folder (image file created above)\nfor i in train_data['input_path'].tolist():\n    dash = i.split('/')\n    \n    # last and 2nd last from path is video frame and id\n    video_id = dash[-2]\n    video_frame = dash[-1]\n    \n    # copy images from input path to new folders\n    image_path = f'/kaggle/working/images/{video_id}_{video_frame}'\n    shutil.copy(src=i, dst=image_path)\nprint('success')","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:43:21.714065Z","iopub.execute_input":"2022-11-16T06:43:21.714436Z","iopub.status.idle":"2022-11-16T06:44:17.190296Z","shell.execute_reply.started":"2022-11-16T06:43:21.714401Z","shell.execute_reply":"2022-11-16T06:44:17.189272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# test\nprint('get few images from images folder：', \n     os.listdir(img_path)[6])\n\n# plt.figure(1)\nimg = Image.open('/kaggle/working/images/video_1_8822.jpg')\n# image = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY )\nplt.figure(figsize=(9,9))\nplt.imshow(img)\nplt.axis('off')\nprint(img.size)","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:44:17.191639Z","iopub.execute_input":"2022-11-16T06:44:17.193351Z","iopub.status.idle":"2022-11-16T06:44:17.703526Z","shell.execute_reply.started":"2022-11-16T06:44:17.193314Z","shell.execute_reply":"2022-11-16T06:44:17.699845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Coco2Yolo labels\nIt is neccessary to convert the coco format to yolo. \n\nfrom coco format: [x, y, wid, height]\nto yolo format: [x_center, y_c, wid, height]","metadata":{}},{"cell_type":"code","source":"# normalize and then convert coco2yolo\n# information and formula obtained from:\n# https://prabhjotkaurgosal.com/weekly-learnings/weekly-learning-blogs/\n\n# bbox is the coco box\n# image width and height to process\ndef coco2yolo(bbox, img_wid, img_height):\n    \n    # convert to float type\n    bbox = np.array(bbox).astype(float)\n    \n    # normalize the dimension\n    # start from the width (x and wid)\n    bbox[:, [0,2]] = bbox[:, [0,2]] / img_wid\n    # then deal with height (y and height)\n    bbox[:, [1,3]] = bbox[:, [1,3]] / img_height\n    \n    # convert the x,y to x_center, y_center\n    # as written in the website the formula for conversion:\n    bbox[:, [0,1]] = bbox[:, [0,1]] + bbox[:, [2,3]] / 2\n    \n    # keep the values between 0-1\n    bbox = np.clip(bbox, a_min=0, a_max=1)\n    \n    return bbox\n    ","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:44:17.705143Z","iopub.execute_input":"2022-11-16T06:44:17.705481Z","iopub.status.idle":"2022-11-16T06:44:17.713656Z","shell.execute_reply.started":"2022-11-16T06:44:17.705448Z","shell.execute_reply":"2022-11-16T06:44:17.712694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# test bbox conversion\nbox_test = [[576,407,75,72],[872,531,27,36]]\n\nprint('coco2yolo:', coco2yolo(img_wid=1280, \n                             img_height=720,\n                             bbox=box_test))","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:44:17.715326Z","iopub.execute_input":"2022-11-16T06:44:17.716031Z","iopub.status.idle":"2022-11-16T06:44:17.727078Z","shell.execute_reply.started":"2022-11-16T06:44:17.715996Z","shell.execute_reply":"2022-11-16T06:44:17.726133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Write labels in output files**\nSame as images written above,\ncopy labels  from ../input/tensorflow-great-barrier-reef/train_images to newly created labels file.\n\n","metadata":{}},{"cell_type":"code","source":"# store labels in new folder\nlabel_bbox = []\nwidth = 1280\nheight = 720\n\n# .iloc function to get row or column data\nfor j in range(train_data.shape[0]):\n    row_data = train_data.iloc[j]\n    img_wid = row_data.width\n    img_height = row_data.height\n    bbox = np.array(row_data.bbox).astype(np.float32).copy()\n    num_bbox = row_data.bbox_nums\n    \n    # form yolo annotation by creating file\n    # and write labels\n    with open(row_data.label_path, 'w') as file:\n        if num_bbox < 1:\n            file.write('')\n            continue\n            \n        # convert all the labels from coco2yolo\n        yolo_box = coco2yolo(bbox, width, height)\n        label_bbox.append(yolo_box)\n        \n        # write labels in new file\n        for k in range(num_bbox):\n            \n            label = ['0'] + \\\n                    yolo_box[k].astype(str).tolist() + \\\n                    ([''] if k+1 == num_bbox else ['\\n'])\n            label = ' '.join(label).strip('')\n            file.write(label)\nprint('success')           ","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:44:17.731743Z","iopub.execute_input":"2022-11-16T06:44:17.732071Z","iopub.status.idle":"2022-11-16T06:44:20.036422Z","shell.execute_reply.started":"2022-11-16T06:44:17.732022Z","shell.execute_reply":"2022-11-16T06:44:20.035450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# test\nprint('check labels folder:', os.listdir('/kaggle/working/labels')[:2],'\\n')\n\ntxt1 = open('/kaggle/working/labels/video_1_5714.txt', 'r')\ntxt2 = open('/kaggle/working/labels/video_0_9516.txt', 'r')\n\nprint('txt1:\\n' + txt1.read() + '\\n')\nprint('txt2:\\n' + txt2.read() + '\\n')\n","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:44:20.040526Z","iopub.execute_input":"2022-11-16T06:44:20.042661Z","iopub.status.idle":"2022-11-16T06:44:20.057193Z","shell.execute_reply.started":"2022-11-16T06:44:20.042624Z","shell.execute_reply":"2022-11-16T06:44:20.056290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# YOLOv5","metadata":{}},{"cell_type":"markdown","source":"**Splitting data**\nData is split into 2 parts: training and testing sections.\n\n！！！can edit how to split\n\n    data has 3 video: video_0, video_1, video_2\n    \n*     train: images from video_0, video_1 (2099+677)\n*     validate: images from video_2 (2143)\n","metadata":{}},{"cell_type":"code","source":"# split images for training and validating\ntrain = train_data[train_data['video_id'].isin([1,2])]\nvalidate = train_data[train_data['video_id'].isin([0])]\n\n# get images and labels\n# train\ntrain_img = list(train['image_path'])\ntrain_lab = list(train['label_path'])\n\n# validate/test\nvalidate_img = list(validate['image_path'])\nvalidate_lab = list(validate['label_path'])\n\n# test the items in each data sets\nprint('train:', len(train), '\\n'\n     'validate:', len(validate), '\\n')","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:44:20.061242Z","iopub.execute_input":"2022-11-16T06:44:20.061999Z","iopub.status.idle":"2022-11-16T06:44:20.080466Z","shell.execute_reply.started":"2022-11-16T06:44:20.061966Z","shell.execute_reply":"2022-11-16T06:44:20.079435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Create train and validation directories to store train and validation data**","metadata":{}},{"cell_type":"code","source":"# train path\nwith open('/kaggle/working/train_img.txt', 'w') as file:\n    for item in train_img:\n        file.write(item + '\\n')\n\n# validate path\nwith open('/kaggle/working/valid_img.txt', 'w') as file:\n    for item in validate_img:\n        file.write(item + '\\n')","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:44:20.084342Z","iopub.execute_input":"2022-11-16T06:44:20.086472Z","iopub.status.idle":"2022-11-16T06:44:20.096712Z","shell.execute_reply.started":"2022-11-16T06:44:20.086437Z","shell.execute_reply":"2022-11-16T06:44:20.095667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Configure yaml file for training**","metadata":{}},{"cell_type":"code","source":"data = {'path': '/kaggle/working',\n        'train': '/kaggle/working/train_img.txt',\n        'val': '/kaggle/working/valid_img.txt',\n        'nc': 1,\n        'names': ['cots']}\n\nwith open(\"/kaggle/working/data.yaml\", 'w') as file:\n    yaml.dump(data, file, default_flow_style=False)\n\n# %cat /kaggle/working/yolov5/data/data.yaml\ny = open(os.path.join('/kaggle/working', 'data.yaml'), 'r')\nprint('yaml:', y.read())","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:44:20.101619Z","iopub.execute_input":"2022-11-16T06:44:20.104158Z","iopub.status.idle":"2022-11-16T06:44:20.115890Z","shell.execute_reply.started":"2022-11-16T06:44:20.104124Z","shell.execute_reply":"2022-11-16T06:44:20.115015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Set parameters**","metadata":{}},{"cell_type":"code","source":"Epochs = 1 # should be 12 or above\nBatch_size = 4\nImg_size = 1280","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:44:20.120313Z","iopub.execute_input":"2022-11-16T06:44:20.122429Z","iopub.status.idle":"2022-11-16T06:44:20.128310Z","shell.execute_reply.started":"2022-11-16T06:44:20.122395Z","shell.execute_reply":"2022-11-16T06:44:20.127430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Change some hyperparameters**\nchange some of hyperparameters in default\nInspiration from：\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/307760\n* mosaic prob = 0.25\n* mixup prob = 0.25\n* rotation = +/-90deg\n\nHowever, default parameters are more suitable and stable to run.","metadata":{}},{"cell_type":"code","source":"%%writefile /kaggle/working/hyper.yaml\nlr0: 0.01  # initial learning rate (SGD=1E-2, Adam=1E-3)\nlrf: 0.1  # final OneCycleLR learning rate (lr0 * lrf)\nmomentum: 0.937  # SGD momentum/Adam beta1\nweight_decay: 0.0005  # optimizer weight decay 5e-4\nwarmup_epochs: 3.0  # warmup epochs \nwarmup_momentum: 0.8  # warmup initial momentum\nwarmup_bias_lr: 0.1  # warmup initial bias lr\nbox: 0.05  \ncls: 0.5 \ncls_pw: 1.0\nobj: 1.0  \nobj_pw: 1.0  \niou_t: 0.20  \nanchor_t: 4.0  # anchor-multiple threshold\n# image adjustment\nfl_gamma: 0.0\nhsv_h: 0.015\nhsv_s: 0.7\nhsv_v: 0.4\ndegrees: 90.0  # image rotation\ntranslate: 0.10\nscale: 0.5\nshear: 0.0\nperspective: 0.0\nflipud: 0.5\nfliplr: 0.5\nmosaic: 0.25  # image mosaic (probability)\nmixup: 0.25 # image mixup (probability)\ncopy_paste: 0.0","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:44:20.131574Z","iopub.execute_input":"2022-11-16T06:44:20.132950Z","iopub.status.idle":"2022-11-16T06:44:20.145631Z","shell.execute_reply.started":"2022-11-16T06:44:20.132916Z","shell.execute_reply":"2022-11-16T06:44:20.144690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!python /kaggle/working/yolov5/train.py --img {Img_size}\\\n                 --batch {Batch_size}\\\n                 --optimizer 'Adam'\\\n                 --epochs {Epochs}\\\n                 --data /kaggle/working/data.yaml\\\n#                  --hyp /kaggle/working/hyper.yaml\\\n                 --weights yolov5s6.pt\n                 --exist-ok","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:44:20.148849Z","iopub.execute_input":"2022-11-16T06:44:20.149947Z","iopub.status.idle":"2022-11-16T06:53:43.032247Z","shell.execute_reply.started":"2022-11-16T06:44:20.149914Z","shell.execute_reply":"2022-11-16T06:53:43.031014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare for inference","metadata":{}},{"cell_type":"code","source":"# split the into 5 folders for later inference\ninput_path = '/kaggle/input/tensorflow-great-barrier-reef/'\n\ndef get_bbox(annots):\n    bboxes = [list(annot.values()) for annot in annots]\n    return bboxes\n\ndef get_path(row):\n    row['image_path'] = f'{input_path}/train_images/video_{row.video_id}/{row.video_frame}.jpg'\n    return row\n\n\n# import data\ninfer = pd.read_csv('/kaggle/input/tensorflow-great-barrier-reef/train.csv')\n\n# get number of images\n# data['annotations'] = data['annotations'].progress_apply(eval)\n# data.sample(3)\ninfer['bbox_nums'] = infer['annotations'].progress_apply(lambda x: str.count(x, 'x'))\n\n# Don't filter for annotated frames. Include frames with no bboxes as well!\ninfer_train = infer\n\n# Annotations \ninfer_train['annotations'] = infer_train['annotations'].progress_apply(lambda x: ast.literal_eval(x))\ninfer_train['bboxes'] = infer_train.annotations.progress_apply(get_bbox)\n\ninfer_train = infer_train.progress_apply(get_path, axis=1)\n\ngkf = GroupKFold(n_splits = 5) \ninfer_train = infer_train.reset_index(drop=True)\ninfer_train['folder'] = -1\nfor folder, (train_index, val_index) in enumerate(gkf.split(infer_train, y = infer_train.video_id.tolist(), groups=infer_train.sequence)):\n    infer_train.loc[val_index, 'folder'] = folder\n\ninfer_train.head(3)","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:53:43.035179Z","iopub.execute_input":"2022-11-16T06:53:43.036406Z","iopub.status.idle":"2022-11-16T06:53:59.136355Z","shell.execute_reply.started":"2022-11-16T06:53:43.036356Z","shell.execute_reply":"2022-11-16T06:53:59.135266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**bbox convert**","metadata":{}},{"cell_type":"code","source":"# Modified from https://www.kaggle.com/awsaf49/great-barrier-reef-yolov5-infer\n\n#     confidence scores\ndef voc2yolo(bboxes, image_height=720, image_width=1280):\n    \"\"\"\n    voc  => [x1, y1, x2, y1]\n    yolo => [xmid, ymid, w, h] (normalized)\n    \"\"\"\n    \n    bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n    \n    bboxes[..., [0, 2]] = bboxes[..., [0, 2]]/ image_width\n    bboxes[..., [1, 3]] = bboxes[..., [1, 3]]/ image_height\n    \n    width = bboxes[..., 2] - bboxes[..., 0]\n    height = bboxes[..., 3] - bboxes[..., 1]\n    \n    bboxes[..., 0] = bboxes[..., 0] + width/2\n    bboxes[..., 1] = bboxes[..., 1] + height/2\n    bboxes[..., 2] = width\n    bboxes[..., 3] = height\n    \n    return bboxes\n\n\n\ndef yolo2coco(bboxes, image_height=720, image_width=1280):\n    \"\"\"\n    yolo => [xmid, ymid, w, h] (normalized)\n    coco => [xmin, ymin, w, h]\n    \n    \"\"\" \n    bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n    \n    # denormalizing\n    bboxes[..., [0, 2]]= bboxes[..., [0, 2]]* image_width\n    bboxes[..., [1, 3]]= bboxes[..., [1, 3]]* image_height\n    \n    # converstion (xmid, ymid) => (xmin, ymin) \n    bboxes[..., [0, 1]] = bboxes[..., [0, 1]] - bboxes[..., [2, 3]]/2\n    \n    return bboxes\n\ndef voc2coco(bboxes, image_height=720, image_width=1280):\n    bboxes  = voc2yolo(bboxes, image_height, image_width)\n    bboxes  = yolo2coco(bboxes, image_height, image_width)\n    return bboxes\n\n\n\ndef plot_one_box(x, img, score=None, color=None, label=None, line_thickness=None):\n    # Plots one bounding box on image img\n    tl = line_thickness or round(0.002 * (img.shape[0] + img.shape[1]) / 2) + 1  # line/font thickness\n    color = color or [random.randint(0, 255) for _ in range(3)]\n    c1, c2 = (int(x[0]), int(x[1])), (int(x[2]), int(x[3]))\n    cv2.rectangle(img, c1, c2, color, thickness=tl, lineType=cv2.LINE_AA)\n    if label:\n        tf = max(tl - 1, 1)  # font thickness\n        t_size = cv2.getTextSize(label, 0, fontScale=tl / 3, thickness=tf)[0]\n        c2 = c1[0] + t_size[0], c1[1] - t_size[1] - 3\n        cv2.rectangle(img, c1, c2, color, -1, cv2.LINE_AA)  # filled\n        cv2.putText(img, \"{}:{:.2f}\".format(label, score), (c1[0], c1[1] - 2), 0, tl / 3, [225, 255, 255], thickness=tf, lineType=cv2.LINE_AA)\n\ndef draw_bboxes(img, bboxes, scores, classes, class_ids, colors = None, show_classes = None, bbox_format = 'yolo', class_name = False, line_thickness = 2):  \n     \n    image = img.copy()\n    show_classes = classes if show_classes is None else show_classes\n    colors = (0, 255 ,0) if colors is None else colors\n    \n    if bbox_format == 'yolo':\n        \n        for idx in range(len(bboxes)):  \n            \n            bbox  = bboxes[idx]\n            cls   = classes[idx]\n            cls_id = class_ids[idx]\n            try:\n                score = scores[idx]\n            except:\n                score = None\n            color = colors[cls_id] if type(colors) is list else colors\n            \n            if cls in show_classes:\n            \n                x1 = round(float(bbox[0])*image.shape[1])\n                y1 = round(float(bbox[1])*image.shape[0])\n                w  = round(float(bbox[2])*image.shape[1]/2) #w/2 \n                h  = round(float(bbox[3])*image.shape[0]/2)\n\n                voc_bbox = (x1-w, y1-h, x1+w, y1+h)\n                plot_one_box(voc_bbox, \n                             image,\n                             score= score if score else None,\n                             color = color,\n                             label = cls if class_name else str(get_label(cls)),\n                             line_thickness = line_thickness)\n            \n    elif bbox_format == 'coco':\n        \n        for idx in range(len(bboxes)):  \n            \n            bbox  = bboxes[idx]\n            cls   = classes[idx]\n            cls_id = class_ids[idx]\n            try:\n                score = scores[idx]\n            except:\n                score = None\n            color = colors[cls_id] if type(colors) is list else colors\n            \n            if cls in show_classes:            \n                x1 = int(round(bbox[0]))\n                y1 = int(round(bbox[1]))\n                w  = int(round(bbox[2]))\n                h  = int(round(bbox[3]))\n\n                voc_bbox = (x1, y1, x1+w, y1+h)\n                plot_one_box(voc_bbox, \n                             image,\n                             score = score if score else None,\n                             color = color,\n                             label = cls if class_name else str(cls_id),\n                             line_thickness = line_thickness)\n                \n    elif bbox_format == 'voc_pascal':\n        \n        for idx in range(len(bboxes)):  \n            \n            bbox  = bboxes[idx]\n            cls   = classes[idx]\n            cls_id = class_ids[idx]\n            try:\n                score = scores[idx]\n            except:\n                score = None\n            color = colors[cls_id] if type(colors) is list else colors\n            \n            if cls in show_classes: \n                x1 = int(round(bbox[0]))\n                y1 = int(round(bbox[1]))\n                x2 = int(round(bbox[2]))\n                y2 = int(round(bbox[3]))\n                voc_bbox = (x1, y1, x2, y2)\n                plot_one_box(voc_bbox, \n                             image,\n                             score = score if score else None,\n                             color = color,\n                             label = cls if class_name else str(cls_id),\n                             line_thickness = line_thickness)\n    else:\n        raise ValueError('wrong bbox format')\n\n    return image\n\n\ndef get_imgsize(row):\n    row['width'], row['height'] = imagesize.get(row['image_path'])\n    return row\n\nnp.random.seed(32)\ncolors = [(np.random.randint(255), np.random.randint(255), np.random.randint(255))\\\n          for idx in range(1)]","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:53:59.138228Z","iopub.execute_input":"2022-11-16T06:53:59.138907Z","iopub.status.idle":"2022-11-16T06:53:59.168144Z","shell.execute_reply.started":"2022-11-16T06:53:59.138871Z","shell.execute_reply":"2022-11-16T06:53:59.167169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**prediction**","metadata":{}},{"cell_type":"code","source":"# Modified from https://www.kaggle.com/awsaf49/great-barrier-reef-yolov5-infer\n\ndef predict(model, img, size=500, augment=False):\n    height, width = img.shape[:2]\n    results = model(img, size=size, augment=augment)  # custom inference size\n    preds   = results.pandas().xyxy[0]\n    bboxes  = preds[['xmin','ymin','xmax','ymax']].values\n    if len(bboxes):\n        bboxes  = voc2coco(bboxes,height,width).astype(int)\n        confs   = preds.confidence.values\n        return bboxes, confs\n    else:\n        return [],[]\n    \ndef format_prediction(bboxes, confs):\n    annot = ''\n    if len(bboxes)>0:\n        for idx in range(len(bboxes)):\n            xmin, ymin, w, h = bboxes[idx]\n            conf             = confs[idx]\n            annot += f'{conf} {xmin} {ymin} {w} {h}'\n            annot +=' '\n        annot = annot.strip(' ')\n    return annot\n\ndef show_img(img, bboxes, confis, bbox_format='yolo', colors=colors):\n    names  = ['starfish']*len(bboxes)\n    labels = [0]*len(bboxes)\n    img    = draw_bboxes(img = img,\n                           bboxes = bboxes,\n                           scores = confis,\n                           classes = names,\n                           class_ids = labels,\n                           class_name = True, \n                           colors = colors, \n                           bbox_format = bbox_format,\n                           line_thickness = 2)\n    return Image.fromarray(img)","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:53:59.169877Z","iopub.execute_input":"2022-11-16T06:53:59.170279Z","iopub.status.idle":"2022-11-16T06:53:59.183349Z","shell.execute_reply.started":"2022-11-16T06:53:59.170203Z","shell.execute_reply":"2022-11-16T06:53:59.182381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**IOU**","metadata":{}},{"cell_type":"code","source":"def IOU_coco(bbox1, bbox2):\n    '''\n        adapted from https://stackoverflow.com/questions/25349178/calculating-percentage-of-bounding-box-overlap-for-image-detector-evaluation\n    '''\n    x_left = max(bbox1[0], bbox2[0])\n    y_top = max(bbox1[1], bbox2[1])\n    x_right = min(bbox1[0] + bbox1[2], bbox2[0] + bbox2[2])\n    y_bottom = min(bbox1[1] + bbox1[3], bbox2[1] + bbox2[3])\n    if x_right < x_left or y_bottom < y_top:\n        return 0.0\n    intersection_area = (x_right - x_left) * (y_bottom - y_top)\n    bb1_area = bbox1[2] * bbox1[3]\n    bb2_area = bbox2[2] * bbox2[3]\n    iou = intersection_area / float(bb1_area + bb2_area - intersection_area)\n\n    assert iou >= 0.0\n    assert iou <= 1.0\n    return iou","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:53:59.186333Z","iopub.execute_input":"2022-11-16T06:53:59.186635Z","iopub.status.idle":"2022-11-16T06:53:59.198038Z","shell.execute_reply.started":"2022-11-16T06:53:59.186609Z","shell.execute_reply":"2022-11-16T06:53:59.197158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_model(model_path, conf=0.01, iou=0.50):\n    model = torch.hub.load('/kaggle/working/yolov5',\n                           'custom',\n                           path=model_path,\n                           source='local',\n                           force_reload=True)  # local repo\n    model.conf = conf  # NMS confidence threshold\n    model.iou  = iou  # NMS IoU threshold\n    model.classes = None   # (optional list) filter by class, i.e. = [0, 15, 16] for persons, cats and dogs\n    model.multi_label = False  # NMS multiple labels per box\n    model.max_det = 1000  # maximum number of detections per image\n    return model","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:53:59.201175Z","iopub.execute_input":"2022-11-16T06:53:59.201497Z","iopub.status.idle":"2022-11-16T06:53:59.210413Z","shell.execute_reply.started":"2022-11-16T06:53:59.201471Z","shell.execute_reply":"2022-11-16T06:53:59.209446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Modified from https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507\n# Additions: \n#     auto converts to xywh format\n#     converts tensors to list of floats\n# Updates:\n# i changed it to yolov5 version. code is still dirty\ndef yolov5_inference(img, model, test_size, conf_threshold = 0.4):\n    bboxes = []\n    bbclasses = []\n    scores = []\n    \n    preproc = ValTransform(legacy = False)\n    \n    tensor_img, _ = preproc(img, None, test_size)\n    tensor_img = torch.from_numpy(tensor_img).unsqueeze(0)\n    tensor_img = tensor_img.float()\n    tensor_img = tensor_img.cuda()\n    \n    with torch.no_grad():\n        outputs = model(tensor_img)\n        outputs = postprocess(\n                    outputs, num_classes, conf_threshold,\n                    nmsthre, class_agnostic=True\n                )\n    \n    if outputs[0] is None:\n        return [], [], []\n    \n    outputs = outputs[0].cpu()\n    bboxes = outputs[:, 0:4]\n    \n    bboxes /= min(test_size[0] / img.shape[0], test_size[1] / img.shape[1])\n    bbclasses = outputs[:, 6]\n    scores = outputs[:, 4] * outputs[:, 5]\n    \n    if len(bboxes) == 0:\n        return [], [], []\n    \n    bboxes = bboxes.numpy()\n    \n    # format to coco\n    bboxes[:, 2] = bboxes[:, 2] - bboxes[:, 0]\n    bboxes[:, 3] = bboxes[:, 3] - bboxes[:, 1]    \n    \n    # Converts tensors to lists\n    return bboxes, bbclasses.tolist(), scores.tolist()","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:53:59.212598Z","iopub.execute_input":"2022-11-16T06:53:59.213253Z","iopub.status.idle":"2022-11-16T06:53:59.223320Z","shell.execute_reply.started":"2022-11-16T06:53:59.213218Z","shell.execute_reply":"2022-11-16T06:53:59.222225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Modified from https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507\n# Additions: \n#     allows customized box color (BGR)\n# Updates:\n# i changed it to yolov5 version. code is still dirty\ndef draw_yolov5_predictions(img, bboxes, scores, bbclasses, classes_dict, boxcolor = (0,0,255)):\n    outimg = img.copy()\n    for i in range(len(bboxes)):\n        box = bboxes[i]\n        cls_id = int(bbclasses[i])\n        score = scores[i]\n        x0 = int(box[0])\n        y0 = int(box[1])\n        x1 = x0 + int(box[2])\n        y1 = y0 + int(box[3])\n\n        cv2.rectangle(outimg, (x0, y0), (x1, y1), boxcolor, 2)\n        cv2.putText(outimg, '{}:{:.1f}%'.format(classes_dict[cls_id], score * 100), (x0, y0 - 3), cv2.FONT_HERSHEY_PLAIN, 0.8, boxcolor, thickness = 1)\n    return outimg","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:53:59.225069Z","iopub.execute_input":"2022-11-16T06:53:59.225450Z","iopub.status.idle":"2022-11-16T06:53:59.236776Z","shell.execute_reply.started":"2022-11-16T06:53:59.225417Z","shell.execute_reply":"2022-11-16T06:53:59.235867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# YOLOv5 Inference Size \nIMG_SIZE = 1280\nAUGMENT  = True\nCONF     = 0.01\nIOU = 0.65\n\n# Which IOU level? 0.3 to 0.8 with step of 0.05)\neval_IOU = 0.50\n\nmodel_path = '/kaggle/input/bestpt/best.pt'","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:53:59.237922Z","iopub.execute_input":"2022-11-16T06:53:59.241548Z","iopub.status.idle":"2022-11-16T06:53:59.250808Z","shell.execute_reply.started":"2022-11-16T06:53:59.241510Z","shell.execute_reply":"2022-11-16T06:53:59.249867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# load model\nmodel = load_model(model_path, conf=CONF, iou=IOU)","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:53:59.253060Z","iopub.execute_input":"2022-11-16T06:53:59.253347Z","iopub.status.idle":"2022-11-16T06:54:02.676520Z","shell.execute_reply.started":"2022-11-16T06:53:59.253323Z","shell.execute_reply":"2022-11-16T06:54:02.675573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Run on not seen data**","metadata":{}},{"cell_type":"code","source":"get_folder = 2\ninfer_test = infer_train[infer_train.folder == get_folder]","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:54:02.678195Z","iopub.execute_input":"2022-11-16T06:54:02.678914Z","iopub.status.idle":"2022-11-16T06:54:02.689643Z","shell.execute_reply.started":"2022-11-16T06:54:02.678868Z","shell.execute_reply":"2022-11-16T06:54:02.688555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Get image path and bbox**","metadata":{}},{"cell_type":"code","source":"# deepcopy bbox\n\ninfer_sample = infer_test\nimage_paths = infer_sample.image_path.tolist()\nground_truth = copy.deepcopy(infer_sample.bboxes.tolist())\ngt_mem = copy.deepcopy(infer_sample.bboxes.tolist())","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:54:02.697383Z","iopub.execute_input":"2022-11-16T06:54:02.697650Z","iopub.status.idle":"2022-11-16T06:54:02.752719Z","shell.execute_reply.started":"2022-11-16T06:54:02.697625Z","shell.execute_reply":"2022-11-16T06:54:02.751811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from IPython.display import display\ni = 2500\ntest_image_path = image_paths[i]\nimg = cv2.imread(test_image_path)[...,::-1]\n\n\n# bboxes, bbclasses, scores, img = tempt_yolov5_inference(model, img)\nbboxes, confis = predict(model, img, size=IMG_SIZE, augment=AUGMENT)\n\nprint(ground_truth[i])\nprint(bboxes)\n\n# # Draw Green ground truth box\n# out_image = draw_yolov5_predictions(img, ground_truth[i], [1.0] * len(ground_truth[i]), [0] * len(ground_truth[i]), COCO_CLASSES, (0,255,0))\nout_image = show_img(img, ground_truth[i],[1.0]*len(ground_truth[i]), bbox_format='coco', colors=[(255, 0, 0)]*(len(bboxes)))\n\n# # Draw Red inference box\nout_image = show_img(np.array(out_image), bboxes,confis, bbox_format='coco', colors=[(0,0,255)]*(len(bboxes)))\n\ndisplay(out_image)","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:54:02.754283Z","iopub.execute_input":"2022-11-16T06:54:02.754631Z","iopub.status.idle":"2022-11-16T06:54:03.205115Z","shell.execute_reply.started":"2022-11-16T06:54:02.754596Z","shell.execute_reply":"2022-11-16T06:54:03.204007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# modified from: https://www.kaggle.com/code/alexchwong/stop-guessing-conf-systematically-evaluate-f2/notebook\n# Confidence scores of true positives, false positives and count false negatives\nTP = [] # Confidence scores of true positives\nFP = [] # Confidence scores of true positives\nFN = 0  # Count of false negative boxes\n\nfor i in range(len(image_paths)):\n    test_image_path = image_paths[i]\n    img = cv2.imread(test_image_path)\n    img = cv2.imread(test_image_path)[...,::-1]\n    bboxes, scores = predict(model, img, size=IMG_SIZE, augment=AUGMENT)\n\n    # Test YOLOV5\n    test_image = ground_truth[i]\n    if len(bboxes) == 0:\n        # all ground_truth are false negative\n        FN += len(test_image)\n    else:\n        bb = bboxes.copy().tolist()\n        for idx, b in enumerate(bb):\n            b.append(scores[idx])\n        bb.sort(key = lambda x: x[4], reverse = True)\n        \n        if len(test_image) == 0:\n            # all bboxes are false positives\n            for b in bb:\n                FP.append(b[4])\n        else:\n            # match bbox with ground_truth\n            for b in bb:\n                matched = False\n                for g in test_image:\n                    # check whether ground_truth box is already matched to an inference bb\n                    if len(g) == 4:\n                        # g bbox is unmatched\n                        if IOU_coco(b, g) >= eval_IOU:\n                            g.append(b[4]) # assign confidence values to g; marks g as matched\n                            matched = True\n                            TP.append(b[4])\n                            break\n                if not matched:\n                    FP.append(b[4])\n            for g in test_image:\n                if len(g) == 4:\n                    FN += 1\nprint('success')","metadata":{"execution":{"iopub.status.busy":"2022-11-16T06:54:03.206564Z","iopub.execute_input":"2022-11-16T06:54:03.207139Z","iopub.status.idle":"2022-11-16T07:02:25.465637Z","shell.execute_reply.started":"2022-11-16T06:54:03.207102Z","shell.execute_reply":"2022-11-16T07:02:25.464379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Display metrics**","metadata":{}},{"cell_type":"code","source":"%matplotlib inline\nplt.hist(TP, 100)\nplt.title(\"CONF of true positives\")\nplt.xlabel('CONF')\nplt.ylabel('TP count')\nplt.show()\n\nprint(f'True positives = {len(TP)}')\nprint(f'False negatives = {FN}')","metadata":{"execution":{"iopub.status.busy":"2022-11-16T07:02:25.467622Z","iopub.execute_input":"2022-11-16T07:02:25.468429Z","iopub.status.idle":"2022-11-16T07:02:25.865081Z","shell.execute_reply.started":"2022-11-16T07:02:25.468383Z","shell.execute_reply":"2022-11-16T07:02:25.863983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# F2 metric\n\n**Take the average of F2 from different IOU**","metadata":{}},{"cell_type":"code","source":"F2_list = []\nF2_max = 0.0\nF2_maxc = -1.0\n\nfor con in np.arange(0.0, 1.0, 0.01):\n    FNcount = FN + sum(1 for i in TP if i < con)\n    TPcount = sum(1 for i in TP if i >= con)\n    FPcount = sum(1 for i in FP if i >= con)\n    R = TPcount / (TPcount + FNcount + 0.0001)\n    P = TPcount / (TPcount + FPcount + 0.0001)\n    F2 = (5 * P * R) / (4 * P + R + 0.0001)\n    F2_list.append((con, F2))\n    if F2_max < F2:\n        F2_max = F2\n        F2_maxc = con\n\nplt.scatter(*zip(*F2_list))\nplt.title(\"CONF vs F2 score\")\nplt.xlabel('CONF')\nplt.ylabel('F2')\nplt.show()\n\nprint(f'F2 max is {F2_max} at CONF = {F2_maxc}')","metadata":{"execution":{"iopub.status.busy":"2022-11-16T07:02:25.866696Z","iopub.execute_input":"2022-11-16T07:02:25.868373Z","iopub.status.idle":"2022-11-16T07:02:26.565979Z","shell.execute_reply.started":"2022-11-16T07:02:25.868329Z","shell.execute_reply":"2022-11-16T07:02:26.564980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**From W&B**","metadata":{}},{"cell_type":"markdown","source":"**Metrics （Results against Epoches）**","metadata":{}},{"cell_type":"code","source":"display(Image.open('/kaggle/input/cots-results/cots_result/metric.png')) ","metadata":{"execution":{"iopub.status.busy":"2022-11-16T07:28:21.592665Z","iopub.execute_input":"2022-11-16T07:28:21.593353Z","iopub.status.idle":"2022-11-16T07:28:21.890694Z","shell.execute_reply.started":"2022-11-16T07:28:21.593317Z","shell.execute_reply":"2022-11-16T07:28:21.889888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Validation batch images**","metadata":{}},{"cell_type":"code","source":"display(Image.open('/kaggle/input/cots-results/cots_result/valid.jpg') )","metadata":{"execution":{"iopub.status.busy":"2022-11-16T07:28:25.392683Z","iopub.execute_input":"2022-11-16T07:28:25.393266Z","iopub.status.idle":"2022-11-16T07:28:26.684703Z","shell.execute_reply.started":"2022-11-16T07:28:25.393229Z","shell.execute_reply":"2022-11-16T07:28:26.682932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Confusion matrix**","metadata":{}},{"cell_type":"code","source":"display(Image.open('/kaggle/input/cots-results/cots_result/confusion_matrix.png') )","metadata":{"execution":{"iopub.status.busy":"2022-11-16T07:28:30.391948Z","iopub.execute_input":"2022-11-16T07:28:30.392426Z","iopub.status.idle":"2022-11-16T07:28:30.791935Z","shell.execute_reply.started":"2022-11-16T07:28:30.392396Z","shell.execute_reply":"2022-11-16T07:28:30.790737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Labels**\n\nIt represents the occurrence of sizes of labels with respect to width and height for the four images at the bottom. The upper one image shows the size of labels in x and y","metadata":{}},{"cell_type":"code","source":"display(Image.open('/kaggle/input/cots-results/cots_result/Labels.jpg') )","metadata":{"execution":{"iopub.status.busy":"2022-11-16T07:28:34.077842Z","iopub.execute_input":"2022-11-16T07:28:34.078982Z","iopub.status.idle":"2022-11-16T07:28:34.654287Z","shell.execute_reply.started":"2022-11-16T07:28:34.078902Z","shell.execute_reply":"2022-11-16T07:28:34.652731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**P curve**","metadata":{}},{"cell_type":"code","source":"display(Image.open('/kaggle/input/cots-results/cots_result/P_curve.png') )","metadata":{"execution":{"iopub.status.busy":"2022-11-16T07:28:38.244431Z","iopub.execute_input":"2022-11-16T07:28:38.244788Z","iopub.status.idle":"2022-11-16T07:28:38.635755Z","shell.execute_reply.started":"2022-11-16T07:28:38.244758Z","shell.execute_reply":"2022-11-16T07:28:38.634759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**R curve**","metadata":{}},{"cell_type":"code","source":"display(Image.open('/kaggle/input/cots-results/cots_result/R_curve.png'))","metadata":{"execution":{"iopub.status.busy":"2022-11-16T07:28:40.747459Z","iopub.execute_input":"2022-11-16T07:28:40.747833Z","iopub.status.idle":"2022-11-16T07:28:41.071854Z","shell.execute_reply.started":"2022-11-16T07:28:40.747787Z","shell.execute_reply":"2022-11-16T07:28:41.070778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**PR curve**\n\nIt shows the tradeoff between precision and recall.","metadata":{}},{"cell_type":"code","source":"display(Image.open('/kaggle/input/cots-results/cots_result/PR_curve.png'))","metadata":{"execution":{"iopub.status.busy":"2022-11-16T07:28:43.975801Z","iopub.execute_input":"2022-11-16T07:28:43.976384Z","iopub.status.idle":"2022-11-16T07:28:44.435965Z","shell.execute_reply.started":"2022-11-16T07:28:43.976351Z","shell.execute_reply":"2022-11-16T07:28:44.435044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**F1 curve**\n\nIt balances the precision and recall. \n\nF1 = (2*precision*recall)/(precision+recall)\n\nHowever, in this case, the F2 score are used.","metadata":{}},{"cell_type":"code","source":"display(Image.open('/kaggle/input/cots-results/cots_result/F1_curve.png') )","metadata":{"execution":{"iopub.status.busy":"2022-11-16T07:28:47.065595Z","iopub.execute_input":"2022-11-16T07:28:47.065980Z","iopub.status.idle":"2022-11-16T07:28:47.377622Z","shell.execute_reply.started":"2022-11-16T07:28:47.065949Z","shell.execute_reply":"2022-11-16T07:28:47.376726Z"},"trusted":true},"execution_count":null,"outputs":[]}]}