{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Intro and refs","metadata":{}},{"cell_type":"markdown","source":"This is my first object detection project and I used the YOLOX framework and pretrained model downloaded from [this](https://github.com/Megvii-BaseDetection/YOLOX) gitub repo.  I used Remek Kinas's [excellent notebook](https://www.kaggle.com/remekkinas/yolox-training-pipeline-cots-dataset-lb-0-507) as an inspiration and reference for some parts, though I wrote most of what's here from scratch to encourage learning (anything copied is referenced inline).  Coming into this with no object detection experience there was a lot to learn, and hopefully this notebook might be helpful to someone else in the same boat.  \n\nI used the the YOLOX small model setting with COCO pretrained weights to perform detection, with a (very low) 320 x 320 image size (to fit in my computers 12gb vram).  Later I experimented with 1280 and 2560 resolutions on a remote server with an A6000, which greatly increased performance (the 320 resolution model was almost useless on the LB dataset).  So far I've only done limited hyperparameter tuning to get to 4.81 on the LB datset, but generally overfitting the train and validation set is an issue (given we are only training on 3 videos of data), so # of epochs is important.  \n\n**Submitting**\n\nThis notebook makes a lot of extra files in the working directory, which interferes with submitting.  I have a simpler [inference only notebook](https://www.kaggle.com/max237/getting-started-with-yolox-inference-only) for submitting\n\n\n**Some Terminology:**    \n\nIoU -  Intersection over union, a measure of how close the predicted bounding box overlaps with the actual box \nNMS - non maximum suppression, a technique to filter and dedup prediction boxes that overlap.  Uses IoU to measure the confidence of each box.  \nconf -  Confidence level/threshold for the prediction - experimenting with the threshold for this is important.  ","metadata":{}},{"cell_type":"markdown","source":"## Install and load dependencies  \n\nInstall YOLOX and any other dependencies.  Some of this can be skipped if running on a kaggle notebook, I set this up to be used on a remote ssh server for training as well which required different settings.  \n\nSome of these are commented out to allow the notebook to run without internet, and i'm using Remek Kina's [yolox-cots-models](https://www.kaggle.com/remekkinas/yolox-cots-models) dataset.  ","metadata":{}},{"cell_type":"code","source":"# install kaggle api (not necessary if running in a kaggle notebook)\n# %pip install --user kaggle","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Download the model repo\n# I'm using a preloaded dataset of this instead to avoid redownloading and installing\n#! git clone https://github.com/Megvii-BaseDetection/YOLOX -q\n    \n#! cp -r /kaggle/input/yolox-cots-models/YOLOX/ /kaggle/working/","metadata":{"execution":{"iopub.status.busy":"2022-02-06T17:55:40.193716Z","iopub.execute_input":"2022-02-06T17:55:40.194235Z","iopub.status.idle":"2022-02-06T17:55:40.21493Z","shell.execute_reply.started":"2022-02-06T17:55:40.19415Z","shell.execute_reply":"2022-02-06T17:55:40.214268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Install the model\n#%cd YOLOX\n\n# Install yolox  \n#!pip install -v -e .\n\n# Reset filepath\n#%cd ..","metadata":{"execution":{"iopub.status.busy":"2022-02-06T17:55:40.216538Z","iopub.execute_input":"2022-02-06T17:55:40.216825Z","iopub.status.idle":"2022-02-06T17:55:40.220858Z","shell.execute_reply.started":"2022-02-06T17:55:40.216791Z","shell.execute_reply":"2022-02-06T17:55:40.220099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load pretrained weights to yolox_s.pth\n#! wget https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_s.pth","metadata":{"execution":{"iopub.status.busy":"2022-02-06T17:55:40.222028Z","iopub.execute_input":"2022-02-06T17:55:40.222281Z","iopub.status.idle":"2022-02-06T17:55:40.22874Z","shell.execute_reply.started":"2022-02-06T17:55:40.222247Z","shell.execute_reply":"2022-02-06T17:55:40.228086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport torch\nfrom torchvision import transforms\nimport math\nimport time\nimport os\nimport shutil\nfrom skimage import io, transform\nimport PIL\nimport cv2\nimport IPython.display as display\nimport ast","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:45:32.809696Z","iopub.execute_input":"2022-02-09T16:45:32.810276Z","iopub.status.idle":"2022-02-09T16:45:35.962272Z","shell.execute_reply.started":"2022-02-09T16:45:32.810184Z","shell.execute_reply":"2022-02-09T16:45:35.961531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from shutil import copyfile\nfrom sklearn.model_selection import KFold\nimport random\nfrom collections import defaultdict\nimport json","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:45:35.963843Z","iopub.execute_input":"2022-02-09T16:45:35.964088Z","iopub.status.idle":"2022-02-09T16:45:36.216082Z","shell.execute_reply.started":"2022-02-09T16:45:35.964053Z","shell.execute_reply":"2022-02-09T16:45:36.215378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Add yolox path and load dependencies from their file structure\nimport sys\nsys.path.append(\"/kaggle/input/yolox-cots-models/YOLOX\")\nsys.path.append(\"./pycocotools-2.0.4\")","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:45:36.217458Z","iopub.execute_input":"2022-02-09T16:45:36.217772Z","iopub.status.idle":"2022-02-09T16:45:36.222221Z","shell.execute_reply.started":"2022-02-09T16:45:36.217733Z","shell.execute_reply":"2022-02-09T16:45:36.221523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Unzip and Install pycocotools from a file\n# This was necessary for inference using some of the YOLOX modules\n! tar -xf ../input/pycocotools/pycocotools-2.0.4.tar","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:45:40.720417Z","iopub.execute_input":"2022-02-09T16:45:40.720983Z","iopub.status.idle":"2022-02-09T16:45:41.437237Z","shell.execute_reply.started":"2022-02-09T16:45:40.720947Z","shell.execute_reply":"2022-02-09T16:45:41.436266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%cd pycocotools-2.0.4/\n!python setup.py build_ext --inplace\n%cd ..","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:45:41.439157Z","iopub.execute_input":"2022-02-09T16:45:41.439572Z","iopub.status.idle":"2022-02-09T16:45:48.982757Z","shell.execute_reply.started":"2022-02-09T16:45:41.439535Z","shell.execute_reply":"2022-02-09T16:45:48.981862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Add yolox path and pycocotools paths (from kaggle datasets)\nimport sys\nsys.path.append(\"../input/yolox-cots-models/YOLOX\")\nsys.path.append(\"./pycocotools-2.0.4/\")","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:46:08.897813Z","iopub.execute_input":"2022-02-09T16:46:08.898377Z","iopub.status.idle":"2022-02-09T16:46:08.902792Z","shell.execute_reply.started":"2022-02-09T16:46:08.898337Z","shell.execute_reply":"2022-02-09T16:46:08.901697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Install other YOLOX dependencies from Ramek's dataset\n! pip install loguru --no-index --find-links=file:///kaggle/input/yolox-cots-models/yolox-dep/\n! pip install thop --no-index --find-links=file:///kaggle/input/yolox-cots-models/yolox-dep/","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:45:48.991683Z","iopub.execute_input":"2022-02-09T16:45:48.992101Z","iopub.status.idle":"2022-02-09T16:46:05.985493Z","shell.execute_reply.started":"2022-02-09T16:45:48.992065Z","shell.execute_reply":"2022-02-09T16:46:05.984683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load yolox dependencies for inference later\nfrom yolox.data.data_augment import ValTransform\nfrom yolox.utils import postprocess","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:46:10.918061Z","iopub.execute_input":"2022-02-09T16:46:10.918856Z","iopub.status.idle":"2022-02-09T16:46:10.924718Z","shell.execute_reply.started":"2022-02-09T16:46:10.918809Z","shell.execute_reply":"2022-02-09T16:46:10.924012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load and preprocess data","metadata":{}},{"cell_type":"markdown","source":"### Format annotations and get cv folds\n\nI create 5 folders with COCO formated data, which makes it easy to feed into the YOLOX training script (which requires this format).  ","metadata":{}},{"cell_type":"code","source":"data_dir = '../input/tensorflow-great-barrier-reef'","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:46:13.442317Z","iopub.execute_input":"2022-02-09T16:46:13.443048Z","iopub.status.idle":"2022-02-09T16:46:13.450189Z","shell.execute_reply.started":"2022-02-09T16:46:13.443006Z","shell.execute_reply":"2022-02-09T16:46:13.449505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(f'{data_dir}/train.csv')\ndf.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:46:14.659815Z","iopub.execute_input":"2022-02-09T16:46:14.660117Z","iopub.status.idle":"2022-02-09T16:46:14.744242Z","shell.execute_reply.started":"2022-02-09T16:46:14.660083Z","shell.execute_reply":"2022-02-09T16:46:14.743544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Limit to annotated points only (a majority don't have annotations) for training\ndf_train = df[df['annotations'] != '[]'].copy(deep=True).reset_index(drop=True)\n# Convert from string \ndf_train['annotations'] = df_train['annotations'].apply(lambda x: ast.literal_eval(x))\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:54:59.208791Z","iopub.execute_input":"2022-02-09T16:54:59.209139Z","iopub.status.idle":"2022-02-09T16:54:59.676965Z","shell.execute_reply.started":"2022-02-09T16:54:59.209103Z","shell.execute_reply":"2022-02-09T16:54:59.676054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Get CV folds**","metadata":{}},{"cell_type":"code","source":"# Generate splits\n\nfolds = 5\nrands = 10 # set seed\ncv_shuffle = True\n\ncv = KFold(n_splits=folds, random_state=rands, shuffle=cv_shuffle)","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:47:13.570134Z","iopub.execute_input":"2022-02-09T16:47:13.570412Z","iopub.status.idle":"2022-02-09T16:47:13.574917Z","shell.execute_reply.started":"2022-02-09T16:47:13.570382Z","shell.execute_reply":"2022-02-09T16:47:13.573811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Copy files inte new directories for each fold\n# YOLOX requires the directories to be named train2017 and val2017\n\nindicies = list(range(len(df_train)))\nrandom.shuffle(indicies) # randomize samples before splitting\n\n# Save a dictionary of indicies for each fold\nfold_indicies = defaultdict(dict)\n\n# Generate files and names for each fold\nfor fold_num, (train_idx, valid_idx) in enumerate(cv.split(indicies)):\n    \n    # Fold directories (drop the folder to prevent mixups if re-running)\n    \n    train_path = f'./fold_{fold_num}/train2017'\n    valid_path = f'./fold_{fold_num}/val2017'\n    \n    try: \n        shutil.rmtree(train_path)\n    except(FileNotFoundError):\n        pass\n    finally:\n        os.makedirs(train_path)\n        \n    try: \n        shutil.rmtree(valid_path)\n    except(FileNotFoundError):\n        pass\n    finally:\n        os.makedirs(valid_path)\n    \n    \n    # Training data per fold\n    for i in train_idx:\n        img_num = df_train.iloc[i, 4].split('-')[-1]\n        img_id = df_train.iloc[i, 4]\n        vid_num = df_train.iloc[i, 0]\n        copyfile(f'{data_dir}/train_images/video_{vid_num}/{img_num}.jpg', \n                 f'{train_path}/{img_id}.jpg')\n    \n    # Validation data per fold\n    for i in valid_idx:\n        img_num = df_train.iloc[i, 4].split('-')[-1]\n        img_id = df_train.iloc[i, 4]\n        vid_num = df_train.iloc[i, 0]\n        copyfile(f'{data_dir}//train_images/video_{vid_num}/{img_num}.jpg', \n                 f'{valid_path}/{img_id}.jpg') # save with a unique index file name matching the df index\n\n    fold_indicies[fold_num]['train'] = train_idx\n    fold_indicies[fold_num]['valid'] = valid_idx\n    ","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:47:20.375249Z","iopub.execute_input":"2022-02-09T16:47:20.375702Z","iopub.status.idle":"2022-02-09T16:49:04.942011Z","shell.execute_reply.started":"2022-02-09T16:47:20.375665Z","shell.execute_reply":"2022-02-09T16:49:04.940356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fold_indicies.keys()","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:49:04.946161Z","iopub.execute_input":"2022-02-09T16:49:04.946457Z","iopub.status.idle":"2022-02-09T16:49:04.954702Z","shell.execute_reply.started":"2022-02-09T16:49:04.946420Z","shell.execute_reply":"2022-02-09T16:49:04.953629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fold_indicies[0].keys()","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:49:04.956127Z","iopub.execute_input":"2022-02-09T16:49:04.957147Z","iopub.status.idle":"2022-02-09T16:49:05.657410Z","shell.execute_reply.started":"2022-02-09T16:49:04.957109Z","shell.execute_reply":"2022-02-09T16:49:05.656716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create coco annotation files\n\nThis section is adapted from Remek's notebook: https://www.kaggle.com/remekkinas/yolox-training-pipeline-cots-dataset-lb-0-507  \nWhich is taken from Awsaf's notebook: https://www.kaggle.com/awsaf49/great-barrier-reef-yolov5-train","metadata":{}},{"cell_type":"code","source":"def save_annot_json(json_annotation, filename):\n    with open(filename, 'w') as f:\n        output_json = json.dumps(json_annotation)\n        f.write(output_json)","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:49:05.659439Z","iopub.execute_input":"2022-02-09T16:49:05.661392Z","iopub.status.idle":"2022-02-09T16:49:05.856451Z","shell.execute_reply.started":"2022-02-09T16:49:05.661350Z","shell.execute_reply":"2022-02-09T16:49:05.855541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def dataset2coco(df):\n    \n    annotion_id = 0\n    \n    annotations_json = {\n        \"info\": [],\n        \"licenses\": [],\n        \"categories\": [],\n        \"images\": [],\n        \"annotations\": []\n    }\n    \n    info = {\n        \"year\": \"2021\",\n        \"version\": \"1\",\n        \"description\": \"COTS dataset - COCO format\",\n        \"contributor\": \"\",\n        \"url\": \"https://kaggle.com\",\n        \"date_created\": \"2022-01-29T15:01:26+00:00\"\n    }\n    annotations_json[\"info\"].append(info)\n    \n    lic = {\n            \"id\": 1,\n            \"url\": \"\",\n            \"name\": \"Unknown\"\n        }\n    annotations_json[\"licenses\"].append(lic)\n\n    classes = {\"id\": 0, \"name\": \"starfish\", \"supercategory\": \"none\"}\n\n    annotations_json[\"categories\"].append(classes)\n\n    \n    for ann_row in df.itertuples():\n            \n        images = {\n            \"id\": ann_row[0],\n            \"license\": 1,\n            \"file_name\": str(ann_row[5]) + '.jpg', # use the image id with video information\n            \"height\": 720,\n            \"width\": 1280,\n            \"date_captured\": \"2021-11-30T15:01:26+00:00\"\n        }\n        \n        annotations_json[\"images\"].append(images)\n        \n        for bbox in ann_row.annotations:\n            # some boxes in COTS are outside the image height and width\n            if (bbox['x'] + bbox['width'] > 1280):\n                b_width = bbox['x'] - 1280 \n            if (bbox['y'] + bbox['height'] > 720):\n                b_height = bbox['y'] - 720 \n                \n            image_annotations = {\n                \"id\": annotion_id,\n                \"image_id\": ann_row[0],\n                \"category_id\": 0,\n                \"bbox\": [bbox['x'], bbox['y'], bbox['width'], bbox['height']],\n                \"area\": bbox['width'] * bbox['height'],\n                \"segmentation\": [],\n                \"iscrowd\": 0\n            }\n            \n            annotion_id += 1\n            annotations_json[\"annotations\"].append(image_annotations)\n        \n        \n    print(f\"Dataset COTS annotation to COCO json format completed! Files: {len(df)}\")\n    return annotations_json","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:49:05.858091Z","iopub.execute_input":"2022-02-09T16:49:05.858397Z","iopub.status.idle":"2022-02-09T16:49:06.167251Z","shell.execute_reply.started":"2022-02-09T16:49:05.858355Z","shell.execute_reply":"2022-02-09T16:49:06.166281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Add a training and  to each fold\n# Use the indicies from the file moving step, which match the file names\nfor fold_num, tv_indicies in fold_indicies.items():\n    train_annot_json = dataset2coco(df_train.iloc[tv_indicies['train'], :])\n    valid_annot_json = dataset2coco(df_train.iloc[tv_indicies['valid'], :])\n\n    if not os.path.exists(f'./fold_{fold_num}/annotations'):\n        os.makedirs(f'./fold_{fold_num}/annotations')\n    \n    save_annot_json(train_annot_json, f'fold_{fold_num}/annotations/train.json')\n    save_annot_json(valid_annot_json, f'fold_{fold_num}/annotations/valid.json')","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:49:06.169530Z","iopub.execute_input":"2022-02-09T16:49:06.169818Z","iopub.status.idle":"2022-02-09T16:49:07.228141Z","shell.execute_reply.started":"2022-02-09T16:49:06.169780Z","shell.execute_reply":"2022-02-09T16:49:07.227377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.iloc[fold_indicies[0]['train']].head(20)","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:49:07.229448Z","iopub.execute_input":"2022-02-09T16:49:07.229911Z","iopub.status.idle":"2022-02-09T16:49:07.263596Z","shell.execute_reply.started":"2022-02-09T16:49:07.229871Z","shell.execute_reply":"2022-02-09T16:49:07.262873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train the model\n\nTo do this using the YOLOX author's provided scripts, we first need to create an Exp config file.  \n\nTutorial: https://github.com/Megvii-BaseDetection/YOLOX/blob/main/docs/train_custom_data.md  \nTrain pipeline and args: https://github.com/Megvii-BaseDetection/YOLOX/blob/main/tools/train.py  \nExp defaults: https://github.com/Megvii-BaseDetection/YOLOX/blob/main/yolox/exp/yolox_base.py  \n\nI currently haven't optimized the training process at all (using the author's defaults), and I'm using only 30 epochs and a small 320x320 to fit in VRAM on my local GPU and train quickly.  ","metadata":{}},{"cell_type":"markdown","source":"**Create the config file**","metadata":{}},{"cell_type":"code","source":"# Create separate experiment configs for each fold\n\nfor i in range(folds):\n    print(i)\n    exp_string = f\"\"\"import os\nfrom yolox.exp import Exp as MyExp\n\nclass Exp(MyExp):\n    def __init__(self):\n        super(Exp, self).__init__()\n        self.depth = 0.33 # values for the yolox_s\n        self.width = 0.50 # values for the yolox_s\n        self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(\".\")[0]\n\n        # Define yourself dataset path\n        self.data_dir = \"./fold_{i}\"\n        self.train_ann = \"train.json\"\n        self.val_ann = \"valid.json\"\n\n        self.num_classes = 1\n\n        self.warmup_epochs = 4\n        self.max_epoch = 18\n        self.data_num_workers = 8\n\n        self.print_interval = 100\n        self.eval_interval = 2\n\n        self.input_size = (2560, 2560)\n        self.test_size = (2560, 2560)\n\"\"\"\n    \n    exp_file_path = f'./barrier_reef_exp_train_{i}.py'\n    with open(exp_file_path, mode='w') as outfile:\n        outfile.write(exp_string)","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:49:54.377000Z","iopub.execute_input":"2022-02-09T16:49:54.377734Z","iopub.status.idle":"2022-02-09T16:49:54.386409Z","shell.execute_reply.started":"2022-02-09T16:49:54.377693Z","shell.execute_reply":"2022-02-09T16:49:54.385450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Keep for inference - exp files for various resolutions\n\nexp_file_path = './barrier_reef_exp.py'\n\nwith open(exp_file_path, mode='w') as outfile:\n    outfile.write(\"\"\"import os\nfrom yolox.exp import Exp as MyExp\n\nclass Exp320(MyExp):\n    def __init__(self):\n        super(Exp320, self).__init__()\n        self.depth = 0.33 # values for the yolox_s\n        self.width = 0.50 # values for the yolox_s\n        self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(\".\")[0]\n\n        # Define yourself dataset path\n        self.data_dir = \"./fold_0\"\n        self.train_ann = \"train.json\"\n        self.val_ann = \"valid.json\"\n\n        self.num_classes = 1\n\n        self.warmup_epochs = 4\n        self.max_epoch = 15\n        self.data_num_workers = 8\n\n        self.print_interval = 40\n        self.eval_interval = 1\n\n        self.input_size = (320, 320)\n        self.test_size = (320, 320)\n\nclass Exp1280(MyExp):\n    def __init__(self):\n        super(Exp1280, self).__init__()\n        self.depth = 0.33 # values for the yolox_s\n        self.width = 0.50 # values for the yolox_s\n        self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(\".\")[0]\n\n        # Define yourself dataset path\n        self.data_dir = \"./fold_0\"\n        self.train_ann = \"train.json\"\n        self.val_ann = \"valid.json\"\n\n        self.num_classes = 1\n\n        self.warmup_epochs = 4\n        self.max_epoch = 15\n        self.data_num_workers = 8\n\n        self.print_interval = 40\n        self.eval_interval = 1\n\n        self.input_size = (1280, 1280)\n        self.test_size = (1280, 1280)\n        \n\nclass Exp2560(MyExp):\n    def __init__(self):\n        super(Exp2560, self).__init__()\n        self.depth = 0.33 # values for the yolox_s\n        self.width = 0.50 # values for the yolox_s\n        self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(\".\")[0]\n\n        # Define yourself dataset path\n        self.data_dir = \"./fold_0\"\n        self.train_ann = \"train.json\"\n        self.val_ann = \"valid.json\"\n\n        self.num_classes = 1\n\n        self.warmup_epochs = 4\n        self.max_epoch = 15\n        self.data_num_workers = 8\n\n        self.print_interval = 40\n        self.eval_interval = 1\n\n        self.input_size = (2560, 2560)\n        self.test_size = (2560, 2560)\n\"\"\")","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:49:54.727914Z","iopub.execute_input":"2022-02-09T16:49:54.728299Z","iopub.status.idle":"2022-02-09T16:49:54.741182Z","shell.execute_reply.started":"2022-02-09T16:49:54.728259Z","shell.execute_reply":"2022-02-09T16:49:54.740378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Run YOLOX training script\n\nI have this commented out for this submission version, but uncommenting the command below would run the training process and save the weights.  \n\nTo run inference in the kaggle notebook, i've uploaded my trained weights as data 'best-ckpt_{resolution}'","metadata":{}},{"cell_type":"code","source":"# Run the training pipeline\n# -d is devices\n# -b is batch size\n# -fp16 is mix precision training\n# -o is occupy GPU memory first for training\n# -c is a checkpoint file (for loading pretrained weights)\n\n# ! python3 YOLOX/tools/train.py -f barrier_reef_exp_train.py -d 1 -b 8 --fp16 -o -c yolox_s.pth --cache","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:49:56.317411Z","iopub.execute_input":"2022-02-09T16:49:56.318277Z","iopub.status.idle":"2022-02-09T16:49:56.322304Z","shell.execute_reply.started":"2022-02-09T16:49:56.318238Z","shell.execute_reply":"2022-02-09T16:49:56.321190Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inference and results","metadata":{}},{"cell_type":"markdown","source":"### Visualize test predictions\n\nTo start, we can usse the YOLOX demo script to run the model with the latest trained weights on some sample images, and then plot the bboxes on top of the images.  The script also helpfully draws the box predictions onto the images and saves them, and i've implemented some code to draw the true boxes to compare.  \n\nExample of how to run the script here: https://github.com/Megvii-BaseDetection/YOLOX/blob/main/docs/quick_run.md  ","metadata":{}},{"cell_type":"code","source":"test_image_path = 'fold_0/val2017/0-9653.jpg'\n# model_weights_path = 'YOLOX_outputs/barrier_reef_exp/best_ckpt.pth' # Local\nmodel_weights_path = '../input/yolox-s-trained-weights/best_ckpt_1280.pth' # Kaggle notebook","metadata":{"execution":{"iopub.status.busy":"2022-02-09T17:04:36.398119Z","iopub.execute_input":"2022-02-09T17:04:36.398386Z","iopub.status.idle":"2022-02-09T17:04:36.402203Z","shell.execute_reply.started":"2022-02-09T17:04:36.398358Z","shell.execute_reply":"2022-02-09T17:04:36.401290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use the demo.py tool to run inference\n# Currently not running this\n\n#! python3 YOLOX/tools/demo.py image \\\n#    -f barrier_reef_exp.py \\\n#    -c {model_weights_path} \\\n#    --path {test_image_path} \\\n#    --conf 0.1 \\\n#    --nms 0.3 \\\n#    --tsize 320 \\\n#    --device gpu \\\n#    --save_result\n    ","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:50:47.252217Z","iopub.execute_input":"2022-02-09T16:50:47.252548Z","iopub.status.idle":"2022-02-09T16:50:47.256357Z","shell.execute_reply.started":"2022-02-09T16:50:47.252514Z","shell.execute_reply":"2022-02-09T16:50:47.255520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize predicted boxes\n# Copy the image path output from the previous step\n\n#img_path = './YOLOX_outputs/barrier_reef_exp/vis_res/2022_02_02_00_24_37/0-9653.jpg'\n#test_img = PIL.Image.open(img_path)\n#display.display(test_img)","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:50:47.604252Z","iopub.execute_input":"2022-02-09T16:50:47.604961Z","iopub.status.idle":"2022-02-09T16:50:47.610267Z","shell.execute_reply.started":"2022-02-09T16:50:47.604922Z","shell.execute_reply":"2022-02-09T16:50:47.609550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the actual ground truth locations (use cv2 to draw in the boxes)\n\ntest_img = cv2.imread(test_image_path)\nboxes = df_train[df_train['image_id'] == test_image_path.split('/')[-1][:-4]]['annotations'].tolist()[0]\n\nfor box in boxes:\n    upper_left = (int(box['x']), int(box['y']))\n    lower_right = (int(box['x'] + box['width']), int(box['y'] + box['height']))\n    color = (255, 0, 0)\n    test_img = cv2.rectangle(test_img, upper_left, lower_right, color=color, thickness = 2)\n\ntest_img = cv2.cvtColor(test_img, cv2.COLOR_BGR2RGB)\ntest_img_pil = PIL.Image.fromarray(test_img)\ndisplay.display(test_img_pil)","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:50:48.207035Z","iopub.execute_input":"2022-02-09T16:50:48.207851Z","iopub.status.idle":"2022-02-09T16:50:48.670841Z","shell.execute_reply.started":"2022-02-09T16:50:48.207799Z","shell.execute_reply":"2022-02-09T16:50:48.670049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Run on the test dataset for submission\n\n**Inference in notebook**\n\nSince we are getting images directly from the api and not from a file, we unfortunately can't just use the YOLOX demo tool to get our box predictions.  Instead, we have to adapt pieces of that tool to fit our needs.  I adapt the [inference function from demo.py](https://github.com/Megvii-BaseDetection/YOLOX/blob/main/tools/demo.py#L132) script for this purpose.  \n\nUsing this also requires installing some new python libaries, which I do offline from Remek's dataset.  Instructions [here](https://www.kaggle.com/samuelepino/pip-installing-packages-with-no-internet).","metadata":{}},{"cell_type":"code","source":"from barrier_reef_exp import Exp1280","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:51:08.084438Z","iopub.execute_input":"2022-02-09T16:51:08.085142Z","iopub.status.idle":"2022-02-09T16:51:08.089850Z","shell.execute_reply.started":"2022-02-09T16:51:08.085103Z","shell.execute_reply":"2022-02-09T16:51:08.088976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Load the model object**\n\nUses the get_model function from the [experiment base class] (https://github.com/Megvii-BaseDetection/YOLOX/blob/main/yolox/exp/yolox_base.py)","metadata":{}},{"cell_type":"code","source":"def get_trained_model(experiment, weights):\n    \n    # Use the experiment built in function to generate the same model we trained with\n    model = experiment.get_model()\n    \n    # Inference on gpu\n    model.cuda()\n    \n    # Turn off training mode so the model so it won't try to calculate loss\n    model.eval()\n    model.head.training=False\n    model.training=False\n    \n    # Load in the weights from training\n    best_weights = torch.load(weights)\n    model.load_state_dict(best_weights['model'])\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:51:11.103688Z","iopub.execute_input":"2022-02-09T16:51:11.104355Z","iopub.status.idle":"2022-02-09T16:51:11.109089Z","shell.execute_reply.started":"2022-02-09T16:51:11.104320Z","shell.execute_reply":"2022-02-09T16:51:11.108413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Extract the boxes from the processed predictions \n\ndef get_boxes(outputs):\n    output = outputs[0][0]\n    \n    if output == None:\n        return {'bboxes': [], 'scores': []}\n    # move to cpu\n    output = output.cpu()\n    \n    img_info = outputs[1]\n    \n    bboxes = output[:, 0:4]/img_info['ratio']\n    scores = output[:, 4] * output[:, 5]\n    \n    return {'bboxes': bboxes, 'scores': scores}","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:51:11.451611Z","iopub.execute_input":"2022-02-09T16:51:11.452389Z","iopub.status.idle":"2022-02-09T16:51:11.457763Z","shell.execute_reply.started":"2022-02-09T16:51:11.452351Z","shell.execute_reply":"2022-02-09T16:51:11.456891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Custom implementation of the inference function from demo.py \n# Takes in an image object instead of a filepath\n\ndef inference(img, model, experiment, device):\n    \n        test_size = experiment.test_size\n        confthre = experiment.test_conf\n        nmsthre = experiment.nmsthre\n    \n        img_info = {\"id\": 0}\n        img_info[\"file_name\"] = None\n\n        height, width = img.shape[:2]\n        img_info[\"height\"] = height\n        img_info[\"width\"] = width\n        img_info[\"raw_img\"] = img\n\n        ratio = min(test_size[0] / img.shape[0], test_size[1] / img.shape[1])\n        img_info[\"ratio\"] = ratio\n        \n        preproc = ValTransform(legacy=False)\n        \n        img, _ = preproc(img, None, test_size)\n        img = torch.from_numpy(img).unsqueeze(0)\n        img = img.float()\n        if device == \"gpu\":\n            img = img.cuda()\n\n        with torch.no_grad():\n            t0 = time.time()\n            outputs = model(img)\n\n            outputs = postprocess(\n                outputs, 1, confthre,\n                nmsthre, class_agnostic=True\n            )\n        return outputs, img_info","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:51:12.629793Z","iopub.execute_input":"2022-02-09T16:51:12.630589Z","iopub.status.idle":"2022-02-09T16:51:12.641995Z","shell.execute_reply.started":"2022-02-09T16:51:12.630550Z","shell.execute_reply":"2022-02-09T16:51:12.641212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get box predictions for a single image and given thresholds\n\ndef barrier_reef_inference(exp_file, weights, test_image, \n                           conf_threshold=0.1, nms_threshold=0.3,\n                           device='gpu'):\n    \n    # Load the experiment file\n    experiment = exp_file\n    \n    # Set up the model and weights\n    model = get_trained_model(experiment, weights)\n    \n    # Set custom thresholds for inference\n    experiment.test_conf = conf_threshold\n    experiment.nmsthre = nms_threshold\n    \n    test_size = experiment.test_size\n    \n    # Run the image through the model\n    outputs = inference(test_image, model, experiment, device)\n    \n    return get_boxes(outputs)\n","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:51:15.175998Z","iopub.execute_input":"2022-02-09T16:51:15.176713Z","iopub.status.idle":"2022-02-09T16:51:15.182391Z","shell.execute_reply.started":"2022-02-09T16:51:15.176674Z","shell.execute_reply":"2022-02-09T16:51:15.181582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Try it out on the test image \n\n# Load as a array first\ntest_image = cv2.imread(test_image_path)\nprint(test_image.shape)\nexperiment = Exp1280()\n\nbox_preds = barrier_reef_inference(experiment, model_weights_path, test_image,\n                                   conf_threshold=0.1, nms_threshold=0.3)\nprint(box_preds)","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:51:21.294552Z","iopub.execute_input":"2022-02-09T16:51:21.295132Z","iopub.status.idle":"2022-02-09T16:51:31.213978Z","shell.execute_reply.started":"2022-02-09T16:51:21.295094Z","shell.execute_reply":"2022-02-09T16:51:31.213166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def visualize_boxes(test_image, boxes, scores):\n\n    for box in boxes:\n        \n        x0 = int(box[0])\n        y0 = int(box[1])\n        x1 = int(box[2])\n        y1 = int(box[3])\n\n        color = (255, 0, 0)\n        test_image = cv2.rectangle(test_image, (x0, y0), (x1, y1), color=color, thickness = 2)\n    \n    test_image = cv2.cvtColor(test_image, cv2.COLOR_BGR2RGB)\n    test_image_pil = PIL.Image.fromarray(test_image)\n    \n    return test_image_pil\n\ntest_image = cv2.imread(test_image_path)\ntest_image_pil = visualize_boxes(test_image, box_preds['bboxes'], box_preds['scores'])\ndisplay.display(test_image_pil)","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:51:31.215760Z","iopub.execute_input":"2022-02-09T16:51:31.216190Z","iopub.status.idle":"2022-02-09T16:51:31.592159Z","shell.execute_reply.started":"2022-02-09T16:51:31.216142Z","shell.execute_reply":"2022-02-09T16:51:31.591199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Compared to the ground truth above, our model is correcty identifying all of the COTS in the test image from the validation set.  ","metadata":{}},{"cell_type":"markdown","source":"### Predict boxes and calculate F2 metric on the validation data set\n\nTo get an idea of the optimal confidence score cutoff, we can calculate the evaluation metric F2 score for different confidence scores.  \n\nThis is currently greatly overestimating the true impact vs. LB, since i'm looking at the performance on all of the training data (to make sure we include empty images), but it gives an idea of how score can change for different confidence thresholds.  Regardless, we are only training on 3 vidoes, so calculating F2 score on a validation dataset with randomly selected images from those videos is also going to greatly overstate performance.  An alternative technique would be to use all images from one of the videos as validation data, but that would cut down on training data significantly.  ","metadata":{}},{"cell_type":"code","source":"def IoU(pred_box, label_box):\n    \n    # Extract prediction boxes\n    x0 = pred_box[0]\n    y0 = pred_box[1]\n    width = pred_box[2]\n    height = pred_box[3]\n    \n    # Extract label boxes\n    x0_l = label_box['x']\n    y0_l = label_box['y']\n    width_l = label_box['width']\n    height_l = label_box['height']\n\n    # Get the overlap in width ranges\n    left_bound = max(x0, x0_l)\n    right_bound = min(x0 + width, x0_l + width_l)\n    upper_bound = min(y0, y0_l)\n    lower_bound = max(y0 - height, y0_l - height_l)\n    #print(left_bound, right_bound, lower_bound,  upper_bound)\n    \n    # Calculate metrics\n    intersection = max(right_bound - left_bound, 0) * max(upper_bound - lower_bound, 0)\n    area = width * height\n    area_l = width_l * height_l\n    union = area + area_l - intersection\n\n    # Calculate IoU\n    return intersection/union\n    \npred_box = [25, 40, 30, 25]\nlabel_box = {'x': 30, 'y': 30, 'width': 25, 'height': 25}\n\nIoU(pred_box, label_box)\n","metadata":{"execution":{"iopub.status.busy":"2022-02-09T16:58:30.643735Z","iopub.execute_input":"2022-02-09T16:58:30.643998Z","iopub.status.idle":"2022-02-09T16:58:30.656054Z","shell.execute_reply.started":"2022-02-09T16:58:30.643971Z","shell.execute_reply":"2022-02-09T16:58:30.655075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**F2 Score**\n\nExplanation of 'micro averaging': \nhttps://www.kaggle.com/enforcer007/what-is-micro-averaged-f1-score?scriptVersionId=36576991\n\nBasically using all of the tp/fp/fn together with equal weights to calculate F2.  ","metadata":{}},{"cell_type":"code","source":"def FScore(results_overall, beta=2):\n    numerator = ((1+beta**2) * results_overall['tp']) \n    denominator = ((1+beta**2) * results_overall['tp'] + beta * results_overall['fn']\n                   + results_overall['fp'])\n    \n    return numerator/denominator","metadata":{"execution":{"iopub.status.busy":"2022-02-09T17:02:50.213175Z","iopub.execute_input":"2022-02-09T17:02:50.213440Z","iopub.status.idle":"2022-02-09T17:02:50.218371Z","shell.execute_reply.started":"2022-02-09T17:02:50.213412Z","shell.execute_reply":"2022-02-09T17:02:50.217323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Another (real) example IoU calc\npred_box1 = (158, 422, 56, 42)\nlabel_box = {'x': 167, 'y': 425, 'width': 47, 'height': 31}\n\nprint(IoU(pred_box1, label_box))","metadata":{"execution":{"iopub.status.busy":"2022-02-09T17:03:03.316533Z","iopub.execute_input":"2022-02-09T17:03:03.317298Z","iopub.status.idle":"2022-02-09T17:03:03.322661Z","shell.execute_reply.started":"2022-02-09T17:03:03.317261Z","shell.execute_reply":"2022-02-09T17:03:03.321904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get all training images (which include empty images)\n\ndf_all = pd.read_csv(f'{data_dir}/train.csv')\n\ndf_all['annotations'] = df_all['annotations'].apply(lambda x: ast.literal_eval(x))\ndf_all.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-02-09T17:03:06.474062Z","iopub.execute_input":"2022-02-09T17:03:06.474593Z","iopub.status.idle":"2022-02-09T17:03:07.089043Z","shell.execute_reply.started":"2022-02-09T17:03:06.474539Z","shell.execute_reply":"2022-02-09T17:03:07.088340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get image list\nimage_list_val = []\nfor val_image_group in os.walk('fold_0/val2017'):\n    for counter, val_image in enumerate(val_image_group[2][1:]):\n        image_list_val.append(val_image)","metadata":{"execution":{"iopub.status.busy":"2022-02-09T17:03:07.886913Z","iopub.execute_input":"2022-02-09T17:03:07.887182Z","iopub.status.idle":"2022-02-09T17:03:07.895026Z","shell.execute_reply.started":"2022-02-09T17:03:07.887152Z","shell.execute_reply":"2022-02-09T17:03:07.894254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Image list for all images \nimage_list_all = []\nfor val_image_group in list(os.walk('../input/tensorflow-great-barrier-reef/train_images'))[1:]:\n    for val_image in val_image_group[2]:\n        image_list_all.append(val_image_group[0] + '/' + val_image)","metadata":{"execution":{"iopub.status.busy":"2022-02-09T17:03:45.120045Z","iopub.execute_input":"2022-02-09T17:03:45.120338Z","iopub.status.idle":"2022-02-09T17:03:56.821064Z","shell.execute_reply.started":"2022-02-09T17:03:45.120305Z","shell.execute_reply":"2022-02-09T17:03:56.820287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Iterate through validation images and calculate F2 score\n# Use the original images that include examples with no starfish\n# Change to batch to improve performance  \n\nfrom collections import OrderedDict, defaultdict\n\ndef get_f2_score(image_list, iou_thresh, conf_threshold, nms_theshold,\n                 experiment, model_weights_path, sample_limit=-1):\n\n    # Track total tp, fp, tn\n    results_overall = {}\n    results_overall['tp'] = 0\n    results_overall['fp'] = 0\n    results_overall['fn'] = 0\n\n    # Shuffle to be less biased towards early images if limited count\n    random.shuffle(image_list)\n\n    for counter, val_image_path in enumerate(image_list):\n        #print(f'\\n iteration {counter}, {val_image}')\n\n        #val_image_path = f'fold_0/val2017/{val_image}'\n        val_image_arr = cv2.imread(val_image_path)\n\n        # Get predicted boxes\n        box_preds = barrier_reef_inference(experiment, model_weights_path, val_image_arr,\n                                   conf_threshold=conf_threshold, nms_threshold=0.3)\n\n        # Order the predictions by score to allow checking in order\n        preds = OrderedDict()\n        for i in range(len(box_preds['bboxes'])):\n            box = box_preds['bboxes'][i]\n            score = box_preds['scores'][i]\n\n            x0 = int(box[0])\n            y0 = int(box[1])\n            x1 = int(box[2])\n            y1 = int(box[3])\n\n            box_width = x1 - x0\n            box_height = y1 - y0\n\n            preds[score.item()] = (x0, y0, box_width, box_height)\n\n        # Get the image name from the file path and extract label boxes\n        split_path = val_image_path.split('/')\n        if '-' in split_path[-1]:\n            img_name = split_path[-1][:-4]\n            label_boxes = df_train[df_train['image_id'] == img_name]['annotations'].tolist()[0]\n        else:\n            img_name = split_path[-2][-1] + '-' + split_path[-1][:-4]\n            label_boxes = df_all[df_all['image_id'] == img_name]['annotations'].tolist()[0]\n\n        #print('label boxes:', label_boxes)\n\n        # Initialize tp/fp/fn counts\n        tp = 0\n        fp = len(preds)\n        fn = len(label_boxes)\n\n        # Check each prediction for matching labels in score order \n        for score, pred_box in preds.items():\n            #print('score, current pred box: ', score, pred_box)\n            # Check it against each possible label\n            for i, label_box in enumerate(label_boxes):\n                #print('current label box:', label_box)    \n                # Calculate IoU\n                iou = IoU(pred_box, label_box)\n                #print('iou: ', iou)\n                if iou >= iou_thresh:\n                    # Add to true positives\n                    tp += 1\n                    # Remove from fn and fp \n                    fp -= 1\n                    fn -= 1\n                    # If a match is found, skip the remaining labels\n                    break\n\n        results_overall['tp'] += tp\n        results_overall['fp'] += fp\n        results_overall['fn'] += fn\n\n        #print('Results (tp, fp, fn): ', tp, fp, fn)\n        #print('\\n')\n        if sample_limit == -1:\n            continue\n        else:\n            if counter >= sample_limit:\n                break\n        \n    return (results_overall, FScore(results_overall))\n\n","metadata":{"execution":{"iopub.status.busy":"2022-02-09T17:04:09.773308Z","iopub.execute_input":"2022-02-09T17:04:09.773655Z","iopub.status.idle":"2022-02-09T17:04:09.788357Z","shell.execute_reply.started":"2022-02-09T17:04:09.773621Z","shell.execute_reply":"2022-02-09T17:04:09.787636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Test\niou_thresh = 0.5\nconf_threshold = 0.1\nnms_threshold = 0.3\nexperiment = Exp1280()\n\nresults_overall, fscore = get_f2_score(image_list_all, iou_thresh, conf_threshold, nms_threshold, \n                                           experiment, model_weights_path, sample_limit=100)\n\nprint('Results overall: ', results_overall)\nprint('F2 Score: ', fscore)","metadata":{"execution":{"iopub.status.busy":"2022-02-09T17:04:57.340599Z","iopub.execute_input":"2022-02-09T17:04:57.341002Z","iopub.status.idle":"2022-02-09T17:05:21.064870Z","shell.execute_reply.started":"2022-02-09T17:04:57.340961Z","shell.execute_reply":"2022-02-09T17:05:21.063456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Currently this is only calculated on the validation set, which is wrong.  ","metadata":{}},{"cell_type":"code","source":"# Check multiple confidence thresholds\n\niou_thresh = 0.5\nnms_threshold = 0.3\nexperiment = Exp1280()\n\n# model_weights_path = 'YOLOX_outputs/barrier_reef_exp/best_ckpt_1280.pth' \n# experiment = Exp1280()\n\n# Increase the sample_limit for more accurate results\nfor conf_threshold in np.arange(0.05, 0.5, 0.05):\n    results_overall, fscore = get_f2_score(image_list_all, iou_thresh, conf_threshold, nms_threshold, \n                                           experiment, model_weights_path, sample_limit=100)\n\n    print('Confidence Threshold: ', conf_threshold)\n    print('Results overall: ', results_overall)\n    print('F2 Score: ', fscore)\n    print('\\n')","metadata":{"execution":{"iopub.status.busy":"2022-02-09T17:05:40.323153Z","iopub.execute_input":"2022-02-09T17:05:40.323756Z","iopub.status.idle":"2022-02-09T17:06:35.442975Z","shell.execute_reply.started":"2022-02-09T17:05:40.323716Z","shell.execute_reply":"2022-02-09T17:06:35.441523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Check the output format","metadata":{}},{"cell_type":"code","source":"for val_image_group in os.walk('fold_0/val2017'):\n    for counter, val_image in enumerate(val_image_group[2]):\n \n        val_image_path = f'fold_0/val2017/{val_image}'\n        val_image_arr = cv2.imread(val_image_path)\n        \n        experiment = Exp1280()\n\n        outputs = barrier_reef_inference(experiment, model_weights_path, val_image_arr,\n                                         conf_threshold=0.1, nms_threshold=0.4)\n\n        bboxes = outputs['bboxes']\n        scores = outputs['scores']\n\n        predictions = []\n\n        for i in range(len(bboxes)):\n            box = bboxes[i]\n            score = scores[i]\n\n            x_min = int(box[0])\n            y_min = int(box[1])\n            x_max = int(box[2])\n            y_max = int(box[3])\n\n            bbox_width = x_max - x_min\n            bbox_height = y_max - y_min\n\n            predictions.append('{:.2f} {} {} {} {}'.format(score, x_min, y_min, bbox_width, bbox_height))\n\n        prediction_str = ' '.join(predictions)\n\n        print('Prediction:', prediction_str)\n        \n        if counter == 10:\n            break","metadata":{"execution":{"iopub.status.busy":"2022-02-09T17:06:49.329333Z","iopub.execute_input":"2022-02-09T17:06:49.330202Z","iopub.status.idle":"2022-02-09T17:06:52.943452Z","shell.execute_reply.started":"2022-02-09T17:06:49.330160Z","shell.execute_reply":"2022-02-09T17:06:52.942048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Call the barrier reef api and send predictions\n\nThis last section is also adapted from [Ramek's notebook](https://www.kaggle.com/remekkinas/yolox-training-pipeline-cots-dataset-lb-0-507).  ","metadata":{}},{"cell_type":"code","source":"import greatbarrierreef\n\nenv = greatbarrierreef.make_env()   # initialize the environment\niter_test = env.iter_test() ","metadata":{"execution":{"iopub.status.busy":"2022-02-09T17:07:13.933187Z","iopub.execute_input":"2022-02-09T17:07:13.933440Z","iopub.status.idle":"2022-02-09T17:07:13.975448Z","shell.execute_reply.started":"2022-02-09T17:07:13.933409Z","shell.execute_reply":"2022-02-09T17:07:13.974763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_dict = {\n    'id': [],\n    'prediction_string': [],\n}\n\nfor (image_np, sample_prediction_df) in iter_test:\n \n    experiment = Exp1280()\n    \n    outputs = barrier_reef_inference(experiment, model_weights_path, image_np[:,:,::-1],\n                                     conf_threshold=0.1, nms_threshold=0.4)\n\n    bboxes = outputs['bboxes']\n    scores = outputs['scores']\n\n    predictions = []\n\n    for i in range(len(bboxes)):\n        box = bboxes[i]\n        score = scores[i]\n\n        x_min = int(box[0])\n        y_min = int(box[1])\n        x_max = int(box[2])\n        y_max = int(box[3])\n\n        bbox_width = x_max - x_min\n        bbox_height = y_max - y_min\n\n        predictions.append('{:.2f} {} {} {} {}'.format(score, x_min, y_min, bbox_width, bbox_height))\n\n    prediction_str = ' '.join(predictions)\n    sample_prediction_df['annotations'] = prediction_str\n    env.predict(sample_prediction_df)\n\n    print('Prediction:', prediction_str)","metadata":{"execution":{"iopub.status.busy":"2022-02-09T17:07:15.286328Z","iopub.execute_input":"2022-02-09T17:07:15.287029Z","iopub.status.idle":"2022-02-09T17:07:16.693055Z","shell.execute_reply.started":"2022-02-09T17:07:15.286990Z","shell.execute_reply":"2022-02-09T17:07:16.691498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df = pd.read_csv('submission.csv')\nsub_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-09T17:07:18.021790Z","iopub.execute_input":"2022-02-09T17:07:18.022614Z","iopub.status.idle":"2022-02-09T17:07:18.034448Z","shell.execute_reply.started":"2022-02-09T17:07:18.022554Z","shell.execute_reply":"2022-02-09T17:07:18.033446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}