{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Guessing Earlier Bounding Boxes\n\nIt seems that COTS are sometimes visible prior to the first appearance of the bounding box.  In this notebook we will try to exploit that and create more annotations in earlier frames.\n\n**Hypothesis:** we can enlarge our training data set by adding earlier bounding boxes prior to the first detected box for each COTS\n\n**Approach:**\n1. Add detections to earlier frames that have a detection in a subsequent frame as follows:\n- identify frames that have less detections than the next one\n- exclude any candidate boxes if the box has an overlap with another box in the previous frame\n- exclude the image \"margins\" to account for that some boxes may not have been visible in the frame\n- shift the bounding box by the average translation of any other matched boxes\n2. Compare some examples frames right before and right after a detection\n3. Save the results into a new training set\n\n**Results:**\nThis approach results in a small increase in number of annotations and frames with at least 1 annotation. Some of these new annotations are not very good fit (see examples below) but some seem pretty good.  I have not tried in a training pipeline but if you do please leave a comment how the new training data set did.\n\n- Previous number of bounding boxes:  11898\n- New number of boxes:  12045\n- Number of boxes increase:  147\n\n- Previous number of frames with boxes:  4919\n- New number of frames with boxes:  4963\n- Number of frames with boxes increase:  44\n\nPlease let me know if you find any bugs! ;)\n\nUnder Development:\n1. Use \"optical flow\" techniques to determine where the boxes may have come from better, fit tighter boxes, and project boxes into margins if possible. \n2. Add even more annotations by looking at even earlier frames potentially using \"reverse\" tracking\n\n# Please UPVOTE if you find this useful!  Thank you!","metadata":{}},{"cell_type":"code","source":"import cv2 as cv\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport math\n\nfrom PIL import Image, ImageDraw\nimport tensorflow as tf\n\nimport os\nimport ast\nimport sys\nimport time\n\nimport greatbarrierreef","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:28.200933Z","iopub.execute_input":"2022-02-08T07:12:28.201956Z","iopub.status.idle":"2022-02-08T07:12:28.207441Z","shell.execute_reply.started":"2022-02-08T07:12:28.201906Z","shell.execute_reply":"2022-02-08T07:12:28.206783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Parameters","metadata":{}},{"cell_type":"code","source":"IMAGE_DIM = (1280,720)\n\nEXCLUDE_MARGIN = 0.02 #we will not be adding boxes in 2% of the image width or height","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:28.208876Z","iopub.execute_input":"2022-02-08T07:12:28.209621Z","iopub.status.idle":"2022-02-08T07:12:28.222991Z","shell.execute_reply.started":"2022-02-08T07:12:28.209571Z","shell.execute_reply":"2022-02-08T07:12:28.221845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Read Data","metadata":{}},{"cell_type":"code","source":"def get_bbox(annots):\n    bboxes = [list(annot.values()) for annot in annots]\n    return bboxes\n\ndef read_data():\n    df_train = pd.read_csv('../input/tensorflow-great-barrier-reef/train.csv')\n    df_train['img_path'] = os.path.join('../input/tensorflow-great-barrier-reef/train_images')+\"/video_\"+df_train.video_id.astype(str)+\"/\"+df_train.video_frame.astype(str)+\".jpg\"\n    df_train['annotations'] = df_train['annotations'].apply(lambda x: ast.literal_eval(x))\n    df_train['bboxes'] = df_train['annotations'].apply(lambda x: get_bbox(x))\n    df_train['Number_bbox'] = df_train['annotations'].apply(lambda x:len(x)) \n    return df_train","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-02-08T07:12:28.224501Z","iopub.execute_input":"2022-02-08T07:12:28.225408Z","iopub.status.idle":"2022-02-08T07:12:28.243803Z","shell.execute_reply.started":"2022-02-08T07:12:28.225342Z","shell.execute_reply":"2022-02-08T07:12:28.242726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = read_data()","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:28.245699Z","iopub.execute_input":"2022-02-08T07:12:28.246189Z","iopub.status.idle":"2022-02-08T07:12:28.786982Z","shell.execute_reply.started":"2022-02-08T07:12:28.246132Z","shell.execute_reply":"2022-02-08T07:12:28.786039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.sample(5).head()","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:28.789551Z","iopub.execute_input":"2022-02-08T07:12:28.789830Z","iopub.status.idle":"2022-02-08T07:12:28.819914Z","shell.execute_reply.started":"2022-02-08T07:12:28.789798Z","shell.execute_reply":"2022-02-08T07:12:28.818904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Add New Bounding Boxes to Previous Frames","metadata":{"execution":{"iopub.status.busy":"2021-12-25T06:16:31.52768Z","iopub.execute_input":"2021-12-25T06:16:31.527996Z","iopub.status.idle":"2021-12-25T06:16:31.533582Z","shell.execute_reply.started":"2021-12-25T06:16:31.527956Z","shell.execute_reply":"2021-12-25T06:16:31.532223Z"}}},{"cell_type":"code","source":"#shift next annotations to previous frame\ndf_shift = df_train.shift(-1).rename(columns={'annotations':'annotations_n1',\n                                             'Number_bbox':'Number_bbox_n1',\n                                             'img_path':'img_path_n1'})\ndf_lagged = pd.concat([df_train, df_shift], axis=1)\n\n#identify frames that have less annotations then the next one\n#these are the candidates for adding earlier annotations\ndf_first_frames = df_lagged[df_lagged.Number_bbox < df_lagged.Number_bbox_n1]","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:28.821548Z","iopub.execute_input":"2022-02-08T07:12:28.821870Z","iopub.status.idle":"2022-02-08T07:12:28.861493Z","shell.execute_reply.started":"2022-02-08T07:12:28.821835Z","shell.execute_reply":"2022-02-08T07:12:28.860204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def intersects(rectangle_a, rectangle_b):\n    '''Checks for intersection of two rectangles specified as [(x1,y1),(x2,y2)]'''\n    if(rectangle_a[1][0]<rectangle_b[0][0] or rectangle_a[1][1]<rectangle_b[0][1]):\n        return False\n    elif(rectangle_a[0][0]>rectangle_b[1][0] or rectangle_a[0][1]>rectangle_b[1][1]):\n        return False\n    else:\n        return True\n        \ndef new_bboxes(prev_bboxes, next_bboxes):\n    '''Returns the bounding boxes that are deemed new in the next frame by checking \n    the centers of the bounding box in the next frame are not contained in\n    one of the previous frame bounding boxes.'''\n    new_bbs =[]\n    delta_xs = [0]\n    delta_ys = [0]\n    delta_ws = [0]\n    delta_hs = [0]\n    for bb in next_bboxes:\n        found = False\n        for prev_bb in prev_bboxes:\n            if intersects([(bb['x'],bb['y']),(bb['x'] + bb['width'],bb['y'] + bb['height'])],\n                         [(prev_bb['x'], prev_bb['y']), (prev_bb['x'] + prev_bb['width'], \n                                                         prev_bb['y'] + prev_bb['height'])]\n                         ):\n                delta_xs.append(bb['x']-prev_bb['x'])\n                delta_ys.append(bb['y']-prev_bb['y'])\n                delta_ws.append(bb['width']-prev_bb['width'])\n                delta_hs.append(bb['height']-prev_bb['height'])\n                found = True\n                break\n        if found == False:\n            #exclude margins\n            if (bb['x'] > IMAGE_DIM[0]*EXCLUDE_MARGIN) & \\\n            (bb['x'] < (IMAGE_DIM[0]-IMAGE_DIM[0]*EXCLUDE_MARGIN)) & \\\n            (bb['y'] > IMAGE_DIM[1]*EXCLUDE_MARGIN) & \\\n            (bb['y'] < (IMAGE_DIM[1]-IMAGE_DIM[1]*EXCLUDE_MARGIN)):\n                new_bb = {'x': bb['x'], 'y': bb['y'], 'width':bb['width'], 'height':bb['height']}\n                new_bbs.append(new_bb)\n                \n    #adjust bounding boxes for avergage drift\n    for b in new_bbs:        \n        delta_x_avg = sum(delta_xs)/len(delta_xs)\n        delta_y_avg = sum(delta_ys)/len(delta_ys)\n        delta_w_avg = sum(delta_ws)/len(delta_ws)\n        delta_h_avg = sum(delta_hs)/len(delta_hs)\n        b['x'] = round(b['x'] + delta_x_avg)\n        b['y'] = round(b['y'] + delta_y_avg)\n        #adjusting width/height does not seem to improve bounding box fit\n        #b['width'] = b['width'] + delta_w_avg\n        #b['height'] = b['height'] + delta_h_avg\n               \n    return new_bbs","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:28.864307Z","iopub.execute_input":"2022-02-08T07:12:28.864586Z","iopub.status.idle":"2022-02-08T07:12:28.885037Z","shell.execute_reply.started":"2022-02-08T07:12:28.864557Z","shell.execute_reply":"2022-02-08T07:12:28.884053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_first_frames['new_annotations'] = df_first_frames.apply(lambda x: \n                                                            new_bboxes(x['annotations'],\n                                                                      x['annotations_n1']),\n                                                          axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:28.886991Z","iopub.execute_input":"2022-02-08T07:12:28.887390Z","iopub.status.idle":"2022-02-08T07:12:28.929060Z","shell.execute_reply.started":"2022-02-08T07:12:28.887343Z","shell.execute_reply":"2022-02-08T07:12:28.928355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## See some examples of new annotations","metadata":{}},{"cell_type":"code","source":"def viz_new_boxes(prev_path, next_path, prev_annots, next_annots, new_annots):    \n    #previuos frame\n    print(prev_path)\n    img = Image.open(prev_path)\n    \n    for box in prev_annots:\n        shape = [box['x'], box['y'], box['x']+box['width'], box['y']+box['height']]\n        ImageDraw.Draw(img).rectangle(shape, outline =\"red\", width=3)\n\n    for box in next_annots:\n        shape = [box['x'], box['y'], box['x']+box['width'], box['y']+box['height']]\n        ImageDraw.Draw(img).rectangle(shape, outline =\"cyan\", width=3)\n\n    \n    for box in new_annots:\n        shape = [box['x'], box['y'], box['x']+box['width'], box['y']+box['height']]\n        ImageDraw.Draw(img).rectangle(shape, outline =\"yellow\", width=3)\n\n    display(img)    \n    \n    #next frame\n    print(next_path)\n    img = Image.open(next_path)\n    \n    for box in next_annots:\n        shape = [box['x'], box['y'], box['x']+box['width'], box['y']+box['height']]\n        ImageDraw.Draw(img).rectangle(shape, outline =\"red\", width=3)\n\n    #for box in new_annots:\n    #    shape = [box['x'], box['y'], box['x']+box['width'], box['y']+box['height']]\n    #    ImageDraw.Draw(img).rectangle(shape, outline =\"orange\", width=3)\n        \n    display(img)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:28.930346Z","iopub.execute_input":"2022-02-08T07:12:28.930861Z","iopub.status.idle":"2022-02-08T07:12:28.942450Z","shell.execute_reply.started":"2022-02-08T07:12:28.930804Z","shell.execute_reply":"2022-02-08T07:12:28.941700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note that we are able to add a few new bounding boxes (yellow) although the fit is not great due to camera movement.  Future development will be to leverage optical flow alogrithm or similar to improve the fit.","metadata":{}},{"cell_type":"code","source":"for index, row in df_first_frames.sample(10, random_state=12).iterrows():\n    viz_new_boxes(row.img_path,\n                  row.img_path_n1,\n                  row.annotations,\n                  row.annotations_n1,\n                  row.new_annotations)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:28.945352Z","iopub.execute_input":"2022-02-08T07:12:28.946041Z","iopub.status.idle":"2022-02-08T07:12:37.504121Z","shell.execute_reply.started":"2022-02-08T07:12:28.945989Z","shell.execute_reply":"2022-02-08T07:12:37.501618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Merge the new and existing annotations","metadata":{}},{"cell_type":"code","source":"df_train = df_train.join(df_first_frames['new_annotations'])","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:37.505643Z","iopub.execute_input":"2022-02-08T07:12:37.505997Z","iopub.status.idle":"2022-02-08T07:12:37.518988Z","shell.execute_reply.started":"2022-02-08T07:12:37.505961Z","shell.execute_reply":"2022-02-08T07:12:37.518100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['new_annotations'] = df_train['new_annotations'].fillna('[]')","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:37.520184Z","iopub.execute_input":"2022-02-08T07:12:37.520912Z","iopub.status.idle":"2022-02-08T07:12:37.531174Z","shell.execute_reply.started":"2022-02-08T07:12:37.520867Z","shell.execute_reply":"2022-02-08T07:12:37.529999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def combine_annotations(a1, a2):\n    if type(a2) == str:\n        a2 = []\n    return a1 + a2\n\ndf_train['annotations_combined'] = df_train.apply(lambda x: combine_annotations(x['annotations'],\n                                                            x['new_annotations']),\n                                                            axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:37.533624Z","iopub.execute_input":"2022-02-08T07:12:37.533988Z","iopub.status.idle":"2022-02-08T07:12:38.039487Z","shell.execute_reply.started":"2022-02-08T07:12:37.533948Z","shell.execute_reply":"2022-02-08T07:12:38.038210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.annotations_combined.iloc[34]","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:38.041401Z","iopub.execute_input":"2022-02-08T07:12:38.041816Z","iopub.status.idle":"2022-02-08T07:12:38.049656Z","shell.execute_reply.started":"2022-02-08T07:12:38.041770Z","shell.execute_reply":"2022-02-08T07:12:38.048583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Count how many new annotations we have","metadata":{}},{"cell_type":"code","source":"df_train['New_number_bbox'] = df_train['annotations_combined'].apply(lambda x: len(x))","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:38.050992Z","iopub.execute_input":"2022-02-08T07:12:38.051254Z","iopub.status.idle":"2022-02-08T07:12:38.081261Z","shell.execute_reply.started":"2022-02-08T07:12:38.051225Z","shell.execute_reply":"2022-02-08T07:12:38.080559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[df_train['New_number_bbox'] > df_train['Number_bbox']]","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:38.085296Z","iopub.execute_input":"2022-02-08T07:12:38.085738Z","iopub.status.idle":"2022-02-08T07:12:38.145011Z","shell.execute_reply.started":"2022-02-08T07:12:38.085688Z","shell.execute_reply":"2022-02-08T07:12:38.144066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prev_box_count = df_train['Number_bbox'].sum()\ncurr_box_count = df_train['New_number_bbox'].sum()\nprev_frames_with_box_count = df_train[df_train.Number_bbox >0].shape[0]\ncurr_frames_with_box_count = df_train[df_train.New_number_bbox >0].shape[0]\nprint(\"Previous number of bounding boxes: \", prev_box_count)\nprint(\"New number of boxes: \", curr_box_count)\nprint(\"Number of boxes increase: \", curr_box_count-prev_box_count)\nprint(\"Previous number of frames with boxes: \", prev_frames_with_box_count)\nprint(\"New number of frames with boxes: \", curr_frames_with_box_count)\nprint(\"Number of frames with boxes increase: \", curr_frames_with_box_count-prev_frames_with_box_count)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:38.147001Z","iopub.execute_input":"2022-02-08T07:12:38.147589Z","iopub.status.idle":"2022-02-08T07:12:38.163067Z","shell.execute_reply.started":"2022-02-08T07:12:38.147541Z","shell.execute_reply":"2022-02-08T07:12:38.161990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create a new train file","metadata":{}},{"cell_type":"code","source":"cols = pd.read_csv('../input/tensorflow-great-barrier-reef/train.csv').columns","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:38.164895Z","iopub.execute_input":"2022-02-08T07:12:38.165261Z","iopub.status.idle":"2022-02-08T07:12:38.207337Z","shell.execute_reply.started":"2022-02-08T07:12:38.165215Z","shell.execute_reply":"2022-02-08T07:12:38.206148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_new = df_train.copy()\ndf_new['annotations'] = df_train['annotations_combined']","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:38.208815Z","iopub.execute_input":"2022-02-08T07:12:38.209081Z","iopub.status.idle":"2022-02-08T07:12:38.225390Z","shell.execute_reply.started":"2022-02-08T07:12:38.209050Z","shell.execute_reply":"2022-02-08T07:12:38.224407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_new[cols].to_csv('/kaggle/working/more_annotations_train.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:38.226709Z","iopub.execute_input":"2022-02-08T07:12:38.227176Z","iopub.status.idle":"2022-02-08T07:12:38.361818Z","shell.execute_reply.started":"2022-02-08T07:12:38.227144Z","shell.execute_reply":"2022-02-08T07:12:38.360864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.read_csv('/kaggle/working/more_annotations_train.csv').head()","metadata":{"execution":{"iopub.status.busy":"2022-02-08T07:12:38.366220Z","iopub.execute_input":"2022-02-08T07:12:38.366537Z","iopub.status.idle":"2022-02-08T07:12:38.411707Z","shell.execute_reply.started":"2022-02-08T07:12:38.366506Z","shell.execute_reply":"2022-02-08T07:12:38.410904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}