{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# TensorFlow - Help Protect the Great Barrier Reef\n### Detect crown-of-thorns starfish in underwater image data\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/31703/logos/header.png?t=2021-10-29-00-30-04)","metadata":{}},{"cell_type":"markdown","source":"## Data Description:\nIn this competition, you will predict the presence and position of crown-of-thorns starfish in sequences of underwater images taken at various times and locations around the Great Barrier Reef. Predictions take the form of a bounding box together with a confidence score for each identified starfish. An image may contain zero or more starfish.\n\nThis competition uses a hidden test set that will be served by an API to ensure you evaluate the images in the same order they were recorded within each video. When your submitted notebook is scored, the actual test data (including a sample submission) will be availabe to your notebook.\n\n### Files\n>- train/ - Folder containing training set photos of the form video_{video_id}/{video_frame_number}.jpg.\n>- train/test.csv - Metadata for the images. As with other test files, most of the test metadata data is only available to your notebook upon submission. Just the first few rows available for download.\n>- video_id - ID number of the video the image was part of. The video ids are not meaningfully ordered.\n>- video_frame - The frame number of the image within the video. Expect to see occasional gaps in the frame number from when the diver surfaced.\n>- sequence - ID of a gap-free subset of a given video. The sequence ids are not meaningfully ordered.\n>- sequence_frame - The frame number within a given sequence.\n>- image_id - ID code for the image, in the format '{video_id}-{video_frame}'\n>- annotations - The bounding boxes of any starfish detections in a string format that can be evaluated directly with Python. Does not use the \nsame format as the predictions you will submit. Not available in test.csv. A bounding box is described by the pixel coordinate (x_min, y_min) of its upper left corner within the image together with its width and height in pixels.\n>- example_sample_submission.csv - A sample submission file in the correct format. The actual sample submission will be provided by the API; this is only provided to illustrate how to properly format predictions. The submission format is further described on the Evaluation page.\n>- example_test.npy - Sample data that will be served by the example API.\n>- greatbarrierreef - The image delivery API that will serve the test set pixel arrays. You may need Python 3.7 and a Linux environment to run the example offline without errors.\n\nTime-series API Details\nThe API serves the images one by one, in order by video and frame number, as pixel arrays.\n\nExpect to see roughly 13,000 images in the test set.\n\nThe API will require roughly two GB of memory after initialization. The initialization step (env.iter_test()) will require meaningfully more memory than that; we recommend you do not load your model until after making that call. The API will also consume less than ten minutes of runtime for loading and serving the data.","metadata":{}},{"cell_type":"markdown","source":"Awesome notebooks referenced:\n>- https://www.kaggle.com/andradaolteanu/greatbarrierreef-yolo-full-guide-train-infer\n>- https://www.kaggle.com/awsaf49/great-barrier-reef-yolov5-train\n>- https://www.kaggle.com/diegoalejogm/great-barrier-reefs-eda-with-animations","metadata":{}},{"cell_type":"markdown","source":"# Import packages","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport cv2\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.express as px\nimport matplotlib.patches as patches\nfrom plotly.offline import plot, iplot, init_notebook_mode\ninit_notebook_mode(connected=True)\nimport glob\nimport shutil\nimport sys\nfrom joblib import Parallel, delayed\nfrom IPython.display import display\nfrom tqdm import tqdm\nfrom matplotlib import animation, rc\nfrom PIL import Image, ImageDraw\nimport ast\nimport yaml\nimport torch\nimport gc\nfrom torchvision.ops import box_iou\n\nSEED = 42\nnp.random.seed(SEED)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:43.122243Z","iopub.execute_input":"2022-02-13T04:44:43.12309Z","iopub.status.idle":"2022-02-13T04:44:47.941064Z","shell.execute_reply.started":"2022-02-13T04:44:43.12296Z","shell.execute_reply":"2022-02-13T04:44:47.940236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load data","metadata":{}},{"cell_type":"code","source":"train_data = pd.read_csv('../input/tensorflow-great-barrier-reef/train.csv')\ntest_data = pd.read_csv('../input/tensorflow-great-barrier-reef/test.csv')\nsample_submission_data = pd.read_csv('../input/tensorflow-great-barrier-reef/example_sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:47.94278Z","iopub.execute_input":"2022-02-13T04:44:47.943027Z","iopub.status.idle":"2022-02-13T04:44:48.001966Z","shell.execute_reply.started":"2022-02-13T04:44:47.942992Z","shell.execute_reply":"2022-02-13T04:44:48.001204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA","metadata":{}},{"cell_type":"code","source":"print(train_data.shape)\ntrain_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:48.003138Z","iopub.execute_input":"2022-02-13T04:44:48.004749Z","iopub.status.idle":"2022-02-13T04:44:48.022969Z","shell.execute_reply.started":"2022-02-13T04:44:48.004718Z","shell.execute_reply":"2022-02-13T04:44:48.022192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate the number of total annotations\ntrain_data[\"no_annotations\"] = train_data[\"annotations\"].apply(lambda x: len(eval(x)))","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:48.025084Z","iopub.execute_input":"2022-02-13T04:44:48.025362Z","iopub.status.idle":"2022-02-13T04:44:48.234468Z","shell.execute_reply.started":"2022-02-13T04:44:48.025328Z","shell.execute_reply":"2022-02-13T04:44:48.233858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(test_data.shape)\ntest_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:48.23563Z","iopub.execute_input":"2022-02-13T04:44:48.235873Z","iopub.status.idle":"2022-02-13T04:44:48.249983Z","shell.execute_reply.started":"2022-02-13T04:44:48.23584Z","shell.execute_reply":"2022-02-13T04:44:48.249358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(sample_submission_data.shape)\nsample_submission_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:48.251079Z","iopub.execute_input":"2022-02-13T04:44:48.251343Z","iopub.status.idle":"2022-02-13T04:44:48.261672Z","shell.execute_reply.started":"2022-02-13T04:44:48.251309Z","shell.execute_reply":"2022-02-13T04:44:48.260973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Distribution by # images","metadata":{}},{"cell_type":"code","source":"temp = train_data[\"no_annotations\"].value_counts().reset_index().rename(columns = {'index':'# Annotations', 'no_annotations': '# Images'})\nfig = px.bar(temp, x='# Annotations', y='# Images')\nfig.update_yaxes(title=\"# Images\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:48.262911Z","iopub.execute_input":"2022-02-13T04:44:48.2644Z","iopub.status.idle":"2022-02-13T04:44:49.13882Z","shell.execute_reply.started":"2022-02-13T04:44:48.26436Z","shell.execute_reply":"2022-02-13T04:44:49.138064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp_1 = train_data.groupby(['video_id']).agg({'image_id':'nunique'}).reset_index().rename(columns = {'image_id':'# Images'})\ntemp_2 = train_data[(train_data['annotations'].apply(lambda x: len(x))>2)].groupby(['video_id']).agg({'image_id':'nunique'}).reset_index().rename(columns = {'image_id':'# Images with annotations'})\ntemp = temp_1.merge(temp_2, how = 'left', on = 'video_id')\nprint('% Images with bounding:', temp['# Images with annotations'].sum()/temp['# Images'].sum() * 100, '%')\ntemp['% Images with annotations'] = temp['# Images with annotations']/temp['# Images'] * 100\ntemp['# Images without annotations'] = temp['# Images'] - temp['# Images with annotations']\nfig = px.bar(temp, x='video_id', y=['# Images without annotations', '# Images with annotations'])\nfig.update_yaxes(title=\"# Images\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:49.140185Z","iopub.execute_input":"2022-02-13T04:44:49.140448Z","iopub.status.idle":"2022-02-13T04:44:49.243887Z","shell.execute_reply.started":"2022-02-13T04:44:49.140413Z","shell.execute_reply":"2022-02-13T04:44:49.24308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp_1 = train_data.groupby(['video_id', 'sequence']).agg({'image_id':'nunique'}).reset_index().rename(columns = {'image_id':'# Images'})\ntemp_2 = train_data[(train_data['annotations'].apply(lambda x: len(x))>2)].groupby(['video_id', 'sequence']).agg({'image_id':'nunique'}).reset_index().rename(columns = {'image_id':'# Images with annotations'})\ntemp_2.fillna(0, inplace = True)\ntemp = temp_1.merge(temp_2, how = 'left', on = ['video_id', 'sequence'])\ntemp['% Images with annotations'] = temp['# Images with annotations']/temp['# Images'] * 100\ntemp.sort_values(by = ['% Images with annotations'], ascending = False)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:49.245228Z","iopub.execute_input":"2022-02-13T04:44:49.245488Z","iopub.status.idle":"2022-02-13T04:44:49.292967Z","shell.execute_reply.started":"2022-02-13T04:44:49.245453Z","shell.execute_reply":"2022-02-13T04:44:49.292297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Check if all images for a video are present in the input directory","metadata":{}},{"cell_type":"code","source":"for i in train_data['video_id'].unique().tolist():\n    all_images_for_video_id = [int(x.split('.jpg')[0]) for x in os.listdir(f'../input/tensorflow-great-barrier-reef/train_images/video_{i}')]\n    print('\\n Number of images in train.csv for video id', i, ':', train_data[train_data['video_id']==i].shape[0])\n    print('\\n Number of images in train folder for video id', i, ':', len(all_images_for_video_id))\n    all_images_for_video_id_in_train_csv = train_data[train_data['video_id']==i]['video_frame'].tolist()\n    all_images_for_video_id_in_train_csv_not_in_train_folder = [x for x in all_images_for_video_id_in_train_csv if x not in all_images_for_video_id]\n    print('\\n Number of images in train.csv but not in train folder for video id', i, ':', len(all_images_for_video_id_in_train_csv_not_in_train_folder))\n    all_images_for_video_id_in_train_folder_not_in_train_csv = [x for x in all_images_for_video_id if x not in all_images_for_video_id_in_train_csv]\n    print('\\n Number of images in train folder but not in train csv for video id', i, ':', len(all_images_for_video_id_in_train_folder_not_in_train_csv))\n    print('---'*50)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:49.29597Z","iopub.execute_input":"2022-02-13T04:44:49.296173Z","iopub.status.idle":"2022-02-13T04:44:52.341348Z","shell.execute_reply.started":"2022-02-13T04:44:49.296133Z","shell.execute_reply":"2022-02-13T04:44:52.34053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## See sample images","metadata":{}},{"cell_type":"code","source":"video_id = 0\ntemp = train_data[(train_data['video_id']==video_id) & (train_data['annotations'].apply(lambda x: len(x))>2)].reset_index(drop = True)\ntemp.tail()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:52.342688Z","iopub.execute_input":"2022-02-13T04:44:52.343114Z","iopub.status.idle":"2022-02-13T04:44:52.369123Z","shell.execute_reply.started":"2022-02-13T04:44:52.343076Z","shell.execute_reply":"2022-02-13T04:44:52.36837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sequence_id = 996\ntemp[temp['sequence']==sequence_id].head()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:52.370196Z","iopub.execute_input":"2022-02-13T04:44:52.370441Z","iopub.status.idle":"2022-02-13T04:44:52.382808Z","shell.execute_reply.started":"2022-02-13T04:44:52.370407Z","shell.execute_reply":"2022-02-13T04:44:52.381928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp[temp['no_annotations']>4].head()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:52.384035Z","iopub.execute_input":"2022-02-13T04:44:52.384361Z","iopub.status.idle":"2022-02-13T04:44:52.39763Z","shell.execute_reply.started":"2022-02-13T04:44:52.384323Z","shell.execute_reply":"2022-02-13T04:44:52.396908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"OpenCV imread, imwrite and imshow indeed all work with the BGR order, so there is no need to change the order when you read an image with cv2.imread and then want to show it with cv2.imshow.\n\nWhile BGR is used consistently throughout OpenCV, most other image processing libraries use the RGB ordering. If you want to use matplotlib's imshow but read the image with OpenCV, you would need to convert from BGR to RGB.","metadata":{}},{"cell_type":"code","source":"video_frame = 9651\n\nimg = cv2.imread('../input/tensorflow-great-barrier-reef/train_images/' + 'video_' + str(video_id) + '/' + str(video_frame) + '.jpg') \nimg = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n\nannot = train_data[(train_data['video_id']==video_id) & (train_data['video_frame']==video_frame)]['annotations'].iloc[0]\n\n# Output img with window name as 'image'\n# Source: https://www.kaggle.com/andradaolteanu/greatbarrierreef-full-guide-to-bboxaugmentation\n\nfig, axs = plt.subplots(figsize=(23, 8))\n\naxs.imshow(img)\n\nfor a in eval(annot):\n    rect = patches.Rectangle((a[\"x\"], a[\"y\"]), a[\"width\"], a[\"height\"], \n                             linewidth=3, edgecolor=\"#FF6103\", facecolor='none')\n    axs.add_patch(rect)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:52.399067Z","iopub.execute_input":"2022-02-13T04:44:52.399394Z","iopub.status.idle":"2022-02-13T04:44:53.061633Z","shell.execute_reply.started":"2022-02-13T04:44:52.399359Z","shell.execute_reply":"2022-02-13T04:44:53.061019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Animation","metadata":{}},{"cell_type":"code","source":"train_data['annotations'] = train_data['annotations'].apply(eval)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:53.062527Z","iopub.execute_input":"2022-02-13T04:44:53.062733Z","iopub.status.idle":"2022-02-13T04:44:53.278501Z","shell.execute_reply.started":"2022-02-13T04:44:53.062705Z","shell.execute_reply":"2022-02-13T04:44:53.277844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rc('animation', html='jshtml')\n\nv_id = 0\n# num_images = train_data[train_data['video_id']==v_id]['video_frame'].tolist()\nnum_images = train_data[(train_data['video_id']==v_id) & (train_data['no_annotations']>0)]['video_frame'].tolist() # Only take images with annotations\n# num_images = num_images[:int(len(num_images)/3)] # Take only a fraction of the images in a video\n\ndef fetch_image_list(df_tmp, video_id, num_images, start_frame_idx):\n    def fetch_image(frame_id):\n        path_base = '/kaggle/input/tensorflow-great-barrier-reef/train_images/video_{}/{}.jpg'\n        raw_img = Image.open(path_base.format(video_id, frame_id))\n\n        row_frame = df_tmp[(df_tmp.video_id == video_id) & (df_tmp.video_frame == frame_id)].iloc[0]\n        bounding_boxes = row_frame.annotations\n\n        for box in bounding_boxes:\n            draw = ImageDraw.Draw(raw_img)\n            x0, y0, x1, y1 = (box['x'], box['y'], box['x']+box['width'], box['y']+box['height'])\n            draw.rectangle((x0, y0, x1, y1), outline=180, width=3)\n        return raw_img\n\n    return [np.array(fetch_image(index)) for index in num_images]\n\nimages = fetch_image_list(train_data, video_id = v_id, num_images = num_images, start_frame_idx = 0)\n\ndef create_animation(ims):\n    fig = plt.figure(figsize=(9, 9))\n    plt.axis('off')\n    im = plt.imshow(ims[0])\n\n    def animate_func(i):\n        im.set_array(ims[i])\n        return [im]\n\n    return animation.FuncAnimation(fig, animate_func, frames = len(ims), interval = 1000//12)\n\ncreate_animation(images)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:44:53.280223Z","iopub.execute_input":"2022-02-13T04:44:53.280697Z","iopub.status.idle":"2022-02-13T04:47:55.933454Z","shell.execute_reply.started":"2022-02-13T04:44:53.280658Z","shell.execute_reply":"2022-02-13T04:47:55.932121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del images","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:47:55.935271Z","iopub.execute_input":"2022-02-13T04:47:55.935534Z","iopub.status.idle":"2022-02-13T04:47:55.941419Z","shell.execute_reply.started":"2022-02-13T04:47:55.935501Z","shell.execute_reply":"2022-02-13T04:47:55.940009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Modelling","metadata":{}},{"cell_type":"markdown","source":"## YOLOv5","metadata":{}},{"cell_type":"markdown","source":"https://machinelearningknowledge.ai/introduction-to-yolov5-object-detection-with-tutorial/#Introduction","metadata":{}},{"cell_type":"markdown","source":"![](https://machinelearningknowledge.ai/ezoimgfmt/953894.smushcdn.com/2611031/wp-content/uploads/2021/06/YOLOv5-Architecture.jpg?lossy=0&strip=1&webp=1&ezimgfmt=rs:696x519/rscb1/ng:webp/ngcb1)","metadata":{}},{"cell_type":"markdown","source":"### Data preprocessing","metadata":{}},{"cell_type":"code","source":"IMAGE_DIR = '/kaggle/images' # directory to save images\nLABEL_DIR = '/kaggle/labels' # directory to save labels\nREMOVE_NOBBOX = True","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:47:55.943055Z","iopub.execute_input":"2022-02-13T04:47:55.943351Z","iopub.status.idle":"2022-02-13T04:47:55.951943Z","shell.execute_reply.started":"2022-02-13T04:47:55.943315Z","shell.execute_reply":"2022-02-13T04:47:55.951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data['old_image_path'] = train_data.apply(lambda x: '../input/tensorflow-great-barrier-reef/train_images/' + 'video_' + str(x['video_id']) + '/' + str(x['video_frame']) + '.jpg', axis = 1)\ntrain_data['image_path']  = f'{IMAGE_DIR}/'+ train_data.image_id+'.jpg'\ntrain_data['label_path']  = f'{LABEL_DIR}/'+ train_data.image_id+'.txt'","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:47:55.95334Z","iopub.execute_input":"2022-02-13T04:47:55.953631Z","iopub.status.idle":"2022-02-13T04:47:56.724769Z","shell.execute_reply.started":"2022-02-13T04:47:55.953597Z","shell.execute_reply":"2022-02-13T04:47:56.724053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = train_data.copy()\n\nif REMOVE_NOBBOX:\n    df = df[df['no_annotations']>0].reset_index(drop = True)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:47:56.725864Z","iopub.execute_input":"2022-02-13T04:47:56.726126Z","iopub.status.idle":"2022-02-13T04:47:56.73852Z","shell.execute_reply.started":"2022-02-13T04:47:56.726092Z","shell.execute_reply":"2022-02-13T04:47:56.737702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(df.shape)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:47:56.739935Z","iopub.execute_input":"2022-02-13T04:47:56.740287Z","iopub.status.idle":"2022-02-13T04:47:56.76184Z","shell.execute_reply.started":"2022-02-13T04:47:56.740246Z","shell.execute_reply":"2022-02-13T04:47:56.761046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# YOLOv5 requires write access\n!mkdir -p {IMAGE_DIR}\n!mkdir -p {LABEL_DIR}\n\ndef make_copy(row):\n    shutil.copyfile(row.old_image_path, row.image_path)\n    return\n\nimage_paths = df.old_image_path.tolist()\n_ = Parallel(n_jobs=-1, backend='threading')(delayed(make_copy)(row) for _, row in tqdm(df.iterrows(), total=len(df)))","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:47:56.763241Z","iopub.execute_input":"2022-02-13T04:47:56.763554Z","iopub.status.idle":"2022-02-13T04:48:18.951587Z","shell.execute_reply.started":"2022-02-13T04:47:56.763517Z","shell.execute_reply":"2022-02-13T04:48:18.950548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['width'] = 1280\ndf['height'] = 720","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:18.953221Z","iopub.execute_input":"2022-02-13T04:48:18.953512Z","iopub.status.idle":"2022-02-13T04:48:18.961249Z","shell.execute_reply.started":"2022-02-13T04:48:18.953474Z","shell.execute_reply":"2022-02-13T04:48:18.960531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_bbox(annots):\n    bboxes = [list(annot.values()) for annot in annots]\n    return bboxes\n\ndef get_imgsize(row):\n    row['width'], row['height'] = imagesize.get(row['image_path'])\n    return row\n\ndf['bboxes'] = df.annotations.apply(get_bbox)\ndf.head(2)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:18.962743Z","iopub.execute_input":"2022-02-13T04:48:18.963029Z","iopub.status.idle":"2022-02-13T04:48:18.99321Z","shell.execute_reply.started":"2022-02-13T04:48:18.96299Z","shell.execute_reply":"2022-02-13T04:48:18.992364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Required data structure","metadata":{}},{"cell_type":"markdown","source":"Label structure:\n    \n>- COCO: [xmin, ymin, w, h]\n>- VOC: [xmin, ymin, xmax, ymax]\n>- YOLO input: [xmid, ymid, w, h] (normalized)\n>- YOLO output: [xmin, ymin, xmax, ymax] --> VOC","metadata":{}},{"cell_type":"markdown","source":"We need to export our labels to YOLO format, with one *.txt file per image (if no objects in image, no *.txt file is required). \n\nThe txt file specifications are:\n>- One row per object\n>- Each row is class [x_center, y_center, width, height] format.\n>- Box coordinates must be in normalized xywh format (from 0 - 1). If your boxes are in pixels, divide x_center and width by image width, and y_center and height by image height.\n>- Class numbers are zero-indexed (start from 0).\n\nCompetition bbox format is COCO hence [x_min, y_min, width, height]. So, we need to convert form COCO to YOLO format.\n\n![](https://editor.analyticsvidhya.com/uploads/95552Screenshot%202021-08-25%20at%2012.06.06%20AM.png)","metadata":{"execution":{"iopub.status.busy":"2022-01-30T09:30:37.679519Z","iopub.execute_input":"2022-01-30T09:30:37.679837Z","iopub.status.idle":"2022-01-30T09:30:38.449321Z","shell.execute_reply.started":"2022-01-30T09:30:37.679806Z","shell.execute_reply":"2022-01-30T09:30:38.448325Z"}}},{"cell_type":"code","source":"# Helper functions\ndef voc2yolo(bboxes, image_height=720, image_width=1280):\n    \"\"\"\n    voc  => [x1, y1, x2, y1]\n    yolo => [xmid, ymid, w, h] (normalized)\n    \"\"\"\n    \n    bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n    \n    bboxes[..., [0, 2]] = bboxes[..., [0, 2]]/ image_width\n    bboxes[..., [1, 3]] = bboxes[..., [1, 3]]/ image_height\n    \n    w = bboxes[..., 2] - bboxes[..., 0]\n    h = bboxes[..., 3] - bboxes[..., 1]\n    \n    bboxes[..., 0] = bboxes[..., 0] + w/2\n    bboxes[..., 1] = bboxes[..., 1] + h/2\n    bboxes[..., 2] = w\n    bboxes[..., 3] = h\n    \n    return bboxes\n\ndef yolo2voc(bboxes, image_height=720, image_width=1280):\n    \"\"\"\n    yolo => [xmid, ymid, w, h] (normalized)\n    voc  => [x1, y1, x2, y1]\n    \n    \"\"\" \n    bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n    \n    bboxes[..., [0, 2]] = bboxes[..., [0, 2]]* image_width\n    bboxes[..., [1, 3]] = bboxes[..., [1, 3]]* image_height\n    \n    bboxes[..., [0, 1]] = bboxes[..., [0, 1]] - bboxes[..., [2, 3]]/2\n    bboxes[..., [2, 3]] = bboxes[..., [0, 1]] + bboxes[..., [2, 3]]\n    \n    return bboxes\n\ndef coco2yolo(bboxes, image_height=720, image_width=1280):\n    \"\"\"\n    coco => [xmin, ymin, w, h]\n    yolo => [xmid, ymid, w, h] (normalized)\n    \"\"\"\n    \n    bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n    \n    # normolizinig\n    bboxes[..., [0, 2]]= bboxes[..., [0, 2]]/ image_width\n    bboxes[..., [1, 3]]= bboxes[..., [1, 3]]/ image_height\n    \n    # convertion (xmin, ymin) => (xmid, ymid)\n    bboxes[..., [0, 1]] = bboxes[..., [0, 1]] + bboxes[..., [2, 3]]/2\n    \n    return bboxes\n\ndef yolo2coco(bboxes, image_height=720, image_width=1280):\n    \"\"\"\n    yolo => [xmid, ymid, w, h] (normalized)\n    coco => [xmin, ymin, w, h]\n    \n    \"\"\" \n    bboxes = bboxes.copy().astype(float) # otherwise all value will be 0 as voc_pascal dtype is np.int\n    \n    # denormalizing\n    bboxes[..., [0, 2]]= bboxes[..., [0, 2]]* image_width\n    bboxes[..., [1, 3]]= bboxes[..., [1, 3]]* image_height\n    \n    # converstion (xmid, ymid) => (xmin, ymin) \n    bboxes[..., [0, 1]] = bboxes[..., [0, 1]] - bboxes[..., [2, 3]]/2\n    \n    return bboxes\n\ndef voc2coco(bboxes, image_height=720, image_width=1280):\n    bboxes  = voc2yolo(bboxes, image_height, image_width)\n    bboxes  = yolo2coco(bboxes, image_height, image_width)\n    return bboxes\n\ndef load_image(image_path):\n    return cv2.cvtColor(cv2.imread(image_path), cv2.COLOR_BGR2RGB)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:18.994703Z","iopub.execute_input":"2022-02-13T04:48:18.995004Z","iopub.status.idle":"2022-02-13T04:48:19.014074Z","shell.execute_reply.started":"2022-02-13T04:48:18.994947Z","shell.execute_reply":"2022-02-13T04:48:19.01314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cnt = 0\nall_bboxes = []\nbboxes_info = []\nfor row_idx in tqdm(range(df.shape[0])):\n    row = df.iloc[row_idx]\n    image_height = row.height\n    image_width  = row.width\n    bboxes_coco  = np.array(row.bboxes).astype(np.float32).copy()\n    num_bbox     = len(bboxes_coco)\n    names        = ['cots']*num_bbox\n    labels       = np.array([0]*num_bbox)[..., None].astype(str)\n    ## Create Annotation(YOLO)\n    with open(row.label_path, 'w') as f:\n        if num_bbox<1:\n            annot = ''\n            f.write(annot)\n            cnt+=1\n            continue\n        \n        bboxes_yolo  = coco2yolo(bboxes_coco, image_height, image_width)\n        all_bboxes.append(bboxes_yolo.astype(float))\n        annots = []\n        \n        # Write annotations in file\n        for i in range(num_bbox):\n            annot = [\"0\"] + \\\n                    bboxes_yolo[i].astype(str).tolist()\n            \n            annots.append(annot)\n        string = '\\n'.join([' '.join(annot) for annot in annots])\n        f.write(string.strip())\n            \ndf[\"yolo_bbox\"] = all_bboxes\nprint('Missing:',cnt)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:19.015414Z","iopub.execute_input":"2022-02-13T04:48:19.015893Z","iopub.status.idle":"2022-02-13T04:48:21.007086Z","shell.execute_reply.started":"2022-02-13T04:48:19.015792Z","shell.execute_reply":"2022-02-13T04:48:21.006413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's read the files\nf1 = open('/kaggle/labels/0-8948.txt', 'r')\nf1.read()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:21.008237Z","iopub.execute_input":"2022-02-13T04:48:21.008659Z","iopub.status.idle":"2022-02-13T04:48:21.017254Z","shell.execute_reply.started":"2022-02-13T04:48:21.00862Z","shell.execute_reply":"2022-02-13T04:48:21.016579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Total # images:', len(os.listdir(\"/kaggle/images\")))","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:21.023124Z","iopub.execute_input":"2022-02-13T04:48:21.023843Z","iopub.status.idle":"2022-02-13T04:48:21.035854Z","shell.execute_reply.started":"2022-02-13T04:48:21.023808Z","shell.execute_reply":"2022-02-13T04:48:21.034463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Retrieve a sample of data\ni = 1008\nimages = os.listdir(\"/kaggle/images\")[i:i+6]\nvid_id = [im.split(\"-\")[0] for im in images]\nseq_id = [im.split(\"-\")[1].split(\".\")[0] for im in images]\n\n# Plot\nfig, axs = plt.subplots(2, 3, figsize=(23, 10))\naxs = axs.flatten()\nfig.suptitle(f\"Sample of images and YOLO bounding boxes\", fontsize = 20)\n\nfor k in range(6):\n    \n    # Get the data\n    im = cv2.imread(f\"/kaggle/images/{images[k]}\")\n    im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)\n    dh, dw, _ = im.shape\n    txt = open(f\"/kaggle/labels/{vid_id[k]}-{seq_id[k]}.txt\", \"r\").read().split('\\n')\n    txt = ' '.join(txt).split(' ')[1:]\n    no_boxes = int((len(txt)+1)/5)\n    \n    # Draw boxes\n    i = 0\n    num_box = 0\n    while num_box < no_boxes:\n        i = i+4\n        num_box += 1\n        box = txt[:i][-4:]\n        i = i + 1\n        \n        # Src: https://github.com/pjreddie/darknet/blob/810d7f797bdb2f021dbe65d2524c2ff6b8ab5c8b/src/image.c#L283-L291\n        # from YOLO to COCO\n        x, y, w, h = box\n        x, y, w, h = float(x), float(y), float(w), float(h)\n\n        l = int((x - w / 2) * dw)\n        r = int((x + w / 2) * dw)\n        t = int((y - h / 2) * dh)\n        b = int((y + h / 2) * dh)\n\n        if l < 0: l = 0\n        if r > dw - 1: r = dw - 1\n        if t < 0: t = 0\n        if b > dh - 1: b = dh - 1\n\n        cv2.rectangle(im, (l, t), (r, b), (255,0,0), 3)\n        \n    # Show image with bboxes\n    axs[k].set_title(f\"Sample {k}\", fontsize = 14)\n    axs[k].imshow(im)\n    axs[k].set_axis_off()\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:21.038257Z","iopub.execute_input":"2022-02-13T04:48:21.038692Z","iopub.status.idle":"2022-02-13T04:48:22.920635Z","shell.execute_reply.started":"2022-02-13T04:48:21.038657Z","shell.execute_reply":"2022-02-13T04:48:22.920001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### YOLO configuration and training","metadata":{}},{"cell_type":"code","source":"# Video IDs 0 and 1 have the highest # annotated images. Hence, taking those for training. However, validation is somwwhat low.\ntrain_ids = [0,2]\nval_ids = [0]\n\n# Get path to images & labels\ntrain_images = list(df[df['video_id'].isin(train_ids)][\"image_path\"])\ntrain_labels = list(df[df['video_id'].isin(train_ids)][\"label_path\"])\n\nval_images = list(df[df['video_id'].isin(val_ids)][\"image_path\"])\nval_labels = list(df[df['video_id'].isin(val_ids)][\"label_path\"])\n\nprint(\"Train Length:\", len(df[df['video_id'].isin(train_ids)]), \"\\n\" +\n      \"Test Length:\", len(df[df['video_id'].isin(val_ids)]))","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:22.921856Z","iopub.execute_input":"2022-02-13T04:48:22.922592Z","iopub.status.idle":"2022-02-13T04:48:22.947446Z","shell.execute_reply.started":"2022-02-13T04:48:22.922553Z","shell.execute_reply":"2022-02-13T04:48:22.946602Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create train and test path data\nwith open(\"../working/train_images.txt\", \"w\") as file:\n    for path in train_images:\n        file.write(path + \"\\n\")\n        \nwith open(\"../working/val_images.txt\", \"w\") as file:\n    for path in val_images:\n        file.write(path + \"\\n\")\n\n# Create configuration\nconfig = {'path': '/kaggle/working',\n          'train': '/kaggle/working/train_images.txt',\n          'val': '/kaggle/working/val_images.txt',\n          'nc': 1,\n          'names': ['cots']}\n\nwith open(\"../working/cots.yaml\", \"w\") as file:\n    yaml.dump(config, file, default_flow_style=False)\n\n        \nprint(\"../working AFTER:\", os.listdir(\"../working\"))","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:22.949075Z","iopub.execute_input":"2022-02-13T04:48:22.949737Z","iopub.status.idle":"2022-02-13T04:48:23.321247Z","shell.execute_reply.started":"2022-02-13T04:48:22.949696Z","shell.execute_reply":"2022-02-13T04:48:23.320415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ---> YOLOv5 install <---\n%cd /kaggle/working     \n!cp -r /kaggle/input/yolov5-lib-ds /kaggle/working/yolov5     \n%cd yolov5     \n%pip install -qr requirements.txt   \n\nfrom yolov5 import utils\ndisplay = utils.notebook_init()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:52:10.108186Z","iopub.execute_input":"2022-02-13T04:52:10.108844Z","iopub.status.idle":"2022-02-13T04:52:20.99121Z","shell.execute_reply.started":"2022-02-13T04:52:10.108808Z","shell.execute_reply":"2022-02-13T04:52:20.990145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%writefile /kaggle/working/hyp.yaml\nlr0: 0.01  # initial learning rate (SGD=1E-2, Adam=1E-3)\nlrf: 0.1  # final OneCycleLR learning rate (lr0 * lrf)\nmomentum: 0.937  # SGD momentum/Adam beta1\nweight_decay: 0.0005  # optimizer weight decay 5e-4\nwarmup_epochs: 3.0  # warmup epochs (fractions ok)\nwarmup_momentum: 0.8  # warmup initial momentum\nwarmup_bias_lr: 0.1  # warmup initial bias lr\nbox: 0.05  # box loss gain\ncls: 0.5  # cls loss gain\ncls_pw: 1.0  # cls BCELoss positive_weight\nobj: 1.0  # obj loss gain (scale with pixels)\nobj_pw: 1.0  # obj BCELoss positive_weight\niou_t: 0.20  # IoU training threshold\nanchor_t: 4.0  # anchor-multiple threshold\n# anchors: 3  # anchors per output layer (0 to ignore)\nfl_gamma: 0.0  # focal loss gamma (efficientDet default gamma=1.5)\nhsv_h: 0.015  # image HSV-Hue augmentation (fraction)\nhsv_s: 0.7  # image HSV-Saturation augmentation (fraction)\nhsv_v: 0.4  # image HSV-Value augmentation (fraction)\ndegrees: 0.0  # image rotation (+/- deg)\ntranslate: 0.10  # image translation (+/- fraction)\nscale: 0.5  # image scale (+/- gain)\nshear: 0.0  # image shear (+/- deg)\nperspective: 0.0  # image perspective (+/- fraction), range 0-0.001\nflipud: 0.5  # image flip up-down (probability)\nfliplr: 0.5  # image flip left-right (probability)\nmosaic: 0.5  # image mosaic (probability)\nmixup: 0.5 # image mixup (probability)\ncopy_paste: 0.0  # segment copy-paste (probability)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:52:20.993789Z","iopub.execute_input":"2022-02-13T04:52:20.994295Z","iopub.status.idle":"2022-02-13T04:52:21.00277Z","shell.execute_reply.started":"2022-02-13T04:52:20.994237Z","shell.execute_reply":"2022-02-13T04:52:21.001983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MODEL     = 'yolov5s'\nBATCH     = 1\nEPOCHS    = 2\nOPTMIZER  = 'Adam'\n\nPROJECT   = 'great-barrier-reef-public' # w&b in yolov5\n\nREMOVE_NOBBOX = True # remove images with no bbox\nROOT_DIR  = '/kaggle/input/tensorflow-great-barrier-reef/'\n\nSIZE = 4000\nWORKERS = 1\nRUN_NAME = f\"{MODEL}_size{SIZE}_epochs{EPOCHS}_batch{BATCH}_simple\"","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:52:21.004356Z","iopub.execute_input":"2022-02-13T04:52:21.004656Z","iopub.status.idle":"2022-02-13T04:52:21.016075Z","shell.execute_reply.started":"2022-02-13T04:52:21.004616Z","shell.execute_reply":"2022-02-13T04:52:21.015182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Training\n!python train.py --img {SIZE}\\\n                --batch {BATCH}\\\n                --epochs {EPOCHS}\\\n                --optimizer {OPTMIZER}\\\n                --data /kaggle/working/cots.yaml\\\n                --hyp /kaggle/working/hyp.yaml\\\n                --weights ../../input/yolo-5s6/best.pt\\\n                --workers {WORKERS}\\\n                --project {PROJECT}\\\n                --name {RUN_NAME}\\\n                --exist-ok","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:54:32.599699Z","iopub.execute_input":"2022-02-13T04:54:32.599998Z","iopub.status.idle":"2022-02-13T04:58:04.954941Z","shell.execute_reply.started":"2022-02-13T04:54:32.599965Z","shell.execute_reply":"2022-02-13T04:58:04.953919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Check model performance","metadata":{}},{"cell_type":"code","source":"IMG_SIZE  = 10000 \nCONF      = 0.275\nIOU       = 0.2","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.341577Z","iopub.status.idle":"2022-02-13T04:48:23.342169Z","shell.execute_reply.started":"2022-02-13T04:48:23.341908Z","shell.execute_reply":"2022-02-13T04:48:23.341935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"OUTPUT_DIR = '{}/{}'.format(PROJECT, RUN_NAME)\n!ls {OUTPUT_DIR}","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.343304Z","iopub.status.idle":"2022-02-13T04:48:23.343888Z","shell.execute_reply.started":"2022-02-13T04:48:23.343638Z","shell.execute_reply":"2022-02-13T04:48:23.343664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,10))\nplt.axis('off')\nplt.imshow(plt.imread(f'{OUTPUT_DIR}/F1_curve.png'));","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.344987Z","iopub.status.idle":"2022-02-13T04:48:23.345579Z","shell.execute_reply.started":"2022-02-13T04:48:23.345338Z","shell.execute_reply":"2022-02-13T04:48:23.345364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,10))\nplt.axis('off')\nplt.imshow(plt.imread(f'{OUTPUT_DIR}/confusion_matrix.png'));","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.3467Z","iopub.status.idle":"2022-02-13T04:48:23.347306Z","shell.execute_reply.started":"2022-02-13T04:48:23.347038Z","shell.execute_reply":"2022-02-13T04:48:23.347065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for metric in ['F1', 'PR', 'P', 'R']:\n    print(f'Metric: {metric}')\n    plt.figure(figsize=(12,10))\n    plt.axis('off')\n    plt.imshow(plt.imread(f'{OUTPUT_DIR}/{metric}_curve.png'));\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.348431Z","iopub.status.idle":"2022-02-13T04:48:23.34901Z","shell.execute_reply.started":"2022-02-13T04:48:23.348764Z","shell.execute_reply":"2022-02-13T04:48:23.348792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(3, 2, figsize = (2*9,3*5), constrained_layout = True)\nfor row in range(3):\n    ax[row][0].imshow(plt.imread(f'{OUTPUT_DIR}/val_batch{row}_labels.jpg'))\n    ax[row][0].set_xticks([])\n    ax[row][0].set_yticks([])\n    ax[row][0].set_title(f'{OUTPUT_DIR}/val_batch{row}_labels.jpg', fontsize = 12)\n    \n    ax[row][1].imshow(plt.imread(f'{OUTPUT_DIR}/val_batch{row}_pred.jpg'))\n    ax[row][1].set_xticks([])\n    ax[row][1].set_yticks([])\n    ax[row][1].set_title(f'{OUTPUT_DIR}/val_batch{row}_pred.jpg', fontsize = 12)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.350137Z","iopub.status.idle":"2022-02-13T04:48:23.350742Z","shell.execute_reply.started":"2022-02-13T04:48:23.350497Z","shell.execute_reply":"2022-02-13T04:48:23.350523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Change our position within the directory back\n%cd /kaggle/working\n\n# --- Trained Model ---\nMODEL_PATH = f'./yolov5/great-barrier-reef-public/{RUN_NAME}/weights/best.pt'\n# MODEL_PATH ='../input/yolo-5s6/best.pt'\n\n# Load the model\nmodel = torch.hub.load(\"../input/yolov5-lib-ds\", \"custom\",\n                       path=MODEL_PATH,\n                       source='local', force_reload=True)\n\n# BoundingBox Confidence\nmodel.conf = CONF\n# Intersection Over Union\nmodel.iou = IOU\n\nmodel.classes = None   # (optional list) filter by class, i.e. = [0, 15, 16] for persons, cats and dogs\nmodel.multi_label = False  # NMS multiple labels per box\nmodel.max_det = 1000  # maximum number of detections per image","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.351865Z","iopub.status.idle":"2022-02-13T04:48:23.352461Z","shell.execute_reply.started":"2022-02-13T04:48:23.35221Z","shell.execute_reply":"2022-02-13T04:48:23.352236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Helper functions\n# Source: https://github.com/awsaf49/bbox/blob/main/bbox/utils.py \n\ndef plot_one_box(x, img, color=None, label=None, line_thickness=None):\n    # Plots one bounding box on image img\n    tl = line_thickness or round(0.002 * (img.shape[0] + img.shape[1]) / 2) + 1  # line/font thickness\n    color = color or [random.randint(0, 255) for _ in range(3)]\n    c1, c2 = (int(x[0]), int(x[1])), (int(x[2]), int(x[3]))\n    cv2.rectangle(img, c1, c2, color, thickness=tl, lineType=cv2.LINE_AA)\n    if label:\n        tf = max(tl - 1, 1)  # font thickness\n        t_size = cv2.getTextSize(label, 0, fontScale=tl / 3, thickness=tf)[0]\n        c2 = c1[0] + t_size[0], c1[1] - t_size[1] - 3\n        cv2.rectangle(img, c1, c2, color, -1, cv2.LINE_AA)  # filled\n        cv2.putText(img, label, (c1[0], c1[1] - 2), 0, tl / 3, [225, 255, 255], thickness=tf, lineType=cv2.LINE_AA)\n        \ndef draw_bboxes(img, bboxes, classes, class_ids, colors = None, show_classes = None, bbox_format = 'yolo', class_name = False, line_thickness = 2):  \n     \n    image = img.copy()\n    show_classes = classes if show_classes is None else show_classes\n    colors = (255, 0 ,0) if colors is None else colors\n    \n    if bbox_format == 'yolo':\n        \n        for idx in range(len(bboxes)):  \n            \n            bbox  = bboxes[idx]\n            cls   = classes[idx]\n            cls_id = class_ids[idx]\n            color = colors[cls_id] if type(colors) is list else colors\n            \n            if cls in show_classes:\n            \n                x1 = round(float(bbox[0])*image.shape[1])\n                y1 = round(float(bbox[1])*image.shape[0])\n                w  = round(float(bbox[2])*image.shape[1]/2) #w/2 \n                h  = round(float(bbox[3])*image.shape[0]/2)\n\n                voc_bbox = (x1-w, y1-h, x1+w, y1+h)\n                plot_one_box(voc_bbox, \n                             image,\n                             color = color,\n                             label = cls if class_name else str(get_label(cls)),\n                             line_thickness = line_thickness)\n            \n    elif bbox_format == 'coco':\n        \n        for idx in range(len(bboxes)):  \n            \n            bbox  = bboxes[idx]\n            cls   = classes[idx]\n            cls_id = class_ids[idx]\n            color = colors[cls_id] if type(colors) is list else colors\n            \n            if cls in show_classes:            \n                x1 = int(round(bbox[0]))\n                y1 = int(round(bbox[1]))\n                w  = int(round(bbox[2]))\n                h  = int(round(bbox[3]))\n\n                voc_bbox = (x1, y1, x1+w, y1+h)\n                plot_one_box(voc_bbox, \n                             image,\n                             color = color,\n                             label = cls if class_name else str(cls_id),\n                             line_thickness = line_thickness)\n\n    elif bbox_format == 'voc':\n        \n        for idx in range(len(bboxes)):  \n            \n            bbox  = bboxes[idx]\n            cls   = classes[idx]\n            cls_id = class_ids[idx]\n            color = colors[cls_id] if type(colors) is list else colors\n            \n            if cls in show_classes: \n                x1 = int(round(bbox[0]))\n                y1 = int(round(bbox[1]))\n                x2 = int(round(bbox[2]))\n                y2 = int(round(bbox[3]))\n                voc_bbox = (x1, y1, x2, y2)\n                plot_one_box(voc_bbox, \n                             image,\n                             color = color,\n                             label = cls if class_name else str(cls_id),\n                             line_thickness = line_thickness)\n    else:\n        raise ValueError('wrong bbox format')\n\n    return image","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.35393Z","iopub.status.idle":"2022-02-13T04:48:23.354509Z","shell.execute_reply.started":"2022-02-13T04:48:23.354278Z","shell.execute_reply":"2022-02-13T04:48:23.354303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def predict(model, img, size=768, augment=False):\n    height, width = img.shape[:2]\n    results = model(img, size=size, augment=augment)  # custom inference size\n    preds   = results.pandas().xyxy[0]\n    bboxes  = preds[['xmin','ymin','xmax','ymax']].values\n    if len(bboxes):\n        bboxes  = voc2coco(bboxes,height,width).astype(int)\n        confs   = preds.confidence.values\n        return bboxes, confs\n    else:\n        return [],[]\n    \ndef format_prediction(bboxes, confs):\n    annot = ''\n    if len(bboxes)>0:\n        for idx in range(len(bboxes)):\n            xmin, ymin, w, h = bboxes[idx]\n            conf             = confs[idx]\n            annot += f'{conf} {xmin} {ymin} {w} {h}'\n            annot +=' '\n        annot = annot.strip(' ')\n    return annot\n\ndef show_img(img, bboxes, bbox_format='yolo', colors = (255, 0 ,0)):\n    names  = ['starfish']*len(bboxes)\n    labels = [0]*len(bboxes)\n    \n    img = draw_bboxes(img = img,\n                           bboxes = bboxes, \n                           classes = names,\n                           class_ids = labels,\n                           class_name = True, \n                           colors = colors, \n                           bbox_format = bbox_format,\n                           line_thickness = 2)\n    \n    return Image.fromarray(img).resize((800, 400))","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.35563Z","iopub.status.idle":"2022-02-13T04:48:23.356209Z","shell.execute_reply.started":"2022-02-13T04:48:23.355964Z","shell.execute_reply":"2022-02-13T04:48:23.355988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def calculate_score(preds, gts, iou_th=0.5):\n    num_tp = 0\n    num_fp = 0\n    num_fn = 0\n    \n    if len(preds)!=0:\n        for p, gt in zip(preds, gts):\n            gt[2] = gt[2] + gt[0]\n            gt[3] = gt[3] + gt[1]\n            p[2] = p[2] + p[0]\n            p[3] = p[3] + p[1]\n            gt = torch.Tensor(np.array(gt).reshape([1,4]))\n            p = torch.Tensor(p.reshape([1,4]))\n            if len(p) and len(gt):\n                iou_matrix = box_iou(p, gt)[0].numpy()\n                tp = iou_matrix[0] >= iou_th\n                tp = tp.sum()\n                fp = len(p) - tp\n                fn = iou_matrix[0] < iou_th\n                fn = fn.sum()\n                num_tp += tp\n                num_fp += fp\n                num_fn += fn\n            elif len(p) == 0 and len(gt):\n                num_fn += len(gt)\n            elif len(p) and len(gt) == 0:\n                num_fp += len(p)\n        score = 5 * num_tp / (5 * num_tp + 4 * num_fn + num_fp)\n    else:\n        score = 0\n    return score","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.357403Z","iopub.status.idle":"2022-02-13T04:48:23.357979Z","shell.execute_reply.started":"2022-02-13T04:48:23.357749Z","shell.execute_reply":"2022-02-13T04:48:23.357773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the data\npath = '/kaggle/images/0-2313.jpg'\nim = cv2.imread('/kaggle/images/' + df[df['image_path']==path]['image_id'].iloc[0] + '.jpg')\nvid_id = df[df['image_path']==path]['image_id'].iloc[0].split('-')[0]\nseq_id = df[df['image_path']==path]['image_id'].iloc[0].split('-')[1]\n\nim = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)\ndh, dw, _ = im.shape\ntxt = open(f\"/kaggle/labels/{vid_id}-{seq_id}.txt\", \"r\").read().split(\" \")[1:]\nno_boxes = int(len(txt)/4)\n\n# Plot\nfig, axs = plt.subplots(1, 1, figsize=(23, 10))\nfig.suptitle(f\"Sample image and YOLO bounding boxes\", fontsize = 20)\n\n# Draw boxes\ni = 0\nwhile i < no_boxes:\n    i = i+4\n    box = txt[:i][-4:]\n\n    # Src: https://github.com/pjreddie/darknet/blob/810d7f797bdb2f021dbe65d2524c2ff6b8ab5c8b/src/image.c#L283-L291\n    # from YOLO to COCO\n    x, y, w, h = box\n    x, y, w, h = float(x), float(y), float(w), float(h)\n\n    l = int((x - w / 2) * dw)\n    r = int((x + w / 2) * dw)\n    t = int((y - h / 2) * dh)\n    b = int((y + h / 2) * dh)\n\n    if l < 0: l = 0\n    if r > dw - 1: r = dw - 1\n    if t < 0: t = 0\n    if b > dh - 1: b = dh - 1\n\n    cv2.rectangle(im, (l, t), (r, b), (255,0,0), 3)\n\n# Show image with bboxes\naxs.imshow(im)\naxs.set_axis_off()\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.359127Z","iopub.status.idle":"2022-02-13T04:48:23.359718Z","shell.execute_reply.started":"2022-02-13T04:48:23.35949Z","shell.execute_reply":"2022-02-13T04:48:23.359515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(l, t), (r, b)","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.360781Z","iopub.status.idle":"2022-02-13T04:48:23.361388Z","shell.execute_reply.started":"2022-02-13T04:48:23.361112Z","shell.execute_reply":"2022-02-13T04:48:23.361139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.set_option('display.max_columns', None)\ndf[df['image_path']==path]","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.36247Z","iopub.status.idle":"2022-02-13T04:48:23.363028Z","shell.execute_reply.started":"2022-02-13T04:48:23.362796Z","shell.execute_reply":"2022-02-13T04:48:23.362822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data[train_data['image_path']==path]","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.364066Z","iopub.status.idle":"2022-02-13T04:48:23.364632Z","shell.execute_reply.started":"2022-02-13T04:48:23.3644Z","shell.execute_reply":"2022-02-13T04:48:23.364426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# BoundingBox Confidence\nmodel.conf = 0.2\n\nfrom IPython.display import display\nimg = cv2.imread(path)[...,::-1]\nbboxes, confis = predict(model, img, size=IMG_SIZE, augment=True)\ndisplay(show_img(img, bboxes, bbox_format='coco', colors = (255,0,0)))\ndisplay(show_img(img, df[df['image_path']==path]['bboxes'].iloc[0], bbox_format='coco', colors = (0,255,0)))","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.365696Z","iopub.status.idle":"2022-02-13T04:48:23.366267Z","shell.execute_reply.started":"2022-02-13T04:48:23.366011Z","shell.execute_reply":"2022-02-13T04:48:23.366036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"F2scores_list = []\nconf_list = []\nimage_paths = df[(df['video_id'].isin(val_ids)) & (df.no_annotations>3)].sample(10).image_path.tolist()\nfor i in tqdm(range(0, 30, 1)):\n    conf_list.append(i/100)\n    model.conf = i/100\n    F2scores_list_i = []\n    for path in image_paths:\n        img = cv2.imread(path)[...,::-1]\n        bboxes, confis = predict(model, img, size=IMG_SIZE, augment=True)\n        F2scores_list_i.append(calculate_score(bboxes, df[df['image_path']==path]['bboxes'].iloc[0]))\n    F2scores_list.append(np.mean(F2scores_list_i))","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.367374Z","iopub.status.idle":"2022-02-13T04:48:23.367932Z","shell.execute_reply.started":"2022-02-13T04:48:23.367707Z","shell.execute_reply":"2022-02-13T04:48:23.367731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.scatter(F2scores_list, conf_list)\nplt.title(\"CONF vs F2 score\")\nplt.xlabel('CONF')\nplt.ylabel('F2')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.368974Z","iopub.status.idle":"2022-02-13T04:48:23.36955Z","shell.execute_reply.started":"2022-02-13T04:48:23.369321Z","shell.execute_reply":"2022-02-13T04:48:23.369345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.iou = 0.2","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.3706Z","iopub.status.idle":"2022-02-13T04:48:23.371164Z","shell.execute_reply.started":"2022-02-13T04:48:23.370915Z","shell.execute_reply":"2022-02-13T04:48:23.37094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from IPython.display import display\nimage_paths = df[df.no_annotations>5].sample(100).image_path.tolist()\nfor idx, path in enumerate(image_paths):\n    img = cv2.imread(path)[...,::-1]\n    bboxes, confis = predict(model, img, size=IMG_SIZE, augment=True)\n    display(show_img(img, bboxes, bbox_format='coco'))\n    gc.collect()\n    torch.cuda.empty_cache()\n    if idx>2:\n        break","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.372225Z","iopub.status.idle":"2022-02-13T04:48:23.372782Z","shell.execute_reply.started":"2022-02-13T04:48:23.372548Z","shell.execute_reply":"2022-02-13T04:48:23.372574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Inference","metadata":{}},{"cell_type":"code","source":"import greatbarrierreef\nenv = greatbarrierreef.make_env() # initialize the environment\niter_test = env.iter_test() # an iterator which loops over the test set and sample submission","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.373836Z","iopub.status.idle":"2022-02-13T04:48:23.374406Z","shell.execute_reply.started":"2022-02-13T04:48:23.374161Z","shell.execute_reply":"2022-02-13T04:48:23.374187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for idx, (img, pred_df) in enumerate(tqdm(iter_test)):\n    bboxes, confs  = predict(model, img, size=IMG_SIZE, augment=True)\n    annot = format_prediction(bboxes, confs)\n    pred_df['annotations'] = annot\n    env.predict(pred_df)\n    if idx<3:\n        display(show_img(img, bboxes, bbox_format='coco'))","metadata":{"execution":{"iopub.status.busy":"2022-02-13T04:48:23.375467Z","iopub.status.idle":"2022-02-13T04:48:23.376014Z","shell.execute_reply.started":"2022-02-13T04:48:23.375781Z","shell.execute_reply":"2022-02-13T04:48:23.375808Z"},"trusted":true},"execution_count":null,"outputs":[]}]}