{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n\n

Great Barrier Reef - Image & Bounding Box Augmentation

\n\n# 1. Introduction\n\n> šŸŒŠ **Competition Goal**: accurately identify starfish (*COTS - coral-eating Crown-Of-Thorns Starfish*) in real-time by building an object detection model trained on underwater videos of coral reefs. This way we can help researchers & scientists to control **COTS outbreaks**, which are a threat to the Great Barrier Reef.\n\n### Crown of Thorns Starfish\n\nšŸŒŸ **What is this creature?** [The crown-of-thorns starfish](https://en.wikipedia.org/wiki/Crown-of-thorns_starfish) is a large starfish that preys upon hard, or stony, coral polyps. It receives its name from the *venomous thorn-like spines* that cover its upper surface, resembling the biblical crown of thorns. It is one of the largest starfish in the world.\n\n
\n\nšŸŒŸ **Why is this a problem?** *One or two Crown-of-Thorn starfish on a reef may be arguably beneficial* for biological diversity as they keep down the growth of fast-growing coral species and leave space for other, slow-growing corals. However, as the starfish population multiplies or the starfish begin eating coral tissue faster than it can grow back a devastating Crown-of-Thorn (COTS) outbreak can occur. It is not known exactly what causes a COTS outbreaks, however, scientists agree it could have something to do with increased levels of nutrients in the water due to agriculture runoff or warming oceans, leading to a plankton bloom which is a necessary food source for starfish larvae ([source here](https://oceangardener.org/crown-of-thorns-starfish/)).\n\nšŸŒŸ **Can an AI spot them?** A COTS outbreak can have devastating impacts to an entire coral reef, and depending on the event the ravenous starfish could wipe out nearly all living corals. Crown-of-Thorns are among some of the *larges starfish species*, generally 25-35cm (10-14inch) in diameter and can grow to a size of 80cm (31inch), this makes them easy to spot on a reef ([source here](https://oceangardener.org/crown-of-thorns-starfish/)).\n\n### ā¬‡ Libraries Below","metadata":{}},{"cell_type":"code","source":"# Libraries\nimport os\nimport sys\nimport wandb\nimport time\nimport random\nfrom tqdm import tqdm\nimport warnings\nimport cv2\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib as mpl\nimport matplotlib.patches as patches\nimport matplotlib.pyplot as plt\nfrom IPython.display import display_html\n\n\n# Environment check\nwarnings.filterwarnings(\"ignore\")\nos.environ[\"WANDB_SILENT\"] = \"true\"\nCONFIG = {'competition': 'greatReef', '_wandb_kernel': 'aot'}\n\n# šŸ Secrets\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nsecret_value_0 = user_secrets.get_secret(\"wandb\")\n\n! wandb login $secret_value_0\n\n# Custom colors\nclass color:\n S = '\\033[1m' + '\\033[94m'\n E = '\\033[0m'\n \nmy_colors = [\"#16558F\", \"#1583D2\", \"#61B0B7\", \"#ADDEFF\", \"#A99AEA\", \"#7158B7\"]\nprint(color.S+\"Notebook Color Scheme:\"+color.E)\nsns.palplot(sns.color_palette(my_colors))\n\n# Set Style\nsns.set_style(\"white\")\nmpl.rcParams['xtick.labelsize'] = 14\nmpl.rcParams['ytick.labelsize'] = 14\nmpl.rcParams['axes.spines.left'] = False\nmpl.rcParams['axes.spines.right'] = False\nmpl.rcParams['axes.spines.top'] = False\nplt.rcParams.update({'font.size': 14})","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-01-05T16:19:54.330191Z","iopub.execute_input":"2022-01-05T16:19:54.330892Z","iopub.status.idle":"2022-01-05T16:20:00.536870Z","shell.execute_reply.started":"2022-01-05T16:19:54.330735Z","shell.execute_reply":"2022-01-05T16:20:00.535872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ā¬‡ Helper Functions Below","metadata":{}},{"cell_type":"code","source":"def show_values_on_bars(axs, h_v=\"v\", space=0.4):\n '''Plots the value at the end of the a seaborn barplot.\n axs: the ax of the plot\n h_v: weather or not the barplot is vertical/ horizontal'''\n \n def _show_on_single_plot(ax):\n if h_v == \"v\":\n for p in ax.patches:\n _x = p.get_x() + p.get_width() / 2\n _y = p.get_y() + p.get_height()\n value = int(p.get_height())\n ax.text(_x, _y, format(value, ','), ha=\"center\") \n elif h_v == \"h\":\n for p in ax.patches:\n _x = p.get_x() + p.get_width() + float(space)\n _y = p.get_y() + p.get_height()\n value = int(p.get_width())\n ax.text(_x, _y, format(value, ','), ha=\"left\")\n\n if isinstance(axs, np.ndarray):\n for idx, ax in np.ndenumerate(axs):\n _show_on_single_plot(ax)\n else:\n _show_on_single_plot(axs)\n \n \n# === šŸ W&B ===\ndef save_dataset_artifact(run_name, artifact_name, path):\n '''Saves dataset to W&B Artifactory.\n run_name: name of the experiment\n artifact_name: under what name should the dataset be stored\n path: path to the dataset'''\n \n run = wandb.init(project='g2net', \n name=run_name, \n config=CONFIG, anonymous=\"allow\")\n artifact = wandb.Artifact(name=artifact_name, \n type='dataset')\n artifact.add_file(path)\n\n wandb.log_artifact(artifact)\n wandb.finish()\n print(\"Artifact has been saved successfully.\")\n \n \ndef create_wandb_plot(x_data=None, y_data=None, x_name=None, y_name=None, title=None, log=None, plot=\"line\"):\n '''Create and save lineplot/barplot in W&B Environment.\n x_data & y_data: Pandas Series containing x & y data\n x_name & y_name: strings containing axis names\n title: title of the graph\n log: string containing name of log'''\n \n data = [[label, val] for (label, val) in zip(x_data, y_data)]\n table = wandb.Table(data=data, columns = [x_name, y_name])\n \n if plot == \"line\":\n wandb.log({log : wandb.plot.line(table, x_name, y_name, title=title)})\n elif plot == \"bar\":\n wandb.log({log : wandb.plot.bar(table, x_name, y_name, title=title)})\n elif plot == \"scatter\":\n wandb.log({log : wandb.plot.scatter(table, x_name, y_name, title=title)})\n \n \ndef create_wandb_hist(x_data=None, x_name=None, title=None, log=None):\n '''Create and save histogram in W&B Environment.\n x_data: Pandas Series containing x values\n x_name: strings containing axis name\n title: title of the graph\n log: string containing name of log'''\n \n data = [[x] for x in x_data]\n table = wandb.Table(data=data, columns=[x_name])\n wandb.log({log : wandb.plot.histogram(table, x_name, title=title)})","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T16:20:00.539840Z","iopub.execute_input":"2022-01-05T16:20:00.540332Z","iopub.status.idle":"2022-01-05T16:20:00.557431Z","shell.execute_reply.started":"2022-01-05T16:20:00.540260Z","shell.execute_reply":"2022-01-05T16:20:00.556354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. šŸŒŠ Dataset Understanding\n\n## 2.1 [train.csv]\n\nThe `train.csv` dataset contains 5 columns that help identify the position within the video and sequence of the .jpg images within the `train_images` folder.\n\nAdditionaly, it has an `annotations` columns, which can be empty (`[]`) or could contain 1 or multiple coordinates for the location (or a bounding box) of the COTS.\n\n
","metadata":{}},{"cell_type":"code","source":"# W&B Experiment\nrun = wandb.init(project='GreatBarrierReef', name='DataUnderstanding', config=CONFIG, anonymous=\"allow\")\n\n# Read training dataset\ntrain_df = pd.read_csv(\"../input/tensorflow-great-barrier-reef/train.csv\")\ntest_df = pd.read_csv(\"../input/tensorflow-great-barrier-reef/test.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:20:00.559186Z","iopub.execute_input":"2022-01-05T16:20:00.559520Z","iopub.status.idle":"2022-01-05T16:20:13.121480Z","shell.execute_reply.started":"2022-01-05T16:20:00.559472Z","shell.execute_reply":"2022-01-05T16:20:13.119956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df1_styler = train_df.sample(n=5, random_state=24).style.set_table_attributes(\"style='display:inline'\").set_caption('Sample Train Data')\ndf2_styler = test_df.head().style.set_table_attributes(\"style='display:inline'\").set_caption('Test Data (the rest is hidden)')\n\ndisplay_html(df1_styler._repr_html_(), raw=True)\nprint(\"\\n\")\ndisplay_html(df2_styler._repr_html_(), raw=True)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T16:20:13.123776Z","iopub.execute_input":"2022-01-05T16:20:13.124096Z","iopub.status.idle":"2022-01-05T16:20:14.876318Z","shell.execute_reply.started":"2022-01-05T16:20:13.124060Z","shell.execute_reply":"2022-01-05T16:20:14.875474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### I. Length of Videos, Sequences and Frames\n\nšŸ” There are **3 total videos**, with the last one having the most frames (.jpg images) out of all. However, they are not extremely imbalanced, with enough frame numbers fro each of the 3 videos.\n\nšŸ” Each **video is split into sequences**. 1 video is split into 4 sequences, while the other 2 videos are split into 8 sequences each. Each sequence has an unique ID and has various numbers of frames, raging from 71 frames per sequence all the way to ~3,000 frames per sequence.","metadata":{}},{"cell_type":"code","source":"fig, ((ax1, ax2)) = plt.subplots(nrows=1, ncols=2, figsize=(23, 10))\n\n# --- Plot 1 ---\ndf1 = train_df[\"video_id\"].value_counts().reset_index()\n\nsns.barplot(data=df1, x=\"index\", y=\"video_id\", ax=ax1,\n palette=my_colors)\nshow_values_on_bars(ax1, h_v=\"v\", space=0.1)\nax1.set_xlabel(\"Video ID\")\nax1.set_ylabel(\"\")\nax1.title.set_text(\"Frequency of Frames per Video\")\nax1.set_yticks([])\n\n# --- Plot 2 ---\ndf2 = train_df[\"sequence\"].value_counts().reset_index()\n\nsns.barplot(data=df2, y=\"index\", x=\"sequence\", order=df2[\"index\"],\n ax=ax2, orient=\"h\", palette=\"BuPu_r\")\nshow_values_on_bars(ax2, h_v=\"h\", space=0.1)\nax2.set_xlabel(\"\")\nax2.set_ylabel(\"Sequence ID\")\nax2.title.set_text(\"Frequency of Frames per Sequence\")\nax2.set_xticks([])\n\nsns.despine(top=True, right=True, left=True, bottom=True, ax=ax1)\nsns.despine(top=True, right=True, left=True, bottom=True, ax=ax2)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:20:14.880073Z","iopub.execute_input":"2022-01-05T16:20:14.880515Z","iopub.status.idle":"2022-01-05T16:20:17.011867Z","shell.execute_reply.started":"2022-01-05T16:20:14.880464Z","shell.execute_reply":"2022-01-05T16:20:17.011216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# šŸ Log plots into W&B Dashboard\ncreate_wandb_plot(x_data=df1.index, \n y_data=df1.video_id, \n x_name=\"Video ID\", y_name=\" \", \n title=\"-Frequency of Frames per Video-\", \n log=\"frames\", plot=\"bar\")\n\ncreate_wandb_plot(x_data=df2.index, \n y_data=df2.sequence, \n x_name=\"Sequence ID\", y_name=\" \", \n title=\"-Frequency of Frames per Sequence-\", \n log=\"frames2\", plot=\"bar\")","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:20:17.012972Z","iopub.execute_input":"2022-01-05T16:20:17.013360Z","iopub.status.idle":"2022-01-05T16:20:21.185668Z","shell.execute_reply.started":"2022-01-05T16:20:17.013322Z","shell.execute_reply":"2022-01-05T16:20:21.184895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### II. Target Variable - `annotations`\n\nWe can compute the total number of annotations per frame (or .jpg image) by counting how many coordinates can be found within a frame.","metadata":{}},{"cell_type":"code","source":"# Calculate the number of total annotations within the frame\ntrain_df[\"no_annotations\"] = train_df[\"annotations\"].apply(lambda x: len(eval(x)))","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:20:21.186902Z","iopub.execute_input":"2022-01-05T16:20:21.187557Z","iopub.status.idle":"2022-01-05T16:20:22.793561Z","shell.execute_reply.started":"2022-01-05T16:20:21.187515Z","shell.execute_reply":"2022-01-05T16:20:22.792450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"šŸ” The distribution of annotations is extremely skewed, with **most of the frames having no annotation** at all.\n\nšŸ” For the frames that do have annotations, most have between **1 and 3 annotations**, with a few outlier frames that have more than 10 unique coordinates (bounding boxes) identified within the image.","metadata":{}},{"cell_type":"code","source":"# % annotations\nn = len(train_df)\nno_annot = round(train_df[train_df[\"no_annotations\"]==0].shape[0]/n*100)\nwith_annot = round(train_df[train_df[\"no_annotations\"]!=0].shape[0]/n*100)\n\nprint(color.S + f\"There are ~{no_annot}% frames with no annotation and\" + color.E,\n \"\\n\",\n color.S + f\"only ~{with_annot}% frames with at least 1 annotation.\" + color.E)\n\n# Plot\nplt.figure(figsize=(23, 6))\nsns.histplot(train_df[\"no_annotations\"], bins=19, kde=True, element=\"step\", \n color=my_colors[5])\n\nplt.xlabel(\"Number of Annotations\")\nplt.ylabel(\"Frequency\")\nplt.title(\"Distribution for Number of Annotations per Frame\")\n\nsns.despine(top=True, right=True, left=False, bottom=True)\n\nn = len(train_df)\nno_annot = round(train_df[train_df.no_annotations==0].shape[0]/n*100)\nwith_annot = round(train_df[train_df.no_annotations!=0].shape[0]/n*100)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:20:22.794968Z","iopub.execute_input":"2022-01-05T16:20:22.795274Z","iopub.status.idle":"2022-01-05T16:20:24.936745Z","shell.execute_reply.started":"2022-01-05T16:20:22.795232Z","shell.execute_reply":"2022-01-05T16:20:24.935744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# šŸ Log info and plots into W&B Dashboard\nwandb.log({\"no annotations\": no_annot,\n \"with annotations\": with_annot})\n\ncreate_wandb_hist(x_data=train_df[\"no_annotations\"],\n x_name=\"Number of Annotations\",\n title=\"Distribution for Number of Annotations per Frame\",\n log=\"annotations\")","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:20:24.938383Z","iopub.execute_input":"2022-01-05T16:20:24.938757Z","iopub.status.idle":"2022-01-05T16:20:28.426414Z","shell.execute_reply.started":"2022-01-05T16:20:24.938728Z","shell.execute_reply":"2022-01-05T16:20:28.425612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I also wanted to look at all sequences and see **how the annotations distribute through time**. We know that each sequence has the frames (.jpg images) numerotated in the order that they appear within the video, from 1 to n. Hence, we can visualize the number of annotations per frame through time to see *if these have irregularities between sequences, or if they have some kind of systematic appearance*.\n\nšŸ” **Sequences 53708, 8503, 60754, 22643 and 8399**: these have **lots of annotations** throughout the entire sequence, with no particular pattern of apparition (what I mean by this is that the annotations don't seem to usually appear either at the beginning, middle nor end of the sequence).\n\nšŸ” **Sequences 44160, 29424, 37114**: these **don't have ANY annotation** appear in any of the frame, meaning that no COTS has been identified and tagged within these images.\n\nšŸ” **All other sequences**: for the remainer of sequences, most have a few or close to no annotation within them. These sequences don't seem to have an apparition pattern either, so I tend to believe that **the COTS appear as sporradic as possible within the videos** (which is very good, we want to mimic a natural setting as much as possible).","metadata":{}},{"cell_type":"code","source":"# List of unique sequence values\nsequences = list(train_df[\"sequence\"].unique())\n\nplt.figure(figsize=(23,20))\nplt.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=0.2, hspace=0.5)\nplt.suptitle(\"Frequency of annotations on sequence length\", fontsize = 20)\n\n# Enumerate through all sequences\nfor k, sequence in enumerate(sequences):\n train_df[train_df[\"sequence\"] == sequence]\n df_seq = train_df[train_df[\"sequence\"] == sequence]\n \n plt.subplot(5, 4, k+1)\n plt.title(f\"Sequence: {sequence}\", fontsize = 12)\n plt.xlabel(\"Seq Frame\", fontsize=10)\n plt.ylabel(\"No. Annot\", fontsize=10)\n plt.xticks(fontsize=10); plt.yticks(fontsize=10)\n sns.lineplot(x=df_seq[\"sequence_frame\"], y=df_seq[\"no_annotations\"],\n color=my_colors[2], lw=3)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:20:28.428073Z","iopub.execute_input":"2022-01-05T16:20:28.428720Z","iopub.status.idle":"2022-01-05T16:20:34.354713Z","shell.execute_reply.started":"2022-01-05T16:20:28.428674Z","shell.execute_reply":"2022-01-05T16:20:34.353386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wandb.finish()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:20:34.356213Z","iopub.execute_input":"2022-01-05T16:20:34.356450Z","iopub.status.idle":"2022-01-05T16:21:07.090079Z","shell.execute_reply.started":"2022-01-05T16:20:34.356421Z","shell.execute_reply":"2022-01-05T16:21:07.089222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2 [train_images]\n\nThe `train_images` folder is structured as follows:\n\n
\n\n### I. Showing 1 Frame\n\nBefore doing anything, let's explore the frames and how do they look like. Again, a **frame is actually a .jpg image**, a picture caught in time within the video.","metadata":{}},{"cell_type":"code","source":"# W&B Experiment\nrun = wandb.init(project='GreatBarrierReef', name='ExampleImages', config=CONFIG, anonymous=\"allow\")\n\n# Create a \"path\" column containing full path to the frames\nbase_folder = \"../input/tensorflow-great-barrier-reef/train_images\"\n\ntrain_df[\"path\"] = base_folder + \"/video_\" + \\\n train_df['video_id'].astype(str) + \"/\" +\\\n train_df['video_frame'].astype(str) +\".jpg\"","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:21:07.093938Z","iopub.execute_input":"2022-01-05T16:21:07.095097Z","iopub.status.idle":"2022-01-05T16:21:15.784212Z","shell.execute_reply.started":"2022-01-05T16:21:07.094793Z","shell.execute_reply":"2022-01-05T16:21:15.783158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# === Show image and annotations if applicable ===\ndef show_image(path, annot, axs=None):\n '''Shows an image and marks any COTS annotated within the frame.\n path: full path to the .jpg image\n annot: string of the annotation for the coordinates of COTS'''\n \n # This is in case we plot only 1 image\n if axs==None:\n fig, axs = plt.subplots(figsize=(23, 8))\n \n img = plt.imread(path)\n axs.imshow(img)\n\n if annot:\n for a in eval(annot):\n rect = patches.Rectangle((a[\"x\"], a[\"y\"]), a[\"width\"], a[\"height\"], \n linewidth=3, edgecolor=\"#FF6103\", facecolor='none')\n axs.add_patch(rect)\n\n axs.axis(\"off\")\n \n \n# === šŸW&B Log ===\ndef wandb_annotation(image, annotations):\n '''Source: https://www.kaggle.com/ayuraj/visualize-bounding-boxes-interactively\n image: the cv2.imread() output\n annotations: the original annotations from the train dataset'''\n \n all_annotations = []\n if annotations:\n for annot in eval(annotations):\n data = {\"position\": {\n \"minX\": annot[\"x\"],\n \"minY\": annot[\"y\"],\n \"maxX\": annot[\"x\"]+annot[\"width\"],\n \"maxY\": annot[\"y\"]+annot[\"height\"]\n },\n \"class_id\" : 1,\n \"domain\" : \"pixel\"}\n all_annotations.append(data)\n \n return wandb.Image(image, \n boxes={\"ground_truth\": {\"box_data\": all_annotations}}\n )","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:21:15.786369Z","iopub.execute_input":"2022-01-05T16:21:15.786644Z","iopub.status.idle":"2022-01-05T16:21:17.287215Z","shell.execute_reply.started":"2022-01-05T16:21:15.786612Z","shell.execute_reply":"2022-01-05T16:21:17.285880Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"šŸ” This is an example of a \"naked\" image - there are **no annotations found**, meaning that there are no COTS present.","metadata":{}},{"cell_type":"code","source":"# Show only 1 image as example\npath = list(train_df[train_df[\"no_annotations\"]==0][\"path\"])[0]\nannot = list(train_df[train_df[\"no_annotations\"]==0][\"annotations\"])[0]\n\n# šŸ Log Image to W&B\nimage = cv2.cvtColor(cv2.imread(path), cv2.COLOR_BGR2RGB)\nwandb_images = []\nwandb_images.append(wandb_annotation(image, annot))\n\nprint(color.S+\"Path:\"+color.E, path)\nprint(color.S+\"Annotation:\"+color.E, annot)\nprint(color.S+\"Frame:\"+color.E)\nshow_image(path, annot, axs=None)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:21:17.290782Z","iopub.execute_input":"2022-01-05T16:21:17.291072Z","iopub.status.idle":"2022-01-05T16:21:19.876598Z","shell.execute_reply.started":"2022-01-05T16:21:17.291041Z","shell.execute_reply":"2022-01-05T16:21:19.875502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"šŸ” The image below is a case that has the most annotations a frame can have (**18** bounding boxes in total).\n\nšŸ” Some COTS can be seen with the naked eye, however others are extremely hidden in the background.","metadata":{}},{"cell_type":"code","source":"# Show only 1 image as example\npath = list(train_df[train_df[\"no_annotations\"]==18][\"path\"])[0]\nannot = list(train_df[train_df[\"no_annotations\"]==18][\"annotations\"])[0]\n\n# šŸ Log Image to W&B\nimage = cv2.cvtColor(cv2.imread(path), cv2.COLOR_BGR2RGB)\nwandb_images.append(wandb_annotation(image, annot))\nwandb.log({\"example_image\": wandb_images})\n\nprint(color.S+\"Path:\"+color.E, path)\nprint(color.S+\"Annotation:\"+color.E, annot)\nprint(color.S+\"Frame:\"+color.E)\nshow_image(path, annot, axs=None)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:21:19.878027Z","iopub.execute_input":"2022-01-05T16:21:19.878311Z","iopub.status.idle":"2022-01-05T16:21:22.549971Z","shell.execute_reply.started":"2022-01-05T16:21:19.878279Z","shell.execute_reply":"2022-01-05T16:21:22.548729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### II. Show Multiple Consecutive Frames\n\nNow let's look at multiple consecutive frames within a few sequences.","metadata":{}},{"cell_type":"code","source":"def show_multiple_images(seq_id, frame_no):\n '''Shows multiple images within a sequence.\n seq_id: a number corresponding with the sequence unique ID\n frame_no: a list containing the first and last frame to plot'''\n \n # Select image paths & their annotations\n paths = list(train_df[(train_df[\"sequence\"]==seq_id) & \n (train_df[\"sequence_frame\"]>=frame_no[0]) & \n (train_df[\"sequence_frame\"]<=frame_no[1])][\"path\"])\n annotations = list(train_df[(train_df[\"sequence\"]==seq_id) & \n (train_df[\"sequence_frame\"]>=frame_no[0]) & \n (train_df[\"sequence_frame\"]<=frame_no[1])][\"annotations\"])\n\n # Plot\n fig, axs = plt.subplots(2, 3, figsize=(23, 10))\n axs = axs.flatten()\n fig.suptitle(f\"Showing consecutive frames for Sequence ID: {seq_id}\", fontsize = 20)\n\n for k, (path, annot) in enumerate(zip(paths, annotations)):\n axs[k].set_title(f\"Frame No: {frame_no[0]+k}\", fontsize = 12)\n show_image(path, annot, axs[k])\n\n plt.tight_layout()\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:21:22.551780Z","iopub.execute_input":"2022-01-05T16:21:22.552091Z","iopub.status.idle":"2022-01-05T16:21:25.219013Z","shell.execute_reply.started":"2022-01-05T16:21:22.552055Z","shell.execute_reply":"2022-01-05T16:21:25.217901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"šŸ” The frames below have **no COTS** identified within them.","metadata":{}},{"cell_type":"code","source":"seq_id = 44160\nframe_no = [51, 56]\n\nshow_multiple_images(seq_id, frame_no)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:21:25.220683Z","iopub.execute_input":"2022-01-05T16:21:25.220993Z","iopub.status.idle":"2022-01-05T16:21:29.260193Z","shell.execute_reply.started":"2022-01-05T16:21:25.220959Z","shell.execute_reply":"2022-01-05T16:21:29.258904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"šŸ” These frames however have **1 and 2 COTS** identified. Notice that in the first 3 frames only 1 COTS is annotated, however the second COTS one is also visible but NOT identified. This COTS is identified and annotated only starting the 4th frame onwards.","metadata":{}},{"cell_type":"code","source":"seq_id = 59337\nframe_no = [38, 43]\n\nshow_multiple_images(seq_id, frame_no)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:21:29.261851Z","iopub.execute_input":"2022-01-05T16:21:29.262197Z","iopub.status.idle":"2022-01-05T16:21:33.359595Z","shell.execute_reply.started":"2022-01-05T16:21:29.262149Z","shell.execute_reply":"2022-01-05T16:21:33.358598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"šŸ” At the polar opposite, these images show the presence of **multiple COTS** within them.\n\nšŸ” My question would be - could we somehow distort/enhance these images so we could better identify the presence of COTS within them? We already know that all the images will have around the same tonal colors (blue, green, yellow) and around the same texture.","metadata":{}},{"cell_type":"code","source":"seq_id = 53708\nframe_no = [801, 806]\n\nshow_multiple_images(seq_id, frame_no)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:21:33.360888Z","iopub.execute_input":"2022-01-05T16:21:33.361124Z","iopub.status.idle":"2022-01-05T16:21:37.353884Z","shell.execute_reply.started":"2022-01-05T16:21:33.361095Z","shell.execute_reply":"2022-01-05T16:21:37.352936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### III. Comparison between No Annotated vs Annotated Images\n\nI wanted to look at multiple random images/frames and see if they look significantly different.","metadata":{}},{"cell_type":"code","source":"def plot_comparison(no_annot, state=24):\n \n # Select image paths & their annotations\n paths_compare = list(train_df[train_df[\"no_annotations\"]==no_annot]\\\n .sample(n=9, random_state=state)[\"path\"])\n annotations_compare = list(train_df[train_df[\"no_annotations\"]==no_annot]\\\n .sample(n=9, random_state=state)[\"annotations\"])\n\n # Plot\n fig, axs = plt.subplots(3, 3, figsize=(23, 13))\n axs = axs.flatten()\n fig.suptitle(f\"{no_annot} annotations\", fontsize = 20)\n\n for k, (path, annot) in enumerate(zip(paths_compare, annotations_compare)):\n video_id = path.split(\"/\")[4]\n frame_id = path.split(\"/\")[-1].split(\".\")[0]\n \n axs[k].set_title(f\"{video_id} | Frame {frame_id}\",\n fontsize = 12)\n show_image(path, annot, axs[k])\n\n plt.tight_layout()\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:21:37.355623Z","iopub.execute_input":"2022-01-05T16:21:37.356369Z","iopub.status.idle":"2022-01-05T16:21:38.773976Z","shell.execute_reply.started":"2022-01-05T16:21:37.356313Z","shell.execute_reply":"2022-01-05T16:21:38.772913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# No annotations\nno_annot = 0\nplot_comparison(no_annot, state=24)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:21:38.775672Z","iopub.execute_input":"2022-01-05T16:21:38.776081Z","iopub.status.idle":"2022-01-05T16:21:43.696486Z","shell.execute_reply.started":"2022-01-05T16:21:38.776026Z","shell.execute_reply":"2022-01-05T16:21:43.695448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 5 annotations\nno_annot = 5\nplot_comparison(no_annot, state=24)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:21:43.698210Z","iopub.execute_input":"2022-01-05T16:21:43.698430Z","iopub.status.idle":"2022-01-05T16:21:49.514996Z","shell.execute_reply.started":"2022-01-05T16:21:43.698403Z","shell.execute_reply":"2022-01-05T16:21:49.513687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 17 annotations\nno_annot = 17\nplot_comparison(no_annot, state=24)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:21:49.516332Z","iopub.execute_input":"2022-01-05T16:21:49.516581Z","iopub.status.idle":"2022-01-05T16:21:54.830457Z","shell.execute_reply.started":"2022-01-05T16:21:49.516551Z","shell.execute_reply":"2022-01-05T16:21:54.829237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Bounding Box Augmentation\n\nIn this part I wanted to explore some ways to do Image Augmentation **and** adjust the annotations (aka bounding boxes) to match all sorts of augmentations applied on the image.\n\nšŸ” Before we do that, we will need to format the annotations we have now:\n* from this: {'x': 628, 'y': 321, 'width': 42, 'height': 47}\n* to this: {'x1': 628, 'y1': 321, 'x2': 670, 'y2': 368} => [628, 321, 670, 368]\n\n
\n\nIn order to do so, we just need to compute as follows:\n* x1 = x\n* y1 = y\n* x2 = x + width\n* y2 = y + height\n\n> šŸ¦¦ **Note**: we are adding and not substracting to y2 because we aren't using a coordinate system, although x and y are coordinates, but an image, so the \"coordinates\" are actually pixels on the surface. Hence, the top left corner of an image has the coordinates `[0, 0]`, while the bottom right corner has the coordinates `[width_max, height_max]`.\n\n*Example*:\n\n* first bbox (bigger one): `{'x': 520, 'y': 151, 'width': 78, 'height': 62}`\n* second bbox (smaller one): `{'x': 598, 'y': 204, 'width': 58, 'height': 32}`\n\n
","metadata":{}},{"cell_type":"code","source":"def format_annotations(x):\n '''Changes annotations from format {x, y, width, height} to {x1, y1, x2, y2}.\n x: a string of the initial format.'''\n \n annotations = eval(x)\n new_annotations = []\n\n if annotations:\n for annot in annotations:\n new_annotations.append([annot[\"x\"],\n annot[\"y\"],\n annot[\"x\"]+annot[\"width\"],\n annot[\"y\"]+annot[\"height\"]\n ])\n \n if new_annotations: return str(new_annotations)\n else: return \"[]\"","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:54:33.170482Z","iopub.execute_input":"2022-01-05T16:54:33.170904Z","iopub.status.idle":"2022-01-05T16:54:33.177665Z","shell.execute_reply.started":"2022-01-05T16:54:33.170857Z","shell.execute_reply":"2022-01-05T16:54:33.176399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a new column with the new formated annotations\ntrain_df[\"f_annotations\"] = train_df[\"annotations\"].apply(lambda x: format_annotations(x))","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:41:05.282994Z","iopub.execute_input":"2022-01-05T16:41:05.283302Z","iopub.status.idle":"2022-01-05T16:41:05.517188Z","shell.execute_reply.started":"2022-01-05T16:41:05.283267Z","shell.execute_reply":"2022-01-05T16:41:05.516297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"šŸ” One last thing I would like to do is create a new function called `show_image_bbox` that receives the new formated annotations ({x1, y1, x2, y2}) and displays the new augmented image.","metadata":{}},{"cell_type":"code","source":"def show_image_bbox(img, annot, axs=None):\n '''Shows an image and marks any COTS annotated within the frame.\n img: the output from cv2.imread()\n annot: FORMATED annotation'''\n \n # This is in case we plot only 1 image\n if axs==None:\n fig, axs = plt.subplots(figsize=(23, 8))\n \n axs.imshow(img)\n\n if annot:\n for a in annot:\n rect = patches.Rectangle((a[0], a[1]), a[2]-a[0], a[3]-a[1], \n linewidth=3, edgecolor=\"#FF6103\", facecolor='none')\n axs.add_patch(rect)\n\n axs.axis(\"off\")","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:54:35.492290Z","iopub.execute_input":"2022-01-05T16:54:35.492623Z","iopub.status.idle":"2022-01-05T16:54:35.499629Z","shell.execute_reply.started":"2022-01-05T16:54:35.492590Z","shell.execute_reply":"2022-01-05T16:54:35.498979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.1 (Random) Horizontal Flip\n\nCreates a class that (randomly) flips the image (and the bounding box with it).\n\n> **Note**: Keep in mind that cv2 works with *BGR* images - so, in order to view the original image within RGB, we need to convert using `cv2.cvtColor(cv2.imread(path), cv2.COLOR_BGR2RGB)`.\n\n*šŸ” Note: Most of my inspiration and research is from here: https://blog.paperspace.com/data-augmentation-for-bounding-boxes/*","metadata":{}},{"cell_type":"code","source":"class RandomHorizontalFlip(object):\n\n def __init__(self, p=0.5):\n # p = probability of the image to be flipped\n # set p = 1 to always flip\n self.p = p\n \n def __call__(self, img, bboxes):\n '''img : the image to be flipped\n bboxes : the annotations within the image'''\n \n # Convert bboxes\n bboxes = np.array(bboxes)\n \n img_center = np.array(img.shape[:2])[::-1]/2\n img_center = np.hstack((img_center, img_center))\n \n # If random number between 0 and 1 < probability p\n if random.random() < self.p:\n # Reverse image elements in the 1st dimension\n img = img[:,::-1,:]\n bboxes[:,[0,2]] = bboxes[:,[0,2]] + 2*(img_center[[0,2]] - bboxes[:,[0,2]])\n \n # Convert the bounding boxes\n box_w = abs(bboxes[:,0] - bboxes[:,2])\n bboxes[:,0] -= box_w\n bboxes[:,2] += box_w\n \n return img, bboxes.tolist()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:45:50.503683Z","iopub.execute_input":"2022-01-05T16:45:50.503971Z","iopub.status.idle":"2022-01-05T16:45:50.512284Z","shell.execute_reply.started":"2022-01-05T16:45:50.503940Z","shell.execute_reply":"2022-01-05T16:45:50.511188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"šŸ” Let's see an example of the original image and then the **flipped** one.","metadata":{}},{"cell_type":"code","source":"# Take an example\npath = list(train_df[train_df[\"no_annotations\"]==18][\"path\"])[0]\n\nimg_original = cv2.cvtColor(cv2.imread(path), cv2.COLOR_BGR2RGB)\nannot_original = eval(list(train_df[train_df[\"no_annotations\"]==18][\"f_annotations\"])[0])\n\n# Horizontal Flip\nhorizontal_flip = RandomHorizontalFlip(p=1) \nimg_flipped, annot_flipped = horizontal_flip(img_original, annot_original)\n\n\n\n# Show the Before and After\nfig, axs = plt.subplots(1, 2, figsize=(23, 10))\naxs = axs.flatten()\nfig.suptitle(f\"(Random) Horizontal Flip\", fontsize = 20)\n\naxs[0].set_title(\"Original Image\", fontsize = 20)\nshow_image_bbox(img_original, annot_original, axs=axs[0])\n\naxs[1].set_title(\"With Horizontal Flip\", fontsize = 20)\nshow_image_bbox(img_flipped, annot_flipped, axs[1])\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:54:38.407064Z","iopub.execute_input":"2022-01-05T16:54:38.407398Z","iopub.status.idle":"2022-01-05T16:54:39.607223Z","shell.execute_reply.started":"2022-01-05T16:54:38.407364Z","shell.execute_reply":"2022-01-05T16:54:39.606229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.2 (Random) Scaling\n\nWhen we scale the image, we **descrease it's original size**. In this case, the *bounding boxes which have an area of less than 25% in the remaining in the transformed image is dropped*. The resolution is maintained, and the remaining area if any is filled by black color.\n\n*šŸ” Note: Most of my inspiration and research is from here: https://blog.paperspace.com/data-augmentation-bounding-boxes-scaling-translation/*","metadata":{}},{"cell_type":"code","source":"# ==== Clips the bboxes ====\ndef bbox_area(bbox):\n return (bbox[:,2] - bbox[:,0])*(bbox[:,3] - bbox[:,1])\n\ndef clip_box(bbox, clip_box, alpha):\n \"\"\"\n Clip the bounding boxes to the borders of an image\n bbox: numpy.ndarray\n Numpy array containing bounding boxes of shape `N X 4` where N is the \n number of bounding boxes and the bounding boxes are represented in the\n format `x1 y1 x2 y2`\n \n clip_box: numpy.ndarray\n An array of shape (4,) specifying the diagonal co-ordinates of the image\n The coordinates are represented in the format `x1 y1 x2 y2`\n \n alpha: float\n If the fraction of a bounding box left in the image after being clipped is \n less than `alpha` the bounding box is dropped. \n \n Returns\n -------\n numpy.ndarray\n Numpy array containing **clipped** bounding boxes of shape `N X 4` where N is the \n number of bounding boxes left are being clipped and the bounding boxes are represented in the\n format `x1 y1 x2 y2` \n \"\"\"\n ar_ = (bbox_area(bbox))\n x_min = np.maximum(bbox[:,0], clip_box[0]).reshape(-1,1)\n y_min = np.maximum(bbox[:,1], clip_box[1]).reshape(-1,1)\n x_max = np.minimum(bbox[:,2], clip_box[2]).reshape(-1,1)\n y_max = np.minimum(bbox[:,3], clip_box[3]).reshape(-1,1)\n \n bbox = np.hstack((x_min, y_min, x_max, y_max, bbox[:,4:]))\n \n delta_area = ((ar_ - bbox_area(bbox))/ar_)\n \n mask = (delta_area < (1 - alpha)).astype(int)\n \n bbox = bbox[mask == 1,:]\n\n\n return bbox","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T16:22:03.962869Z","iopub.execute_input":"2022-01-05T16:22:03.963128Z","iopub.status.idle":"2022-01-05T16:22:05.422635Z","shell.execute_reply.started":"2022-01-05T16:22:03.963097Z","shell.execute_reply":"2022-01-05T16:22:05.421558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class RandomScale(object):\n\n def __init__(self, scale = 0.2, diff = False):\n \n # scale must always be a positive number\n self.scale = scale\n self.scale = (max(-1, -self.scale), self.scale)\n \n # Maintain the aspect ratio\n # (scaling factor remains the same for width & height)\n self.diff = diff\n \n \n def __call__(self, img, bboxes):\n \n # Convert bboxes\n bboxes = np.array(bboxes)\n\n #Chose a random digit to scale by \n img_shape = img.shape\n\n if self.diff:\n scale_x = random.uniform(*self.scale)\n scale_y = random.uniform(*self.scale)\n else:\n scale_x = random.uniform(*self.scale)\n scale_y = scale_x\n\n resize_scale_x = 1 + scale_x\n resize_scale_y = 1 + scale_y\n\n # Resize the image by scale factor\n img = cv2.resize(img, None, fx = resize_scale_x, fy = resize_scale_y)\n\n bboxes[:,:4] = bboxes[:,:4] * [resize_scale_x, resize_scale_y, resize_scale_x, resize_scale_y]\n\n # The black image (the remaining area after we have clipped the image)\n canvas = np.zeros(img_shape, dtype = np.uint8)\n\n # Determine the size of the scaled image\n y_lim = int(min(resize_scale_y,1)*img_shape[0])\n x_lim = int(min(resize_scale_x,1)*img_shape[1])\n\n canvas[:y_lim,:x_lim,:] = img[:y_lim,:x_lim,:]\n\n img = canvas\n # Adjust the bboxes - remove all annotations that dissapeared after the scaling\n bboxes = clip_box(bboxes, [0,0,1 + img_shape[1], img_shape[0]], 0.25)\n\n return img, bboxes.tolist()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:22:05.424094Z","iopub.execute_input":"2022-01-05T16:22:05.424307Z","iopub.status.idle":"2022-01-05T16:22:06.889998Z","shell.execute_reply.started":"2022-01-05T16:22:05.424281Z","shell.execute_reply":"2022-01-05T16:22:06.888008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"šŸ” Let's see an example of the original image and then the **scaled** one.","metadata":{}},{"cell_type":"code","source":"random.seed(24)\n\n# Scaling\nscale = RandomScale(scale=1.3, diff = False) \nimg_scaled, annot_scaled = scale(img_original, annot_original)\n\n\n\n# Show the Before and After\nfig, axs = plt.subplots(1, 2, figsize=(23, 10))\naxs = axs.flatten()\nfig.suptitle(f\"(Random) Image Scaling\", fontsize = 20)\n\naxs[0].set_title(\"Original Image\", fontsize = 20)\nshow_image_bbox(img_original, annot_original, axs=axs[0])\n\naxs[1].set_title(\"Scaled (zoomed in) Image\", fontsize = 20)\nshow_image_bbox(img_scaled, annot_scaled, axs[1])\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:22:06.891410Z","iopub.execute_input":"2022-01-05T16:22:06.891690Z","iopub.status.idle":"2022-01-05T16:22:09.511795Z","shell.execute_reply.started":"2022-01-05T16:22:06.891662Z","shell.execute_reply":"2022-01-05T16:22:09.510784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.3 (Random) Translate\n\nWhen we translate the image we **move it around on the canvas**. It's like if you would look through a camera lence at a piece of paper on a table and then you would move it left, right, up or down, leaving some parts of the table exposed and some areas or the paper not visible.\n\nAs in the case of scaling, the *bounding boxes which have an area of less than 25% in the remaining in the transformed image is dropped*. The resolution is maintained, and the remaining area if any is filled by black color.\n\n*šŸ” Note: Most of my inspiration and research is from here: https://blog.paperspace.com/data-augmentation-bounding-boxes-scaling-translation/*","metadata":{}},{"cell_type":"code","source":"class RandomTranslate(object):\n\n def __init__(self, translate = 0.2, diff = False):\n \n self.translate = translate\n self.translate = (-self.translate, self.translate)\n \n # Maintain the aspect ratio\n # (scaling factor remains the same for width & height)\n self.diff = diff\n \n def __call__(self, img, bboxes): \n \n # Convert bboxes\n bboxes = np.array(bboxes)\n \n # Chose a random digit to scale by \n img_shape = img.shape\n\n # Percentage of the dimension of the image to translate\n translate_factor_x = random.uniform(*self.translate)\n translate_factor_y = random.uniform(*self.translate)\n\n if not self.diff:\n translate_factor_y = translate_factor_x\n\n canvas = np.zeros(img_shape).astype(np.uint8)\n\n corner_x = int(translate_factor_x*img.shape[1])\n corner_y = int(translate_factor_y*img.shape[0])\n\n #Change the origin to the top-left corner of the translated box\n orig_box_cords = [max(0,corner_y), max(corner_x,0), min(img_shape[0], corner_y + img.shape[0]), min(img_shape[1],corner_x + img.shape[1])]\n\n mask = img[max(-corner_y, 0):min(img.shape[0], -corner_y + img_shape[0]), max(-corner_x, 0):min(img.shape[1], -corner_x + img_shape[1]),:]\n canvas[orig_box_cords[0]:orig_box_cords[2], orig_box_cords[1]:orig_box_cords[3],:] = mask\n img = canvas\n\n bboxes[:,:4] += [corner_x, corner_y, corner_x, corner_y]\n\n bboxes = clip_box(bboxes, [0,0,img_shape[1], img_shape[0]], 0.25)\n\n return img, bboxes.tolist()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:22:09.513469Z","iopub.execute_input":"2022-01-05T16:22:09.514335Z","iopub.status.idle":"2022-01-05T16:22:11.072443Z","shell.execute_reply.started":"2022-01-05T16:22:09.514270Z","shell.execute_reply":"2022-01-05T16:22:11.071501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"šŸ” Let's see an example of the original image and then the **translated** one.","metadata":{}},{"cell_type":"code","source":"random.seed(25)\n\n# Translate\ntranslate = RandomTranslate(translate=0.4, diff = False) \nimg_translated, annot_translated = translate(img_original, annot_original)\n\n\n\n# Show the Before and After\nfig, axs = plt.subplots(1, 2, figsize=(23, 10))\naxs = axs.flatten()\nfig.suptitle(f\"(Random) Image Translation\", fontsize = 20)\n\naxs[0].set_title(\"Original Image\", fontsize = 20)\nshow_image_bbox(img_original, annot_original, axs=axs[0])\n\naxs[1].set_title(\"Translated (shifted) Image\", fontsize = 20)\nshow_image_bbox(img_translated, annot_translated, axs[1])\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:22:11.073924Z","iopub.execute_input":"2022-01-05T16:22:11.074241Z","iopub.status.idle":"2022-01-05T16:22:13.741906Z","shell.execute_reply.started":"2022-01-05T16:22:11.074199Z","shell.execute_reply":"2022-01-05T16:22:13.740552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.4 (Random) Rotation\n\nRotation is when (you guessed) you rotate the image a random number degrees (and it might be the hardest one to deal with when trying to accomodate bounding boxes with it).\n\n*šŸ” Note: Most of my inspiration and research is from here: https://blog.paperspace.com/data-augmentation-for-object-detection-rotation-and-shearing/*\n\nTODO: needs work, bboxes don't rotate with the image\n\n### ā¬‡ļø Function for Image Rotation","metadata":{}},{"cell_type":"code","source":"# === Image Rotation ===\n\ndef rotate_im(image, angle):\n '''image: numpy array of the image'''\n '''angle: a float that specifies the angle the image should be rotated.'''\n\n # Image dimensions\n (h, w) = image.shape[:2]\n # Image Centre\n (cX, cY) = (w // 2, h // 2)\n\n # Rotation Matrix from cv2\n M = cv2.getRotationMatrix2D((cX, cY), angle, 1.0)\n # Sine & Cosine - rotation components of the matrix\n cos = np.abs(M[0, 0])\n sin = np.abs(M[0, 1])\n\n # NEW Bounding Dimensions of the image\n nW = int((h * sin) + (w * cos))\n nH = int((h * cos) + (w * sin))\n\n # Adjust the rotation matrix to take into account translation\n M[0, 2] += (nW / 2) - cX\n M[1, 2] += (nH / 2) - cY\n\n # Perform the Rotation\n image = cv2.warpAffine(image, M, (nW, nH))\n\n return image","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T16:22:13.743353Z","iopub.execute_input":"2022-01-05T16:22:13.743637Z","iopub.status.idle":"2022-01-05T16:22:15.271117Z","shell.execute_reply.started":"2022-01-05T16:22:13.743608Z","shell.execute_reply":"2022-01-05T16:22:15.270024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ā¬‡ļø Functions for BBox Rotation","metadata":{}},{"cell_type":"code","source":"# === Get Corners of Bounding Boxes ===\n\ndef get_corners(bboxes):\n '''bboxes: array of the original bounding boxes.'''\n \n width = (bboxes[:,2] - bboxes[:,0]).reshape(-1,1)\n height = (bboxes[:,3] - bboxes[:,1]).reshape(-1,1)\n \n x1 = bboxes[:,0].reshape(-1,1)\n y1 = bboxes[:,1].reshape(-1,1)\n \n x2 = x1 + width\n y2 = y1 \n \n x3 = x1\n y3 = y1 + height\n \n x4 = bboxes[:,2].reshape(-1,1)\n y4 = bboxes[:,3].reshape(-1,1)\n \n # Each bounding box is described by 8 coordinates x1,y1,x2,y2,x3,y3,x4,y4\n corners = np.hstack((x1,y1,x2,y2,x3,y3,x4,y4))\n \n return corners\n\n\n# === Box Rotation ===\n\ndef rotate_box(corners, angle, cx, cy, h, w):\n '''\n corners: output from get_corners()\n angle: a float that specifies the angle the image should be rotated\n cx, cy: coordinates for the xenter of the image\n h, w: height and width of the image\n '''\n \n # corners = x1,y1,x2,y2,x3,y3,x4,y4\n corners = corners.reshape(-1,2)\n corners = np.hstack((corners, np.ones((corners.shape[0],1), dtype = type(corners[0][0]))))\n \n # Rotation Matrix from cv2\n M = cv2.getRotationMatrix2D((cx, cy), angle, 1.0)\n # Sine & Cosine - rotation components of the matrix\n cos = np.abs(M[0, 0])\n sin = np.abs(M[0, 1])\n \n # NEW Bounding Dimensions of the image\n nW = int((h * sin) + (w * cos))\n nH = int((h * cos) + (w * sin))\n \n # Adjust the rotation matrix to take into account translation\n M[0, 2] += (nW / 2) - cx\n M[1, 2] += (nH / 2) - cy\n \n # Prepare the vector to be transformed\n calculated = np.dot(M,corners.T).T\n calculated = calculated.reshape(-1,8)\n \n return calculated\n\n\n# === Get the Enclosing Box ===\n\ndef get_enclosing_box(corners):\n '''corners: output from get_corners()'''\n \n x_ = corners[:,[0,2,4,6]]\n y_ = corners[:,[1,3,5,7]]\n \n xmin = np.min(x_,1).reshape(-1,1)\n ymin = np.min(y_,1).reshape(-1,1)\n xmax = np.max(x_,1).reshape(-1,1)\n ymax = np.max(y_,1).reshape(-1,1)\n \n # Notation where each bounding box is determined by 4 coordinates or two corners\n final = np.hstack((xmin, ymin, xmax, ymax,corners[:,8:]))\n \n return final","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-05T16:22:15.272663Z","iopub.execute_input":"2022-01-05T16:22:15.273324Z","iopub.status.idle":"2022-01-05T16:22:16.872870Z","shell.execute_reply.started":"2022-01-05T16:22:15.273276Z","shell.execute_reply":"2022-01-05T16:22:16.871852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class RandomRotate(object):\n\n def __init__(self, angle = 10):\n \n self.angle = angle\n self.angle = (-self.angle, self.angle)\n \n \n def __call__(self, img, bboxes):\n\n # Convert bboxes\n bboxes = np.array(bboxes)\n \n # Compute the random angle\n angle = random.uniform(*self.angle)\n\n # width, height and center of the image\n w,h = img.shape[1], img.shape[0]\n cx, cy = w//2, h//2\n\n # Rotate the image\n img = rotate_im(img, angle)\n\n # --- Rotate the bounding boxes ---\n # Get the 4 point corner coordinates\n corners = get_corners(bboxes)\n corners = np.hstack((corners, bboxes[:,4:]))\n # Rotate the bounding box\n corners[:,:8] = rotate_box(corners[:,:8], angle, cx, cy, h, w)\n # Get the enclosing (new bboxes)\n new_bbox = get_enclosing_box(corners)\n\n # Get scaling factors to clip the image and bboxes\n scale_factor_x = img.shape[1] / w\n scale_factor_y = img.shape[0] / h\n\n # Rescale the image - to w,h and not nW,nH\n img = cv2.resize(img, (w,h))\n\n # Clip boxes (in case there are any outside of the rotated image)\n bboxes[:,:4] = bboxes[:,:4] / [scale_factor_x, scale_factor_y, scale_factor_x, scale_factor_y] \n bboxes = clip_box(bboxes, [0,0,w, h], 0.25)\n\n return img, bboxes.tolist()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:22:16.874628Z","iopub.execute_input":"2022-01-05T16:22:16.874940Z","iopub.status.idle":"2022-01-05T16:22:18.311674Z","shell.execute_reply.started":"2022-01-05T16:22:16.874900Z","shell.execute_reply":"2022-01-05T16:22:18.310591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"šŸ” Let's see an example of the original image and then the **rotated** one.","metadata":{}},{"cell_type":"code","source":"random.seed(25)\n\n# Translate\nrotate = RandomRotate(angle=25) \nimg_rotated, annot_rotated = rotate(img_original, annot_original)\n\n\n\n# Show the Before and After\nfig, axs = plt.subplots(1, 2, figsize=(23, 10))\naxs = axs.flatten()\nfig.suptitle(f\"(Random) Image Rotation\", fontsize = 20)\n\naxs[0].set_title(\"Original Image\", fontsize = 20)\nshow_image_bbox(img_original, annot_original, axs=axs[0])\n\naxs[1].set_title(\"Rotated Image\", fontsize = 20)\nshow_image_bbox(img_rotated, annot_rotated, axs[1])\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:22:18.313494Z","iopub.execute_input":"2022-01-05T16:22:18.313715Z","iopub.status.idle":"2022-01-05T16:22:20.980255Z","shell.execute_reply.started":"2022-01-05T16:22:18.313689Z","shell.execute_reply":"2022-01-05T16:22:20.979116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.5 (Random) Shearing\n\nFinally, shearing is when the image is shifted, like it is dragged from one corner and opposite to the other, so the image ends up looking sort of like a parallelogram.\n\n*šŸ” Note: Most of my inspiration and research is from here: https://blog.paperspace.com/data-augmentation-for-object-detection-rotation-and-shearing/*","metadata":{}},{"cell_type":"code","source":"class RandomShear(object):\n\n def __init__(self, shear_factor = 0.2):\n \n self.shear_factor = shear_factor\n self.shear_factor = (-self.shear_factor, self.shear_factor)\n \n shear_factor = random.uniform(*self.shear_factor)\n \n \n def __call__(self, img, bboxes):\n \n # Convert bboxes\n bboxes = np.array(bboxes)\n\n # Get the shear factor and size of the image\n shear_factor = random.uniform(*self.shear_factor)\n w,h = img.shape[1], img.shape[0]\n\n # Flip the image and boxes horizontally\n if shear_factor < 0:\n img, bboxes = HorizontalFlip()(img, bboxes)\n\n # Apply the shear transformation\n M = np.array([[1, abs(shear_factor), 0],[0,1,0]])\n nW = img.shape[1] + abs(shear_factor*img.shape[0])\n\n bboxes[:,[0,2]] += ((bboxes[:,[1,3]]) * abs(shear_factor) ).astype(int) \n\n # Transform using cv2 warpAffine (like in rotation)\n img = cv2.warpAffine(img, M, (int(nW), img.shape[0]))\n\n # Flip the image back again\n if shear_factor < 0:\n img, bboxes = HorizontalFlip()(img, bboxes)\n\n # Resize\n img = cv2.resize(img, (w,h))\n\n scale_factor_x = nW / w\n bboxes[:,:4] = bboxes[:,:4] / [scale_factor_x, 1, scale_factor_x, 1] \n \n return img, bboxes.tolist()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:22:20.981984Z","iopub.execute_input":"2022-01-05T16:22:20.982249Z","iopub.status.idle":"2022-01-05T16:22:22.488019Z","shell.execute_reply.started":"2022-01-05T16:22:20.982218Z","shell.execute_reply":"2022-01-05T16:22:22.487162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"šŸ” Let's see an example of the original image and then the **sheared** one.","metadata":{}},{"cell_type":"code","source":"random.seed(25)\n\n# Translate\nshear = RandomShear(shear_factor=0.9) \nimg_sheared, annot_sheared = shear(img_original, annot_original)\n\n\n\n# Show the Before and After\nfig, axs = plt.subplots(1, 2, figsize=(23, 10))\naxs = axs.flatten()\nfig.suptitle(f\"(Random) Image Shear\", fontsize = 20)\n\naxs[0].set_title(\"Original Image\", fontsize = 20)\nshow_image_bbox(img_original, annot_original, axs=axs[0])\n\naxs[1].set_title(\"Sheared Image\", fontsize = 20)\nshow_image_bbox(img_sheared, annot_sheared, axs[1])\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:22:22.489628Z","iopub.execute_input":"2022-01-05T16:22:22.489966Z","iopub.status.idle":"2022-01-05T16:22:25.133432Z","shell.execute_reply.started":"2022-01-05T16:22:22.489923Z","shell.execute_reply":"2022-01-05T16:22:25.132259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### šŸ Log Augmented Images to W&B\n\nLet's now log the a sample of each augmentation to the W&B Dashboard.","metadata":{}},{"cell_type":"code","source":"# === šŸW&B Log (redone for formated annotations) ===\ndef wandb_bboxes(image, annotations):\n '''Source: https://www.kaggle.com/ayuraj/visualize-bounding-boxes-interactively\n image: the cv2.imread() output\n annotations: the FORMATED annotations from the train dataset'''\n \n all_annotations = []\n if annotations:\n for annot in annotations:\n data = {\"position\": {\n \"minX\": annot[0],\n \"minY\": annot[1],\n \"maxX\": annot[2],\n \"maxY\": annot[3]\n },\n \"class_id\" : 1,\n \"domain\" : \"pixel\"}\n all_annotations.append(data)\n \n return wandb.Image(image, \n boxes={\"ground_truth\": {\"box_data\": all_annotations}}\n )\n\n# Log all augmented images to the Dashboard\nwandb.log({\"flipped\": wandb_bboxes(img_flipped, annot_flipped)})\nwandb.log({\"scaled\": wandb_bboxes(img_scaled, annot_scaled)})\nwandb.log({\"translated\": wandb_bboxes(img_translated, annot_translated)})\nwandb.log({\"rotated\": wandb_bboxes(img_rotated, annot_rotated)})\nwandb.log({\"sheared\": wandb_bboxes(img_sheared, annot_sheared)})","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:22:25.135190Z","iopub.execute_input":"2022-01-05T16:22:25.135550Z","iopub.status.idle":"2022-01-05T16:22:28.428620Z","shell.execute_reply.started":"2022-01-05T16:22:25.135507Z","shell.execute_reply":"2022-01-05T16:22:28.427628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wandb.finish()","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:22:28.433112Z","iopub.execute_input":"2022-01-05T16:22:28.433404Z","iopub.status.idle":"2022-01-05T16:23:21.313441Z","shell.execute_reply.started":"2022-01-05T16:22:28.433371Z","shell.execute_reply":"2022-01-05T16:23:21.311968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Final Changes to Train datasets\n\nThis is the part where we create the last *helper features* for our dataset.\n\n## What is the COCO format?\n\nšŸ” As we have seen, an Object Detection model locates an object within an image using a **bounding box**. However, this bounding box can have multiple ways of being displayed, as there is no \"wrong\" way to locate a rectangle within an image:\n\n* `[x, y, width, height]` - this is the case in our training dataset (also called the COCO format).\n* `[x1, y1, x2, y2]` - the *formated* version we have created during the BBox Augmentation phase, also called `[xmin, ymin, xmax, ymax]`. This format is used within the [SSD/ RCNN/ Fast RCNN/ Faster RCNN models](https://lohithmunakala.medium.com/bounding-box-formats-for-models-like-yolo-ssd-rcnn-fast-rcnn-faster-rcnn-807be7721527).\n* `[x_center, y_center, width, height]` - this is the YOLO format, or rather the format used when training using the YOLO model. x_center, y_center are the normalized coordinates of the center of the bounding box and width, height are the normalized width and height of the image.\n\nšŸ” **COCO** comes from Common Objects in Context, which is a database that aims to support and improve models for Object Detection, Instance Segmentation and Image Captioning.","metadata":{}},{"cell_type":"code","source":"# Create sepparate paths for images and their labels (annotations)\n# these will come in handy later for the YOLO model\ntrain_df[\"path_images\"] = \"/kaggle/images/video_\" + train_df[\"video_id\"].astype(str) + \"_\" + \\\n train_df[\"video_frame\"].astype(str) + \".jpg\"\ntrain_df[\"path_labels\"] = \"/kaggle/labels/video_\" + train_df[\"video_id\"].astype(str) + \"_\" + \\\n train_df[\"video_frame\"].astype(str) + \".txt\"\n\n# Save the width and height of the images\n# it is the same for the entire dataset\ntrain_df[\"width\"] = 1280\ntrain_df[\"height\"] = 720\n\n# Simplify the annotation format\ntrain_df[\"coco_bbox\"] = train_df[\"annotations\"].apply(lambda annot: [list(item.values()) for item in eval(annot)])\n\n# Data Sample\ntrain_df.sample(5, random_state=24)","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:23:21.315273Z","iopub.execute_input":"2022-01-05T16:23:21.315514Z","iopub.status.idle":"2022-01-05T16:23:21.657953Z","shell.execute_reply.started":"2022-01-05T16:23:21.315481Z","shell.execute_reply":"2022-01-05T16:23:21.656980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save dataset\ntrain_df.to_csv(\"train.csv\", index=False)\n\n\n# šŸ Save dataset Artifact\nsave_dataset_artifact(run_name=\"save-train-data\",\n artifact_name=\"train_meta\",\n path=\"../input/2021-greatbarrierreef-prep-data/train.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-01-05T16:23:21.659296Z","iopub.execute_input":"2022-01-05T16:23:21.659527Z","iopub.status.idle":"2022-01-05T16:23:40.716532Z","shell.execute_reply.started":"2022-01-05T16:23:21.659493Z","shell.execute_reply":"2022-01-05T16:23:40.715300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n### šŸ W&B Dashboard\n\n> My W&B Dashboard is [here](https://wandb.ai/andrada/GreatBarrierReef/workspace?workspace=user-andrada).\n\n
\n\n
\n\n### My Specs\n\n* šŸ–„ Z8 G4 Workstation\n* šŸ’¾ 2 CPUs & 96GB Memory\n* šŸŽ® NVIDIA Quadro RTX 8000\n* šŸ’» Zbook Studio G7 on the go","metadata":{}}]}