{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Protecting the Great Barrier Reef \n\nIn this notebook, I will present an approach to object detection using deep computer vision, for the purpose of solving the Kaggle challenge by Tensorflow, to find the crown of throwns starfish damaging the Great Barrier Reef ecology.","metadata":{}},{"cell_type":"markdown","source":"## Imports","metadata":{"id":"5wuAfvt8gvy8"}},{"cell_type":"code","source":"!pip install --user webcolors  # used to convert RGB values to color names (in English)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:04:22.548109Z","iopub.execute_input":"2021-12-16T20:04:22.548544Z","iopub.status.idle":"2021-12-16T20:04:30.696304Z","shell.execute_reply.started":"2021-12-16T20:04:22.548508Z","shell.execute_reply":"2021-12-16T20:04:30.695429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import ast, os, sys\n\nimport matplotlib.pyplot as plt\nimport numpy as np  # array operations\nimport pandas as pd \nfrom PIL import Image  # image processing\nimport seaborn as sns  # another plotting library\nimport scipy  # scientific computing \nimport tensorflow as tf  # AI/ML\nimport webcolors ","metadata":{"id":"X8NJMKgqbfve","execution":{"iopub.status.busy":"2021-12-16T20:04:30.701495Z","iopub.execute_input":"2021-12-16T20:04:30.702032Z","iopub.status.idle":"2021-12-16T20:04:30.711419Z","shell.execute_reply.started":"2021-12-16T20:04:30.701993Z","shell.execute_reply":"2021-12-16T20:04:30.710755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot pretty figures\n%matplotlib inline\nsns.set_style('darkgrid')","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:04:30.716106Z","iopub.execute_input":"2021-12-16T20:04:30.716822Z","iopub.status.idle":"2021-12-16T20:04:30.727427Z","shell.execute_reply.started":"2021-12-16T20:04:30.716782Z","shell.execute_reply":"2021-12-16T20:04:30.726212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Double-check Versions","metadata":{"id":"ja73-dtbcPaJ"}},{"cell_type":"code","source":"print(sys.version)  # Python version\nprint(tf.__version__) # on Kaggle, this will be 2.6","metadata":{"id":"dMvucPqKcCiC","outputId":"10c2d0de-1110-425f-b190-05942ed89964","execution":{"iopub.status.busy":"2021-12-16T20:04:30.732877Z","iopub.execute_input":"2021-12-16T20:04:30.733536Z","iopub.status.idle":"2021-12-16T20:04:30.741512Z","shell.execute_reply.started":"2021-12-16T20:04:30.733499Z","shell.execute_reply":"2021-12-16T20:04:30.740714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploratory Data Analysis\n\nHere I will share what I've found interesting when exploring this dataset so far, since it might help contextualize the decisions I make later on, when implementing the machine learning models.\n\n*Abbreviations*:     \n- COTS = \"Crown-of-thorns starfish\"\n\nNote: for the `data_path` variable - I clicked on the \"copy\" button icon next to the dataset folder icon in the \"Data\" tab (on the Kaggle kernel).","metadata":{"id":"xgRbIFh0g3h0"}},{"cell_type":"code","source":"data_path = '../input/tensorflow-great-barrier-reef'\ndf = pd.read_csv(f\"{data_path}/train.csv\")","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:04:30.743632Z","iopub.execute_input":"2021-12-16T20:04:30.744975Z","iopub.status.idle":"2021-12-16T20:04:30.799330Z","shell.execute_reply.started":"2021-12-16T20:04:30.744938Z","shell.execute_reply":"2021-12-16T20:04:30.798624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head(56)  # show enough rows to see what annotations look like","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:04:30.801587Z","iopub.execute_input":"2021-12-16T20:04:30.801781Z","iopub.status.idle":"2021-12-16T20:04:30.819112Z","shell.execute_reply.started":"2021-12-16T20:04:30.801758Z","shell.execute_reply":"2021-12-16T20:04:30.818346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### How to Be an Observant Surveyor\n\nMy goal in this analysis is to build an object-detection system that can scale up the efforts of manual surveyors in the Great Barrier Reef. With that in mind... what makes a human surveyor great at spotting COTS in the first place?","metadata":{"id":"AkMbPlRLg7nT"}},{"cell_type":"markdown","source":"**Question 1**: Do the COTS tend to lump close together?\n\n*Part 1:* On average, how many COTS are seen together in a single video frame?\nTo do this, let's start by first adding a column with the counts of COTS seen in each particular frame:","metadata":{"id":"FW8-Q0t6hgxt"}},{"cell_type":"code","source":"type(df['annotations'][17])  # although the data type visually looks like a list, the CSV is all text","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:04:30.826497Z","iopub.execute_input":"2021-12-16T20:04:30.826682Z","iopub.status.idle":"2021-12-16T20:04:30.834675Z","shell.execute_reply.started":"2021-12-16T20:04:30.826659Z","shell.execute_reply":"2021-12-16T20:04:30.833787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We know there may be multiple COTS spotted in a single frame, so let's count up each that is spotted in a new column. We'll using the `{` to know how many COTS are in each: ","metadata":{}},{"cell_type":"code","source":"count_func = lambda string: string.count('{')\nspotted = df['annotations'].apply(count_func)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:04:30.836424Z","iopub.execute_input":"2021-12-16T20:04:30.836615Z","iopub.status.idle":"2021-12-16T20:04:30.858506Z","shell.execute_reply.started":"2021-12-16T20:04:30.836592Z","shell.execute_reply":"2021-12-16T20:04:30.857887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.assign(starfish_spotted=spotted)\ndf.head()","metadata":{"id":"NABW28cMhzMk","execution":{"iopub.status.busy":"2021-12-16T20:04:30.860219Z","iopub.execute_input":"2021-12-16T20:04:30.860496Z","iopub.status.idle":"2021-12-16T20:04:30.873840Z","shell.execute_reply.started":"2021-12-16T20:04:30.860463Z","shell.execute_reply":"2021-12-16T20:04:30.873149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Cool beans! Now we can calculate the average of COTS spotted in a given frame:","metadata":{}},{"cell_type":"code","source":"mean_starfish_spotted_in_a_frame = round(df[\"starfish_spotted\"].mean(), 4)\nprint(f\"On average, {mean_starfish_spotted_in_a_frame} COTS are seen together in a single video frame.\")","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:04:30.875031Z","iopub.execute_input":"2021-12-16T20:04:30.876625Z","iopub.status.idle":"2021-12-16T20:04:30.881534Z","shell.execute_reply.started":"2021-12-16T20:04:30.876585Z","shell.execute_reply":"2021-12-16T20:04:30.880782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Wowza, that seems very low. Let's also visualize the distribution of the `starfish_spotted` column using a histogram and PDF:","metadata":{}},{"cell_type":"code","source":"def plot_histogram_from_df(df, column, title, x_axis, y_axis):\n    \"\"\"\n    Plots the PDF of a column in a given DataFrame, using Matplotlib.\n    \n    Credit for the equation used for plotting the PDF goes to the NumPy documentation:\n        https://numpy.org/doc/stable/reference/random/generated/numpy.random.normal.html\n    \n    Args:\n        df(pandas.DataFrame)\n        column(str): name of the column being plotted\n        title(str), x_axis(str), y_axis(str): will be added to the plot\n        \n    Returns: None\n    \"\"\"\n    # A: calculate the mean and std dev of the column\n    mu, sigma = df[column].mean(), df[column].std()\n    # B: init the histogram\n    bin_edges, bins_probabilites, ignored = plt.hist(df[column], density=True)\n    # C: plot the PDF \n    plt.plot(bins_probabilites, 1/(sigma * np.sqrt(2 * np.pi)) *\n               np.exp(-(bins_probabilites - mu)**2 / (2 * sigma**2)),\n             linewidth=2, color='r')\n    # D: make the plot more presentable\n    plt.title(title)\n    plt.xlabel(x_axis)\n    plt.ylabel(y_axis)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:04:30.882844Z","iopub.execute_input":"2021-12-16T20:04:30.883304Z","iopub.status.idle":"2021-12-16T20:04:30.892080Z","shell.execute_reply.started":"2021-12-16T20:04:30.883258Z","shell.execute_reply":"2021-12-16T20:04:30.891149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_histogram_from_df(df, 'starfish_spotted', \"PDF of COTS Spotted in Video Frames\", \"No. of COTS\", \"Probability\")","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:04:30.893313Z","iopub.execute_input":"2021-12-16T20:04:30.893634Z","iopub.status.idle":"2021-12-16T20:04:31.151761Z","shell.execute_reply.started":"2021-12-16T20:04:30.893598Z","shell.execute_reply":"2021-12-16T20:04:31.151090Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One takeaway on this: the distribution of COTS per video frame is heavily skewed, and the majority of them have none at all. This reinforces the idea that we'll want to weigh the `recall` highly in evaluating the eventual model we build, so we can detect the relatively low number of COTS that exist per image.","metadata":{}},{"cell_type":"markdown","source":"*Part 2:* On average, how many video frames do we go without seeing any COTS in the provided videos?\n\nWith this question I am trying to get another measurement of how closely the groups of COTS are to one another. The approach which I'll take here is to gather a distribution of the numbers of frames that happen sequentially in the training data, in which there are zero COTS spotted.\n\nNote that one limitation of this approach is that certain frames might be of the same location on the Great Barrier Reef (since we don't know if the camera-person is always moving). Regardless, I think we'll go ahead with this approach anyway, since I believe it's reasonable to assume the camera is moving for most of the time in the giving videos; therefore, the amount of frames in between the time we spot any COTS is like a \"proxy\" for how close they are together.","metadata":{"id":"nO2Vr_fAhzk7"}},{"cell_type":"code","source":"def zero_sequence_lengths(a):\n    \"\"\"Compute the lengths of the sequences of consecutive zeros in an array.\n    \n    This is a modification of code by Warren Weckesser, originally posted on Stack Overflow:\n    https://stackoverflow.com/questions/24885092/finding-the-consecutive-zeros-in-a-numpy-array\n    \n    Example:\n        >>> a = np.array([[1, 2, 3, 0, 0, 0, 0, 0, 0, 4, 5, 6, 0, 0, 0, 0, 9, 8, 7, 0, 10, 11]])\n        >>> zero_sequence_lengths(a)\n            array([6, 4, 1])\n            \n    Args:\n        a(numpy.ndarray): 1-dimensional. Can have positive or negative numbers.\n        \n    Returns: ndarray. 1D array-like object.\n    \"\"\"\n    # A: Create an array that is 1 where a is 0, and pad each end with an extra 0\n    is_zero = np.concatenate(([0], np.equal(a, 0).view(np.int8), [0]))\n    # B: Zero out any of the\"in between\" 1's - only 1's at edges remain\n    ones_at_edges = np.abs(np.diff(is_zero))\n    # C: Get the indices of all the remaining 1's (the starts and ends)\n    sequences = np.where(ones_at_edges == 1)[0].reshape(-1, 2)\n    # D: Compute a 1D array with just the lengths of these sequences\n    return np.squeeze(np.diff(sequences, axis=1))","metadata":{"id":"imZE_hTuh-ji","execution":{"iopub.status.busy":"2021-12-16T20:04:31.152987Z","iopub.execute_input":"2021-12-16T20:04:31.153394Z","iopub.status.idle":"2021-12-16T20:04:31.160906Z","shell.execute_reply.started":"2021-12-16T20:04:31.153350Z","shell.execute_reply":"2021-12-16T20:04:31.160259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I will define an \"empty frame\" as one having no COTS, and use that to make the variable names more brief:","metadata":{}},{"cell_type":"code","source":"# Compute the lengths of these consecutive frames with zero COTS\nconsecutive_empty_frames = zero_sequence_lengths(df[\"starfish_spotted\"])\n# print the mean\navg_empty = round(consecutive_empty_frames.mean(), 1)\nprint(f\"There is average of {avg_empty} 'empty' frames in-between the video frames that contain COTS.\")","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:04:31.162078Z","iopub.execute_input":"2021-12-16T20:04:31.162405Z","iopub.status.idle":"2021-12-16T20:04:31.174231Z","shell.execute_reply.started":"2021-12-16T20:04:31.162362Z","shell.execute_reply":"2021-12-16T20:04:31.173470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So there's our answer! Out of curiosity, let's make a PDF from the distribution of these lengths:","metadata":{}},{"cell_type":"code","source":"def plot_histogram_from_arr(array, title, x_axis, y_axis):\n    \"\"\"\n    Plots the PDF of an array, using Matplotlib.\n    \n    Credit for the equation used for plotting the PDF goes to the NumPy documentation:\n        https://numpy.org/doc/stable/reference/random/generated/numpy.random.normal.html\n    \n    Args:\n        array(array-like object): 1-dimensional, has numerical values\n        title(str), x_axis(str), y_axis(str): will be added to the plot\n        \n    Returns: None\n    \"\"\"\n    # A: calculate the mean and std dev of the array\n    mu, sigma = array.mean(), array.std()\n    # B: init the histogram\n    bin_edges, bins_probabilites, ignored = plt.hist(array, density=True)\n    # C: plot the PDF \n    plt.plot(bins_probabilites, 1/(sigma * np.sqrt(2 * np.pi)) *\n               np.exp(-(bins_probabilites - mu)**2 / (2 * sigma**2)),\n             linewidth=2, color='r')\n    # D: make the plot more presentable\n    plt.title(title)\n    plt.xlabel(x_axis)\n    plt.ylabel(y_axis)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:04:31.175942Z","iopub.execute_input":"2021-12-16T20:04:31.176261Z","iopub.status.idle":"2021-12-16T20:04:31.183458Z","shell.execute_reply.started":"2021-12-16T20:04:31.176229Z","shell.execute_reply":"2021-12-16T20:04:31.182615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_histogram_from_arr(consecutive_empty_frames, \"PDF of Empty Video Frames\", \"No. of Frames\", \"Probability\")","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:04:31.185070Z","iopub.execute_input":"2021-12-16T20:04:31.185356Z","iopub.status.idle":"2021-12-16T20:04:31.478043Z","shell.execute_reply.started":"2021-12-16T20:04:31.185323Z","shell.execute_reply":"2021-12-16T20:04:31.477409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Given these plots, it might suggest that the COTS don't tend to grow near to each other, as the majority of empty sequences have a length that is between 0-1000. This also begs another question though: what percentage do empty video frames make out of the entire training dataset?","metadata":{}},{"cell_type":"code","source":"def percent_fmt(percent):\n    return f\"{round(percent, 3)}%\"\n\nall_empty = np.sum(consecutive_empty_frames)\nwedges = [all_empty, (df.size - all_empty)]\nplt.pie(wedges, labels=['Empty Frames', 'Non-empty Frames'], autopct=percent_fmt)\nplt.title(\"Percentages of Empty and Non-Empty Video Frames\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:04:31.479374Z","iopub.execute_input":"2021-12-16T20:04:31.479624Z","iopub.status.idle":"2021-12-16T20:04:31.571189Z","shell.execute_reply.started":"2021-12-16T20:04:31.479590Z","shell.execute_reply":"2021-12-16T20:04:31.570447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, should we be using a sequential model for this problem? The answer might be a very definite maybe.\n\nOn the one hand, there are some pretty large sequences of video frames without any COTS - this would suggest that they spread far out from each other.\n\nOn the other hand, we can see (from above) that about 88.7% of the video frames in the training dataset have at least 1 COTS. Therefore, the dataset is not exactly \"sparse\" for finding COTS.\n\nFor now, let's do a few more exploratory questions - and then later, it might be useful to try both an object detection model (e.g. YOLOv5) and some kind of sequential model for this problem.","metadata":{}},{"cell_type":"markdown","source":"**Question 2:** What are some \"giveaways\" that a certain object in an video frame is that of a COTS?\n\n- *Part 1:* What is the distribution of the observed colors of COTS in the videos?\n\nTo approach this question, we need to first figure out what we mean by \"observed color\" of a single COTS. For the code below, I decided to just take the pixel value that remains after the bounding box is downsampled to be just a single pixel, with RGB channels. This of course means it takes on the dimensions of `(3, 1, 1)`.","metadata":{"id":"a29qvuhOh-5A"}},{"cell_type":"code","source":"from scipy.spatial import KDTree\n\n\ndef rgb_to_color(rgb):\n    \"\"\"Finds the color that most closely names the given RGB value.\n    \n    Adapted from the code by Mir AbdulHaseeb at this link:\n    https://medium.com/codex/rgb-to-color-names-in-python-the-robust-way-ec4a9d97a01f\n    \n    Arg:\n        rgb(np.ndarray): 3 values in an array, one for each of the RGB channels\n        \n    Returns: str\n    \"\"\"\n    # A: map all the hex codes in CSS3 --> respective color names\n    css3_dict = webcolors.CSS3_HEX_TO_NAMES\n    color_names = []\n    rgb_values = []\n    for color_hex, color_name in css3_dict.items():\n        color_names.append(color_name)\n        rgb_values.append(webcolors.hex_to_rgb(color_hex))\n    # B: retrieve the closest matching color\n    kdt_db = KDTree(rgb_values)\n    _, index = kdt_db.query(rgb)\n    return color_names[index]\n    \n    # a dictionary of all the hex and their respective names in css3\n    css3_db = css3_hex_to_names\n    names = []\n    rgb_values = []\n    for color_hex, color_name in css3_dict.items():\n        names.append(color_name)\n        rgb_values.append(hex_to_rgb(color_hex))\n    \n    kdt_db = KDTree(rgb_values)\n    distance, index = kdt_db.query(rgb_tuple)\n    return names[index]\n\n\ndef get_downsampled_color(img, annotation):\n    \"\"\"Compute the downsampled color from a 3-dimensional array (of pixel values).\n    \n    To account for the variation of color within a single bounding box, \n    this function first downsamples the pixels representing the COTS down to a \n    single pixel across the RGB channels (using bicubic interpolation). \n    Then, it uses a KDTree to convert that single RGB pair to a color name.\n    \n    Args:\n        img(PIL.Image)\n        annotation(dict): contains the coords of the upper left corner of a single bounding box,\n                          along with its width and height\n    \n    Returns: ndarray of 3 n\n    \"\"\"\n    # A: get the pixels of just the COTS (using the bounding box)\n    upper_left_x, upper_left_y = (annotation['x'], annotation['y'])\n    width, height = annotation['width'],  annotation['height']\n    lower_right_x, lower_right_y = upper_left_x + width, upper_left_y + height\n    cots_pixels = img.crop((upper_left_x, upper_left_y, lower_right_x, lower_right_y))\n    # B: resize it to a single pixel (with 3 channels)\n    cots_pixel = np.array(cots_pixels.resize((1, 1), resample=Image.BICUBIC).getdata()).reshape(-1)\n    # C: return the name of this pixel\n    return rgb_to_color(cots_pixel)\n\n\ndef compute_cots_color_distribution(df, data_path):\n    \"\"\"Compute the distribution of colors shown by the COTS bounding boxes.\n    \n    By colors, we mean the \"downsampled\" color of each bounding box, \n    found using bicubic interpolation.\n    \n    Args: \n        df(pandas.DataFrame): the rows of the train.csv\n        data_path(str): the root of the dataset\n    \n    Returns: ndarray of strings\n    \"\"\"\n    # A: init a list\n    colors = list()\n    # B: traverse the DataFrame rows\n    for _, frame in df.iterrows():\n        f = frame\n        # B1: get the image\n        img_path = f\"{data_path}/train_images/video_{f['video_id']}/{f['video_frame']}.jpg\"\n        img = Image.open(img_path)\n        # B2: get the annotations of this image\n        annotations = a = ast.literal_eval(f[\"annotations\"])\n        # B3: for each annotation - get the mean color\n        colors.extend([get_downsampled_color(img, annotation) for annotation in a])\n    # C: return the list\n    return np.array(colors)","metadata":{"id":"_0yCuOIyi1L_","execution":{"iopub.status.busy":"2021-12-16T20:04:31.572682Z","iopub.execute_input":"2021-12-16T20:04:31.572929Z","iopub.status.idle":"2021-12-16T20:04:31.586646Z","shell.execute_reply.started":"2021-12-16T20:04:31.572896Z","shell.execute_reply":"2021-12-16T20:04:31.585799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Warning: the code below still needs to be vectorized! Its runtime is on the order of minutes not seconds (on the GPU).","metadata":{}},{"cell_type":"code","source":"colors = compute_cots_color_distribution(df, data_path)  ","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:04:31.588165Z","iopub.execute_input":"2021-12-16T20:04:31.588494Z","iopub.status.idle":"2021-12-16T20:10:07.757795Z","shell.execute_reply.started":"2021-12-16T20:04:31.588454Z","shell.execute_reply":"2021-12-16T20:10:07.757003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"colors","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:10:07.760641Z","iopub.execute_input":"2021-12-16T20:10:07.760881Z","iopub.status.idle":"2021-12-16T20:10:07.769750Z","shell.execute_reply.started":"2021-12-16T20:10:07.760856Z","shell.execute_reply":"2021-12-16T20:10:07.769055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def compute_mode_color(color_strings):\n    \"\"\"Computes the mode of the RGB colors of COTS.\"\"\"\n    mode, counts = scipy.stats.mode(color_strings, axis=None)\n    return mode[0]","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:10:07.772255Z","iopub.execute_input":"2021-12-16T20:10:07.773050Z","iopub.status.idle":"2021-12-16T20:10:07.777796Z","shell.execute_reply.started":"2021-12-16T20:10:07.772993Z","shell.execute_reply":"2021-12-16T20:10:07.777141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mode_color = compute_mode_color(colors)\nprint(f\"Most often, the COTS is a '{mode_color}' color.\")","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:10:07.778832Z","iopub.execute_input":"2021-12-16T20:10:07.779185Z","iopub.status.idle":"2021-12-16T20:10:07.796812Z","shell.execute_reply.started":"2021-12-16T20:10:07.779150Z","shell.execute_reply":"2021-12-16T20:10:07.795439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Because the color is now a categorical variable, we'll finally plot a bar chart to see its distribution.","metadata":{}},{"cell_type":"code","source":"from collections import Counter\n\ndef plot_histogram_from_strings(arr_strings, title, x_axis, y_axis):\n    \"\"\"\n    Plots the PDF of an array of strings using Matplotlib.\n    \n    Credit to the Matplotlib documentation for giving example of how to plot categorical variables:\n        https://matplotlib.org/stable/gallery/lines_bars_and_markers/categorical_variables.html#sphx-glr-gallery-lines-bars-and-markers-categorical-variables-py\n    \n    Args:\n        array(array-like object): 1-dimensional, has string values\n        title(str), x_axis(str), y_axis(str): will be added to the plot\n        \n    Returns: None\n    \"\"\"\n    # A: map each unique value to its frequency\n    string_counts = Counter(arr_strings)\n    # B: make the plot \n    fig, ax = plt.subplots(1, 1, figsize=(45, 12))\n    ax.bar(string_counts.keys(), string_counts.values())\n    plt.title(title)\n    plt.xlabel(x_axis)\n    plt.ylabel(y_axis)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:10:07.798203Z","iopub.execute_input":"2021-12-16T20:10:07.798501Z","iopub.status.idle":"2021-12-16T20:10:07.805116Z","shell.execute_reply.started":"2021-12-16T20:10:07.798443Z","shell.execute_reply":"2021-12-16T20:10:07.804321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_histogram_from_strings(colors, \"PDF of the Colors of COTS\", \"Color\", \"Probability\")","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:10:07.806550Z","iopub.execute_input":"2021-12-16T20:10:07.806820Z","iopub.status.idle":"2021-12-16T20:10:08.513734Z","shell.execute_reply.started":"2021-12-16T20:10:07.806787Z","shell.execute_reply":"2021-12-16T20:10:08.513075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- *Part 2:* What is the distribution of the observed sizes of COTS in the videos?\n\nFor this question, I will choose the area of the bounding box around each COTS to approximate their sizes:","metadata":{"id":"pc4Nmd6ei1pw"}},{"cell_type":"code","source":"def compute_cots_size_distribution(df):\n    \"\"\"Compute the distribution of sizes taken up by the COTS.\n    \n    By size, we mean the area of the bounding box, (width * height), \n    which both come from the \"annotations\" column.\n    \n    Args: \n        df(pandas.DataFrame): the rows of the train.csv\n    \n    Returns: ndarray of numerical values\n    \"\"\"\n    ### HELPER \n    def compute_area(box) -> float:\n        \"\"\"Returns area of bounding box found in the image.\"\"\"\n        return box[\"width\"] * box[\"height\"]\n    \n    ### MAIN\n    # A: init a list\n    sizes = list()\n    # B: traverse the DataFrame rows\n    for _, frame in df.iterrows():\n        f = frame\n        # B1: get the bounding boxes of this image\n        boxes = ast.literal_eval(f[\"annotations\"])\n        # B2: for each annotation - get the area\n        sizes.extend([compute_area(box) for box in boxes])\n    # C: return the list\n    return np.array(sizes)","metadata":{"id":"7oy5aiL1i4PH","execution":{"iopub.status.busy":"2021-12-16T20:10:08.514936Z","iopub.execute_input":"2021-12-16T20:10:08.515649Z","iopub.status.idle":"2021-12-16T20:10:08.522898Z","shell.execute_reply.started":"2021-12-16T20:10:08.515612Z","shell.execute_reply":"2021-12-16T20:10:08.522160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sizes = compute_cots_size_distribution(df)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:10:08.524851Z","iopub.execute_input":"2021-12-16T20:10:08.525324Z","iopub.status.idle":"2021-12-16T20:10:09.855818Z","shell.execute_reply.started":"2021-12-16T20:10:08.525277Z","shell.execute_reply":"2021-12-16T20:10:09.855096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sizes","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:10:09.857034Z","iopub.execute_input":"2021-12-16T20:10:09.857299Z","iopub.status.idle":"2021-12-16T20:10:09.864366Z","shell.execute_reply.started":"2021-12-16T20:10:09.857249Z","shell.execute_reply":"2021-12-16T20:10:09.863683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_histogram_from_arr(sizes, \"PDF of COTS Sizes\", \"Area of Bounding Box (sq. pixels)\", \"Probability\")","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:10:09.865524Z","iopub.execute_input":"2021-12-16T20:10:09.866026Z","iopub.status.idle":"2021-12-16T20:10:10.121830Z","shell.execute_reply.started":"2021-12-16T20:10:09.865990Z","shell.execute_reply":"2021-12-16T20:10:10.121157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Woah - this PDF is heavily skewed. Instead of using the same `plot_histogram_from_arr()` function from before, let's look deeper into the range of values, and see if there might be outliers:","metadata":{}},{"cell_type":"code","source":"plt.boxplot(sizes)\nplt.title(\"Box Plot of of COTS Sizes\")\nplt.ylabel(\"Area of Bounding Box (sq. pixels)\")\nplt.xlabel(\"Videos 0-2\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:10:10.122931Z","iopub.execute_input":"2021-12-16T20:10:10.123375Z","iopub.status.idle":"2021-12-16T20:10:10.302847Z","shell.execute_reply.started":"2021-12-16T20:10:10.123336Z","shell.execute_reply":"2021-12-16T20:10:10.302113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Yup. Looks like although the majority of bounding box sizes fall in a relatively small range, the overall range of sizes is huge. Perhaps there are some close-up shots of the COTS in the data?\n\nTo get a more accurate picture of this distribution, let's plot the PDF again after removing outliers (based on the IQR):","metadata":{}},{"cell_type":"code","source":"def find_remove_outlier_iqr(counts):\n    \"\"\"Find and delete the outliers from a dataset.\n    \n    Here, we assume the data is not normally distributed. \n    There we use IQR to find the outliers - specifically, by \n    taking out any values more than ((1.5 * IQR) + the 75th percentile), or \n    less than (the 25th percentile - 1.5 * IQR).\n    \n    Args:\n        counts(np.ndarray) - presumbly an array of numerical values\n        \n    Returns: Tuple[np.ndarray] - 2 new arrays:\n        a new array without any outliers\n        a new array containing the outliers that were removed\n    \"\"\"\n    # A: calculate interquartile range\n    q25, q75 = np.percentile(counts, 25), np.percentile(counts, 75)\n    iqr = q75 - q25\n    print(f\"The IQR of this array is: {iqr}.\")\n    # B: calculate the outlier thresholds\n    cut_off = iqr * 1.5\n    lower, upper = q25 - cut_off, q75 + cut_off\n    # C: identify outliers\n    outliers = counts[np.logical_or(counts.any() < lower, counts.any() > upper)]\n    # D: remove outliers\n    outliers_removed = counts[np.logical_and(lower < counts, counts < upper)]\n    return outliers_removed, outliers","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:10:10.304294Z","iopub.execute_input":"2021-12-16T20:10:10.304567Z","iopub.status.idle":"2021-12-16T20:10:10.310992Z","shell.execute_reply.started":"2021-12-16T20:10:10.304531Z","shell.execute_reply":"2021-12-16T20:10:10.310359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sizes_no_outliers, _ = find_remove_outlier_iqr(sizes)","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:10:10.312250Z","iopub.execute_input":"2021-12-16T20:10:10.312767Z","iopub.status.idle":"2021-12-16T20:10:10.323025Z","shell.execute_reply.started":"2021-12-16T20:10:10.312727Z","shell.execute_reply":"2021-12-16T20:10:10.322270Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_histogram_from_arr(sizes_no_outliers, \"PDF of COTS Sizes\", \"Area of Bounding Box (sq. pixels)\", \"Probability\")","metadata":{"execution":{"iopub.status.busy":"2021-12-16T20:10:10.324397Z","iopub.execute_input":"2021-12-16T20:10:10.324927Z","iopub.status.idle":"2021-12-16T20:10:10.558481Z","shell.execute_reply.started":"2021-12-16T20:10:10.324763Z","shell.execute_reply":"2021-12-16T20:10:10.557824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*Much* better! So now we can see that the area of the bounding boxes containing the COTS is relatively small for the most part, with most having somewhere between 1,000 to 2,500 square pixels.","metadata":{}},{"cell_type":"markdown","source":"## Modelling [TODO]","metadata":{"id":"9D4YhwbDceiK"}},{"cell_type":"markdown","source":"**Step 1:** Now let's load the YOLOv5 model from the PyTorch Hub. Note, there is also an implementation of this model [available on GitHub](https://github.com/ultralytics/yolov5/blob/c1249a47c7fe19e2067cb25ed8347e67d26ff1f1/models/tf.py#L323), which uses Tensorflow 2.x and is primarily authored by Jiacong Fang. ","metadata":{"id":"U7Jxmx77pST0"}},{"cell_type":"code","source":"!git clone https://github.com/ultralytics/yolov5  # clone\n%cd yolov5\n%pip install -qr requirements.txt  # install","metadata":{"id":"_ElWlt5Vofxl","outputId":"ce560c94-6b08-4a41-ccbd-08657a34cfbf"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls","metadata":{"id":"EYVbFjR8f9Q5","outputId":"f113fb3e-e21f-4da4-8966-08279040c827"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import utils\ndisplay = utils.notebook_init()  # running the checks","metadata":{"id":"Ux8otJbFsE4d","outputId":"0c05a743-77a8-4fab-d02f-9223a3c4827a"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Step 2:** Now, let export to a `SavedModel` (this is using the \"nano\" version of YOLOv5, the smallest). Note that only need to do this *once*:","metadata":{"id":"OGbp452cpp1C"}},{"cell_type":"code","source":"!python export.py --weights yolov5n.pt --include saved_model","metadata":{"id":"FM_p2nh2ptm2","outputId":"c7103709-d637-4086-f0e6-28f4f329ab42"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Step 4:** Now, let's test loading the model back in via Tensorflow 2:","metadata":{"id":"h1TK9CS0tWZZ"}},{"cell_type":"code","source":"tf_model = tf.keras.models.load_model('./yolov5n_saved_model')","metadata":{"id":"5CNlkCvPsfrp","outputId":"37e6a138-5052-404f-98f2-8d24ac586db7"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Woohoo! We got a warning that the model has no training configuration, but that is to be expected (since we want to do all the training code henceforth ourselves, using Tensorflow code). \n\nLet's get started!","metadata":{"id":"gm9_kzUBuINz"}},{"cell_type":"code","source":"","metadata":{"id":"3zARnYg9keEb"},"execution_count":null,"outputs":[]}]}