{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Contents\n\n1. [About this notebook](#1)\n1. [Set up environment](#2)\n1. [Let's run detector!](#3)"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n# <div class=\"alert alert-block alert-success\">About this notebook</div>\n\nI will show an example of how to run [MegaDetector](https://github.com/microsoft/CameraTraps#models) on the kaggle notebook (of course with GPU!). \n\n## Motivation\n\nIf we check [Winning Solutions of Previous iWildcam Challenges (2018 , 2019 and 2020)](https://www.kaggle.com/c/iwildcam2021-fgvc8/discussion/225143), we can find that many solutions use Megadetector to crop animals from given dataset.\n\nIn order to detect which and how many animal species are present in an image, it is necessary to detect the animals in the given image. However, the annotation data given did not include the position of animals in the image, only categories. It is impossible to annotate all images by ourselves, but with MegaDetector we can do it!\n\n## Issue\n\nAccording to [Using the models](https://github.com/microsoft/CameraTraps/blob/master/megadetector.md#1-run_tf_detectorpy), tensorflow version specified to 1.13.1 (It should be a little wider, but it assumes at least tensorflow version 1.x.). But in kaggle notebook image now, tensorflow2.4.1 and CUDA10.2 are installed. So this caused tensorflow1.13.1 to make errors, and it was very difficult to set up the environment for execution. By the way, it can run on the CPU, but the processing speed is slow, so running it on the GPU was a must. Due to these reasons, users who do not have local GPU machines or are not familiar with the environment construction would have difficulty using MegaDetector, thus limiting the range of strategies."},{"metadata":{},"cell_type":"markdown","source":"### Note: License for MegaDetector\n\n**CameraTraps**\n\nCopyright (c) Microsoft Corporation. All rights reserved.\n\nReleased under the MIT license\n\nhttps://github.com/microsoft/CameraTraps/blob/master/LICENSE\n\n**ai4eutils**\n\nCopyright (c) Microsoft Corporation. All rights reserved.\n\nReleased under the MIT license\n\nhttps://github.com/microsoft/ai4eutils/blob/master/LICENSE"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2\"></a> <br>\n# <div class=\"alert alert-block alert-warning\">Set up environment</div> \n\nLet's set up environment following [megadetector.md](https://github.com/microsoft/CameraTraps/blob/master/megadetector.md)"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# required library\n!pip install humanfriendly","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!git clone https://github.com/microsoft/CameraTraps/\n!git clone https://github.com/microsoft/ai4eutils/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Weight for MegaDetector\n!wget https://lilablobssc.blob.core.windows.net/models/camera_traps/megadetector/md_v4.1.0/md_v4.1.0.pb","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import argparse\nimport glob\nimport os\nimport sys\nimport time\nimport warnings\n\nimport humanfriendly\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image, ImageFile, ImageFont, ImageDraw\nimport statistics\nimport tensorflow.compat.v1 as tf \ntf.disable_v2_behavior()\nfrom tqdm import tqdm\n%matplotlib inline\n\nfrom CameraTraps.ct_utils import truncate_float","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The documentation specifies to use tf1.3.1, but to make it work on the kaggle notebook, we will use tf.compat.v1 of version 2.4.1."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ignoring all \"PIL cannot read EXIF metainfo for the images\" warnings\nwarnings.filterwarnings('ignore', '(Possibly )?corrupt EXIF data', UserWarning)\n\n# Metadata Warning, tag 256 had too many entries: 42, expected 1\nwarnings.filterwarnings('ignore', 'Metadata warning', UserWarning)\n\n# Numpy FutureWarnings from tensorflow import\nwarnings.filterwarnings('ignore', category=FutureWarning)\n\nprint('TensorFlow version:', tf.__version__)\nprint('Is GPU available? tf.test.is_gpu_available:', tf.test.is_gpu_available())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I've ported the necessary code to make the model work on the kaggle notebook. For the sake of clarity of notebook, I'll let the cells be hidden. If you are interested, please open them and have a look. I had trial and error to do this, so please comment if you find unnecessary code or a better way."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# From https://github.com/microsoft/CameraTraps/blob/master/data_management/annotations/annotation_constants.py\n\nNUM_DETECTOR_CATEGORIES = 4  # this is for choosing colors, so ignoring the \"empty\" class\n\nbbox_categories = [\n    {'id': 0, 'name': 'empty'},\n    {'id': 1, 'name': 'animal'},\n    {'id': 2, 'name': 'person'},\n    {'id': 3, 'name': 'group'},  # group of animals\n    {'id': 4, 'name': 'vehicle'}\n]\n\nbbox_category_id_to_name = {}\nbbox_category_name_to_id = {}\n\nfor cat in bbox_categories:\n    bbox_category_id_to_name[cat['id']] = cat['name']\n    bbox_category_name_to_id[cat['name']] = cat['id']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# From https://github.com/microsoft/CameraTraps/blob/master/visualization/visualization_utils.py\n\nCOLORS = [\n        'AliceBlue', 'Red', 'RoyalBlue', 'Gold', 'Chartreuse', 'Aqua', 'Azure',\n        'Beige', 'Bisque', 'BlanchedAlmond', 'BlueViolet', 'BurlyWood', 'CadetBlue',\n        'AntiqueWhite', 'Chocolate', 'Coral', 'CornflowerBlue', 'Cornsilk', 'Crimson',\n        'Cyan', 'DarkCyan', 'DarkGoldenRod', 'DarkGrey', 'DarkKhaki', 'DarkOrange',\n        'DarkOrchid', 'DarkSalmon', 'DarkSeaGreen', 'DarkTurquoise', 'DarkViolet',\n        'DeepPink', 'DeepSkyBlue', 'DodgerBlue', 'FireBrick', 'FloralWhite',\n        'ForestGreen', 'Fuchsia', 'Gainsboro', 'GhostWhite', 'GoldenRod',\n        'Salmon', 'Tan', 'HoneyDew', 'HotPink', 'IndianRed', 'Ivory', 'Khaki',\n        'Lavender', 'LavenderBlush', 'LawnGreen', 'LemonChiffon', 'LightBlue',\n        'LightCoral', 'LightCyan', 'LightGoldenRodYellow', 'LightGray', 'LightGrey',\n        'LightGreen', 'LightPink', 'LightSalmon', 'LightSeaGreen', 'LightSkyBlue',\n        'LightSlateGray', 'LightSlateGrey', 'LightSteelBlue', 'LightYellow', 'Lime',\n        'LimeGreen', 'Linen', 'Magenta', 'MediumAquaMarine', 'MediumOrchid',\n        'MediumPurple', 'MediumSeaGreen', 'MediumSlateBlue', 'MediumSpringGreen',\n        'MediumTurquoise', 'MediumVioletRed', 'MintCream', 'MistyRose', 'Moccasin',\n        'NavajoWhite', 'OldLace', 'Olive', 'OliveDrab', 'Orange', 'OrangeRed',\n        'Orchid', 'PaleGoldenRod', 'PaleGreen', 'PaleTurquoise', 'PaleVioletRed',\n        'PapayaWhip', 'PeachPuff', 'Peru', 'Pink', 'Plum', 'PowderBlue', 'Purple',\n        'RosyBrown', 'Aquamarine', 'SaddleBrown', 'Green', 'SandyBrown',\n        'SeaGreen', 'SeaShell', 'Sienna', 'Silver', 'SkyBlue', 'SlateBlue',\n        'SlateGray', 'SlateGrey', 'Snow', 'SpringGreen', 'SteelBlue', 'GreenYellow',\n        'Teal', 'Thistle', 'Tomato', 'Turquoise', 'Violet', 'Wheat', 'White',\n        'WhiteSmoke', 'Yellow', 'YellowGreen'\n    ]\n\ndef render_detection_bounding_boxes(detections, image,\n                                    label_map={},\n                                    classification_label_map={},\n                                    confidence_threshold=0.8, thickness=4, expansion=0,\n                                    classification_confidence_threshold=0.3,\n                                    max_classifications=3):\n    \"\"\"\n    Renders bounding boxes, label, and confidence on an image if confidence is above the threshold.\n    This works with the output of the batch processing API.\n    Supports classification, if the detection contains classification results according to the\n    API output version 1.0.\n    Args:\n        detections: detections on the image, example content:\n            [\n                {\n                    \"category\": \"2\",\n                    \"conf\": 0.996,\n                    \"bbox\": [\n                        0.0,\n                        0.2762,\n                        0.1234,\n                        0.2458\n                    ]\n                }\n            ]\n            ...where the bbox coordinates are [x, y, box_width, box_height].\n            (0, 0) is the upper-left.  Coordinates are normalized.\n            Supports classification results, if *detections* have the format\n            [\n                {\n                    \"category\": \"2\",\n                    \"conf\": 0.996,\n                    \"bbox\": [\n                        0.0,\n                        0.2762,\n                        0.1234,\n                        0.2458\n                    ]\n                    \"classifications\": [\n                        [\"3\", 0.901],\n                        [\"1\", 0.071],\n                        [\"4\", 0.025]\n                    ]\n                }\n            ]\n        image: PIL.Image object, output of generate_detections.\n        label_map: optional, mapping the numerical label to a string name. The type of the numerical label\n            (default string) needs to be consistent with the keys in label_map; no casting is carried out.\n        classification_label_map: optional, mapping of the string class labels to the actual class names.\n            The type of the numerical label (default string) needs to be consistent with the keys in\n            label_map; no casting is carried out.\n        confidence_threshold: optional, threshold above which the bounding box is rendered.\n        thickness: line thickness in pixels. Default value is 4.\n        expansion: number of pixels to expand bounding boxes on each side.  Default is 0.\n        classification_confidence_threshold: confidence above which classification result is retained.\n        max_classifications: maximum number of classification results retained for one image.\n    image is modified in place.\n    \"\"\"\n\n    display_boxes = []\n    display_strs = []  # list of lists, one list of strings for each bounding box (to accommodate multiple labels)\n    classes = []  # for color selection\n\n    for detection in detections:\n\n        score = detection['conf']\n        if score >= confidence_threshold:\n\n            x1, y1, w_box, h_box = detection['bbox']\n            display_boxes.append([y1, x1, y1 + h_box, x1 + w_box])\n            clss = detection['category']\n            label = label_map[clss] if clss in label_map else clss\n            displayed_label = ['{}: {}%'.format(label, round(100 * score))]\n\n            if 'classifications' in detection:\n\n                # To avoid duplicate colors with detection-only visualization, offset\n                # the classification class index by the number of detection classes\n                clss = annotation_constants.NUM_DETECTOR_CATEGORIES + int(detection['classifications'][0][0])\n                classifications = detection['classifications']\n                if len(classifications) > max_classifications:\n                    classifications = classifications[0:max_classifications]\n                for classification in classifications:\n                    p = classification[1]\n                    if p < classification_confidence_threshold:\n                        continue\n                    class_key = classification[0]\n                    if class_key in classification_label_map:\n                        class_name = classification_label_map[class_key]\n                    else:\n                        class_name = class_key\n                    displayed_label += ['{}: {:5.1%}'.format(class_name.lower(), classification[1])]\n\n            # ...if we have detection results\n            display_strs.append(displayed_label)\n            classes.append(clss)\n\n        # ...if the confidence of this detection is above threshold\n\n    # ...for each detection\n    display_boxes = np.array(display_boxes)\n\n    draw_bounding_boxes_on_image(image, display_boxes, classes,\n                                 display_strs=display_strs, thickness=thickness, expansion=expansion)\n\n\ndef draw_bounding_boxes_on_image(image,\n                                 boxes,\n                                 classes,\n                                 thickness=4,\n                                 expansion=0,\n                                 display_strs=()):\n    \"\"\"\n    Draws bounding boxes on image.\n    Args:\n      image: a PIL.Image object.\n      boxes: a 2 dimensional numpy array of [N, 4]: (ymin, xmin, ymax, xmax).\n             The coordinates are in normalized format between [0, 1].\n      classes: a list of ints or strings (that can be cast to ints) corresponding to the class labels of the boxes.\n             This is only used for selecting the color to render the bounding box in.\n      thickness: line thickness in pixels. Default value is 4.\n      expansion: number of pixels to expand bounding boxes on each side.  Default is 0.\n      display_strs: list of list of strings.\n                             a list of strings for each bounding box.\n                             The reason to pass a list of strings for a\n                             bounding box is that it might contain\n                             multiple labels.\n    \"\"\"\n\n    boxes_shape = boxes.shape\n    if not boxes_shape:\n        return\n    if len(boxes_shape) != 2 or boxes_shape[1] != 4:\n        # print('Input must be of size [N, 4], but is ' + str(boxes_shape))\n        return  # no object detection on this image, return\n    for i in range(boxes_shape[0]):\n        if display_strs:\n            display_str_list = display_strs[i]\n            draw_bounding_box_on_image(image,\n                                       boxes[i, 0], boxes[i, 1], boxes[i, 2], boxes[i, 3],\n                                       classes[i],\n                                       thickness=thickness, expansion=expansion,\n                                       display_str_list=display_str_list)\n\n\ndef draw_bounding_box_on_image(image,\n                               ymin,\n                               xmin,\n                               ymax,\n                               xmax,\n                               clss=None,\n                               thickness=4,\n                               expansion=0,\n                               display_str_list=(),\n                               use_normalized_coordinates=True,\n                               label_font_size=16):\n    \"\"\"\n    Adds a bounding box to an image.\n    Bounding box coordinates can be specified in either absolute (pixel) or\n    normalized coordinates by setting the use_normalized_coordinates argument.\n    Each string in display_str_list is displayed on a separate line above the\n    bounding box in black text on a rectangle filled with the input 'color'.\n    If the top of the bounding box extends to the edge of the image, the strings\n    are displayed below the bounding box.\n    Args:\n    image: a PIL.Image object.\n    ymin: ymin of bounding box - upper left.\n    xmin: xmin of bounding box.\n    ymax: ymax of bounding box.\n    xmax: xmax of bounding box.\n    clss: str, the class of the object in this bounding box - will be cast to an int.\n    thickness: line thickness. Default value is 4.\n    expansion: number of pixels to expand bounding boxes on each side.  Default is 0.\n    display_str_list: list of strings to display in box\n        (each to be shown on its own line).\n        use_normalized_coordinates: If True (default), treat coordinates\n        ymin, xmin, ymax, xmax as relative to the image.  Otherwise treat\n        coordinates as absolute.\n    label_font_size: font size to attempt to load arial.ttf with\n    \"\"\"\n    if clss is None:\n        color = COLORS[1]\n    else:\n        color = COLORS[int(clss) % len(COLORS)]\n\n    draw = ImageDraw.Draw(image)\n    im_width, im_height = image.size\n    if use_normalized_coordinates:\n        (left, right, top, bottom) = (xmin * im_width, xmax * im_width,\n                                      ymin * im_height, ymax * im_height)\n    else:\n        (left, right, top, bottom) = (xmin, xmax, ymin, ymax)\n\n    if expansion > 0:\n        left -= expansion\n        right += expansion\n        top -= expansion\n        bottom += expansion\n\n    draw.line([(left, top), (left, bottom), (right, bottom),\n               (right, top), (left, top)], width=thickness, fill=color)\n\n    try:\n        font = ImageFont.truetype('arial.ttf', label_font_size)\n    except IOError:\n        font = ImageFont.load_default()\n\n    # If the total height of the display strings added to the top of the bounding\n    # box exceeds the top of the image, stack the strings below the bounding box\n    # instead of above.\n    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]\n\n    # Each display_str has a top and bottom margin of 0.05x.\n    total_display_str_height = (1 + 2 * 0.05) * sum(display_str_heights)\n\n    if top > total_display_str_height:\n        text_bottom = top\n    else:\n        text_bottom = bottom + total_display_str_height\n\n    # Reverse list and print from bottom to top.\n    for display_str in display_str_list[::-1]:\n        text_width, text_height = font.getsize(display_str)\n        margin = np.ceil(0.05 * text_height)\n\n        draw.rectangle(\n            [(left, text_bottom - text_height - 2 * margin), (left + text_width,\n                                                              text_bottom)],\n            fill=color)\n\n        draw.text(\n            (left + margin, text_bottom - text_height - margin),\n            display_str,\n            fill='black',\n            font=font)\n\n        text_bottom -= (text_height + 2 * margin)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# From https://github.com/microsoft/CameraTraps/blob/master/visualization/visualization_utils.py\n\ndef open_image(input_file):\n    \"\"\"\n    Opens an image in binary format using PIL.Image and convert to RGB mode. This operation is lazy; image will\n    not be actually loaded until the first operation that needs to load it (for example, resizing), so file opening\n    errors can show up later.\n    Args:\n        input_file: an image in binary format read from the POST request's body or\n            path to an image file (anything that PIL can open)\n    Returns:\n        an PIL image object in RGB mode\n    \"\"\"\n\n    image = Image.open(input_file)\n    if image.mode not in ('RGBA', 'RGB', 'L'):\n        raise AttributeError('Input image {} uses unsupported mode {}'.format(input_file, image.mode))\n    if image.mode == 'RGBA' or image.mode == 'L':\n        # PIL.Image.convert() returns a converted copy of this image\n        image = image.convert(mode='RGB')\n    return image\n\ndef load_image(input_file):\n    \"\"\"\n    Loads the image at input_file as a PIL Image into memory; Image.open() used in open_image() is lazy and\n    errors will occur downstream if not explicitly loaded\n    Args:\n        input_file: an image in binary format read from the POST request's body or\n            path to an image file (anything that PIL can open)\n    Returns:\n        an PIL image object in RGB mode\n    \"\"\"\n    image = open_image(input_file)\n    image.load()\n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# From https://github.com/microsoft/CameraTraps/blob/master/detection/run_tf_detector.py\n\nclass ImagePathUtils:\n    \"\"\"A collection of utility functions supporting this stand-alone script\"\"\"\n\n    # Stick this into filenames before the extension for the rendered result\n    DETECTION_FILENAME_INSERT = '_detections'\n\n    image_extensions = ['.jpg', '.jpeg', '.gif', '.png']\n\n    @staticmethod\n    def is_image_file(s):\n        \"\"\"\n        Check a file's extension against a hard-coded set of image file extensions    '\n        \"\"\"\n        ext = os.path.splitext(s)[1]\n        return ext.lower() in ImagePathUtils.image_extensions\n\n    @staticmethod\n    def find_image_files(strings):\n        \"\"\"\n        Given a list of strings that are potentially image file names, look for strings\n        that actually look like image file names (based on extension).\n        \"\"\"\n        return [s for s in strings if ImagePathUtils.is_image_file(s)]\n\n    @staticmethod\n    def find_images(dir_name, recursive=False):\n        \"\"\"\n        Find all files in a directory that look like image file names\n        \"\"\"\n        if recursive:\n            strings = glob.glob(os.path.join(dir_name, '**', '*.*'), recursive=True)\n        else:\n            strings = glob.glob(os.path.join(dir_name, '*.*'))\n\n        image_strings = ImagePathUtils.find_image_files(strings)\n\n        return image_strings","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# From https://github.com/microsoft/CameraTraps/blob/master/detection/run_tf_detector.py\n\nclass TFDetector:\n    \"\"\"\n    A detector model loaded at the time of initialization. It is intended to be used with\n    the MegaDetector (TF). The inference batch size is set to 1; code needs to be modified\n    to support larger batch sizes, including resizing appropriately.\n    \"\"\"\n\n    # Number of decimal places to round to for confidence and bbox coordinates\n    CONF_DIGITS = 3\n    COORD_DIGITS = 4\n\n    # MegaDetector was trained with batch size of 1, and the resizing function is a part\n    # of the inference graph\n    BATCH_SIZE = 1\n\n    # An enumeration of failure reasons\n    FAILURE_TF_INFER = 'Failure TF inference'\n    FAILURE_IMAGE_OPEN = 'Failure image access'\n\n    DEFAULT_RENDERING_CONFIDENCE_THRESHOLD = 0.85  # to render bounding boxes\n    DEFAULT_OUTPUT_CONFIDENCE_THRESHOLD = 0.1  # to include in the output json file\n\n    DEFAULT_DETECTOR_LABEL_MAP = {\n        '1': 'animal',\n        '2': 'person',\n        '4': 'vehicle'  # will be available in megadetector v4\n    }\n\n    NUM_DETECTOR_CATEGORIES = 4  # animal, person, group, vehicle - for color assignment\n\n    def __init__(self, model_path):\n        \"\"\"Loads the model at model_path and start a tf.Session with this graph. The necessary\n        input and output tensor handles are obtained also.\"\"\"\n        detection_graph = TFDetector.__load_model(model_path)\n        self.tf_session = tf.Session(graph=detection_graph)\n\n        self.image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')\n        self.box_tensor = detection_graph.get_tensor_by_name('detection_boxes:0')\n        self.score_tensor = detection_graph.get_tensor_by_name('detection_scores:0')\n        self.class_tensor = detection_graph.get_tensor_by_name('detection_classes:0')\n\n    @staticmethod\n    def round_and_make_float(d, precision=4):\n        return truncate_float(float(d), precision=precision)\n\n    @staticmethod\n    def __convert_coords(np_array):\n        \"\"\" Two effects: convert the numpy floats to Python floats, and also change the coordinates from\n        [y1, x1, y2, x2] to [x1, y1, width_box, height_box] (in relative coordinates still).\n        Args:\n            np_array: array of predicted bounding box coordinates from the TF detector\n        Returns: array of predicted bounding box coordinates as Python floats and in [x1, y1, width_box, height_box]\n        \"\"\"\n        # change from [y1, x1, y2, x2] to [x1, y1, width_box, height_box]\n        width_box = np_array[3] - np_array[1]\n        height_box = np_array[2] - np_array[0]\n\n        new = [np_array[1], np_array[0], width_box, height_box]  # cannot be a numpy array; needs to be a list\n\n        # convert numpy floats to Python floats\n        for i, d in enumerate(new):\n            new[i] = TFDetector.round_and_make_float(d, precision=TFDetector.COORD_DIGITS)\n        return new\n\n    @staticmethod\n    def __load_model(model_path):\n        \"\"\"Loads a detection model (i.e., create a graph) from a .pb file.\n        Args:\n            model_path: .pb file of the model.\n        Returns: the loaded graph.\n        \"\"\"\n        print('TFDetector: Loading graph...')\n        detection_graph = tf.Graph()\n        with detection_graph.as_default():\n            od_graph_def = tf.GraphDef()\n            with tf.gfile.GFile(model_path, 'rb') as fid:\n                serialized_graph = fid.read()\n                od_graph_def.ParseFromString(serialized_graph)\n                tf.import_graph_def(od_graph_def, name='')\n        print('TFDetector: Detection graph loaded.')\n\n        return detection_graph\n\n    def _generate_detections_one_image(self, image):\n        np_im = np.asarray(image, np.uint8)\n        im_w_batch_dim = np.expand_dims(np_im, axis=0)\n\n        # need to change the above line to the following if supporting a batch size > 1 and resizing to the same size\n        # np_images = [np.asarray(image, np.uint8) for image in images]\n        # images_stacked = np.stack(np_images, axis=0) if len(images) > 1 else np.expand_dims(np_images[0], axis=0)\n\n        # performs inference\n        (box_tensor_out, score_tensor_out, class_tensor_out) = self.tf_session.run(\n            [self.box_tensor, self.score_tensor, self.class_tensor],\n            feed_dict={self.image_tensor: im_w_batch_dim})\n\n        return box_tensor_out, score_tensor_out, class_tensor_out\n\n    def generate_detections_one_image(self, image, image_id,\n                                      detection_threshold=DEFAULT_OUTPUT_CONFIDENCE_THRESHOLD):\n        \"\"\"Apply the detector to an image.\n        Args:\n            image: the PIL Image object\n            image_id: a path to identify the image; will be in the `file` field of the output object\n            detection_threshold: confidence above which to include the detection proposal\n        Returns:\n        A dict with the following fields, see https://github.com/microsoft/CameraTraps/tree/siyu/inference_refactor/api/batch_processing#batch-processing-api-output-format\n            - image_id (always present)\n            - max_detection_conf\n            - detections, which is a list of detection objects containing `category`, `conf` and `bbox`\n            - failure\n        \"\"\"\n        result = {\n            'file': image_id\n        }\n        try:\n            b_box, b_score, b_class = self._generate_detections_one_image(image)\n\n            # our batch size is 1; need to loop the batch dim if supporting batch size > 1\n            boxes, scores, classes = b_box[0], b_score[0], b_class[0]\n\n            detections_cur_image = []  # will be empty for an image with no confident detections\n            max_detection_conf = 0.0\n            for b, s, c in zip(boxes, scores, classes):\n                if s > detection_threshold:\n                    detection_entry = {\n                        'category': str(int(c)),  # use string type for the numerical class label, not int\n                        'conf': truncate_float(float(s),  # cast to float for json serialization\n                                               precision=TFDetector.CONF_DIGITS),\n                        'bbox': TFDetector.__convert_coords(b)\n                    }\n                    detections_cur_image.append(detection_entry)\n                    if s > max_detection_conf:\n                        max_detection_conf = s\n\n            result['max_detection_conf'] = truncate_float(float(max_detection_conf),\n                                                          precision=TFDetector.CONF_DIGITS)\n            result['detections'] = detections_cur_image\n\n        except Exception as e:\n            result['failure'] = TFDetector.FAILURE_TF_INFER\n            print('TFDetector: image {} failed during inference: {}'.format(image_id, str(e)))\n\n        return result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I modified [load_and_run_detectors function](https://github.com/microsoft/CameraTraps/blob/master/detection/run_tf_detector.py) lightly to return the list of detected results. This function is responsible for performing the object detection. Also, the original implementation to save images with bbox  is retained. If we want to use it for creating datasets, we may want to tidy up the code for better performance."},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"# From https://github.com/microsoft/CameraTraps/blob/master/detection/run_tf_detector.py\n\ndef load_and_run_detector(model_file, image_file_names, output_dir,\n                          render_confidence_threshold=TFDetector.DEFAULT_RENDERING_CONFIDENCE_THRESHOLD):\n    if len(image_file_names) == 0:\n        print('Warning: no files available')\n        return\n\n    # load and run detector on target images, and visualize the results\n    start_time = time.time()\n    tf_detector = TFDetector(model_file)\n    elapsed = time.time() - start_time\n    print('Loaded model in {}'.format(humanfriendly.format_timespan(elapsed)))\n\n    detection_results = []\n    time_load = []\n    time_infer = []\n    detection_categories = []\n\n    # since we'll be writing a bunch of files to the same folder, rename\n    # as necessary to avoid collisions\n    output_file_names = {}\n\n    for im_file in tqdm(image_file_names):\n        try:\n            start_time = time.time()\n\n            image = load_image(im_file)\n\n            elapsed = time.time() - start_time\n            time_load.append(elapsed)\n            print(time_load)\n        except Exception as e:\n            print('Image {} cannot be loaded. Exception: {}'.format(im_file, e))\n            result = {\n                'file': im_file,\n                'failure': TFDetector.FAILURE_IMAGE_OPEN\n            }\n            detection_results.append(result)\n            continue\n\n        try:\n            start_time = time.time()\n\n            result = tf_detector.generate_detections_one_image(image, im_file)\n            \n            #print(\"Detection result is:\", result)\n            \n            detection_results.append(result)\n            \n            if result[\"detections\"] == []:\n                detection_categories.append(0)\n            else:\n                    \n                detection_categories.append(result[\"detections\"][0][\"category\"])\n\n            elapsed = time.time() - start_time\n            time_infer.append(elapsed)\n        except Exception as e:\n            print('An error occurred while running the detector on image {}. Exception: {}'.format(im_file, e))\n            # the error code and message is written by generate_detections_one_image,\n            # which is wrapped in a big try catch\n            continue\n\n        try:\n            # image is modified in place\n            render_detection_bounding_boxes(result['detections'], image,\n                                                      label_map=TFDetector.DEFAULT_DETECTOR_LABEL_MAP,\n                                                      confidence_threshold=render_confidence_threshold)\n            fn = os.path.basename(im_file).lower()\n            name, ext = os.path.splitext(fn)\n            fn = '{}{}{}'.format(name, ImagePathUtils.DETECTION_FILENAME_INSERT, '.jpg')  # save all as JPG\n            if fn in output_file_names:\n                n_collisions = output_file_names[fn]  # if there were a collision, the count is at least 1\n                fn = str(n_collisions) + '_' + fn\n                output_file_names[fn] = n_collisions + 1\n            else:\n                output_file_names[fn] = 0\n\n            output_full_path = os.path.join(output_dir, fn)\n            image.save(output_full_path)\n\n        except Exception as e:\n            print('Visualizing results on the image {} failed. Exception: {}'.format(im_file, e))\n            continue\n\n    ave_time_load = statistics.mean(time_load)\n    ave_time_infer = statistics.mean(time_infer)\n    if len(time_load) > 1 and len(time_infer) > 1:\n        std_dev_time_load = humanfriendly.format_timespan(statistics.stdev(time_load))\n        std_dev_time_infer = humanfriendly.format_timespan(statistics.stdev(time_infer))\n    else:\n        std_dev_time_load = 'not available'\n        std_dev_time_infer = 'not available'\n    print('On average, for each image,')\n    print('- loading took {}, std dev is {}'.format(humanfriendly.format_timespan(ave_time_load),\n                                                    std_dev_time_load))\n    print('- inference took {}, std dev is {}'.format(humanfriendly.format_timespan(ave_time_infer),\n                                                      std_dev_time_infer))\n    \n    return detection_results","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n# <div class=\"alert alert-block alert-info\">Let's run detector!</div>\n\nWith the environment in place, let's try processing a image."},{"metadata":{"trusted":true},"cell_type":"code","source":"# List images in test dataset\nimage_file_names = glob.glob(\"../input/iwildcam2021-fgvc8/test/*\")\n\n# Choice 100th image\nimage_file_names = image_file_names[100:101]\n\n# Model to use\nmodel_file = \"./md_v4.1.0.pb\"\n\n#output directory of processed images\noutput_dir = \"./result\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run detector."},{"metadata":{"trusted":true},"cell_type":"code","source":"detection_results = load_and_run_detector(model_file, image_file_names, output_dir, render_confidence_threshold=0.5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check result!"},{"metadata":{"trusted":true},"cell_type":"code","source":"detection_results","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see result with original image."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#Refered: https://www.kaggle.com/qinhui1999/how-to-use-bbox-for-iwildcam-2020 \n\ndef draw_bboxs(detections_list, im):\n    \"\"\"\n    detections_list: list of set includes bbox.\n    im: image read by Pillow.\n    \"\"\"\n    \n    for detection in detections_list:\n        x1, y1,w_box, h_box = detection[\"bbox\"]\n        ymin,xmin,ymax, xmax=y1, x1, y1 + h_box, x1 + w_box\n        draw = ImageDraw.Draw(im)\n        \n        imageWidth=im.size[0]\n        imageHeight= im.size[1]\n        (left, right, top, bottom) = (xmin * imageWidth, xmax * imageWidth,\n                                      ymin * imageHeight, ymax * imageHeight)\n        \n        draw.line([(left, top), (left, bottom), (right, bottom),\n               (right, top), (left, top)], width=4, fill='Red')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"size = (480,270)\nim = Image.open(\"../input/iwildcam2021-fgvc8/test/8d11eea8-21bc-11ea-a13a-137349068a90.jpg\")\nim = im.resize(size)\n\n# Overwrite bbox\ndraw_bboxs(detection_results[0]['detections'], im)\n\n# Show\nplt.imshow(im)\nplt.title(f\"image with bbox\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We got bbox information, so we can crop images of animal from each wildcam images."},{"metadata":{"trusted":true},"cell_type":"code","source":"x1, y1,w_box, h_box = detection_results[0]['detections'][0][\"bbox\"]\nymin,xmin,ymax, xmax = y1, x1, y1 + h_box, x1 + w_box\nimageWidth=im.size[0]\nimageHeight= im.size[1]\narea = (xmin * imageWidth, ymin * imageHeight, xmax * imageWidth,\n        ymax * imageHeight)\n\nout = im.crop(area)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(out)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}