{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/open-images-2019-object-detection","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Introduction to Tensorflow Hub for Object Detection (with SSD+MobileNetV2)\n\nThe purpose of this kernel is to walk you through the process of downloading a pre-trained TF model from the Tensorflow Hub, building a Tensorflow graph and performing off-the-shelf predictions. This kernel is a fork of the excellent [baseline kernel by Vikram](https://www.kaggle.com/vikramtiwari/baseline-predictions-using-inception-resnet-v2), but differs on those points:\n1. We will be using MobileNet v2 instead of Inception-ResNet. This means that the inference will be faster*, but the accuracy ends up worse.\n2. This will focus on understanding how Tensorflow works, rather than providing a boilerplate. For a simple solution, please check out [this tutorial](https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/object_detection.ipynb), which is what Vikram's kernel was inspired from.\n\n\\* Each image takes roughly 0.2s to process, whereas Inception-ResNet takes around 1.2s.\n\n## Changelog\n\n* V6: As pointed out by [Nicolas in his kernel](https://www.kaggle.com/nhlr21/tf-hub-bounding-boxes-coordinates-corrected/notebook) and in this [discussion thread](https://www.kaggle.com/c/open-images-2019-object-detection/discussion/98205), the coordinates are inverted between the Kaggle competition and the original TF Hub graph. This version corrects this problem. Please go give them an upvote; this is a really good catch from them!\n\n## References\n\n* Vikram's Original Kernel: https://www.kaggle.com/vikramtiwari/baseline-predictions-using-inception-resnet-v2\n* TFHub Demo: https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/object_detection.ipynb\n* TFHub Model link: https://tfhub.dev/google/openimages_v4/ssd/mobilenet_v2/1\n* Discussion on inveted BBox: https://www.kaggle.com/c/open-images-2019-object-detection/discussion/98205\n* Corrected BBox: https://www.kaggle.com/nhlr21/tf-hub-bounding-boxes-coordinates-corrected/notebook"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import os\nfrom pprint import pprint\nfrom six import BytesIO\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nimport tensorflow_hub as hub\nfrom PIL import Image, ImageColor, ImageDraw, ImageFont, ImageOps\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Utility Functions\n\nIn this section, we define a few functions that will be used for processing images and formatting the output prediction. You can safely skip this section and use the following functions as is:\n* `format_prediction_string(image_id, result)`: `image_id` is the ID of the test image you are trying to label. `result` is the dictionary created from running a `tf.Session`. The output is a formatted output row (i.e. `{Label Confidence XMin YMin XMax YMax},{...}`), so we need to modify the order from Tensorflow, which is by default `YMin XMin YMax XMax` (Thanks to [Nicolas for discovering this](https://www.kaggle.com/nhlr21/tf-hub-bounding-boxes-coordinates-corrected/notebook)).\n* `draw_boxes(image, boxes, class_names, scores, max_boxes=10, min_score=0.1)`: `image` is a numpy array representing an image, `boxes`, `class_names`, and `scores` are directly retrieved from the model predictions.\n* `display_image(image)`: Display a numpy array representing an `image`."},{"metadata":{"trusted":true},"cell_type":"code","source":"def format_prediction_string(image_id, result):\n    prediction_strings = []\n    \n    for i in range(len(result['detection_scores'])):\n        class_name = result['detection_class_names'][i].decode(\"utf-8\")\n        YMin,XMin,YMax,XMax = result['detection_boxes'][i]\n        score = result['detection_scores'][i]\n        \n        prediction_strings.append(\n            f\"{class_name} {score} {XMin} {YMin} {XMax} {YMax}\"\n        )\n        \n    prediction_string = \" \".join(prediction_strings)\n\n    return {\n        \"ImageID\": image_id,\n        \"PredictionString\": prediction_string\n    }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_image(image):\n    fig = plt.figure(figsize=(20, 15))\n    plt.grid(False)\n    plt.axis('off')\n    plt.imshow(image)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Unhide the cell below to see how the intermediate function `draw_bounding_box_on_image` is constructed."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def draw_bounding_box_on_image(image,\n                               ymin,\n                               xmin,\n                               ymax,\n                               xmax,\n                               color,\n                               font,\n                               thickness=4,\n                               display_str_list=()):\n    \"\"\"Adds a bounding box to an image.\"\"\"\n    draw = ImageDraw.Draw(image)\n    im_width, im_height = image.size\n    (left, right, top, bottom) = (xmin * im_width, xmax * im_width,\n                                  ymin * im_height, ymax * im_height)\n    draw.line([(left, top), (left, bottom), (right, bottom), (right, top),\n               (left, top)],\n              width=thickness,\n              fill=color)\n\n    # If the total height of the display strings added to the top of the bounding\n    # box exceeds the top of the image, stack the strings below the bounding box\n    # instead of above.\n    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]\n    # Each display_str has a top and bottom margin of 0.05x.\n    total_display_str_height = (1 + 2 * 0.05) * sum(display_str_heights)\n\n    if top > total_display_str_height:\n        text_bottom = top\n    else:\n        text_bottom = bottom + total_display_str_height\n    # Reverse list and print from bottom to top.\n    for display_str in display_str_list[::-1]:\n        text_width, text_height = font.getsize(display_str)\n        margin = np.ceil(0.05 * text_height)\n        draw.rectangle([(left, text_bottom - text_height - 2 * margin),\n                        (left + text_width, text_bottom)],\n                       fill=color)\n        draw.text((left + margin, text_bottom - text_height - margin),\n                  display_str,\n                  fill=\"black\",\n                  font=font)\n        text_bottom -= text_height - 2 * margin","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def draw_boxes(image, boxes, class_names, scores, max_boxes=10, min_score=0.1):\n    \"\"\"Overlay labeled boxes on an image with formatted scores and label names.\"\"\"\n    colors = list(ImageColor.colormap.values())\n\n    try:\n        font = ImageFont.truetype(\n            \"/usr/share/fonts/truetype/liberation/LiberationSansNarrow-Regular.ttf\",\n            25)\n    except IOError:\n        print(\"Font not found, using default font.\")\n        font = ImageFont.load_default()\n\n    for i in range(min(boxes.shape[0], max_boxes)):\n        if scores[i] >= min_score:\n            ymin, xmin, ymax, xmax = tuple(boxes[i].tolist())\n            display_str = \"{}: {}%\".format(class_names[i].decode(\"ascii\"),\n                                           int(100 * scores[i]))\n            color = colors[hash(class_names[i]) % len(colors)]\n            image_pil = Image.fromarray(np.uint8(image)).convert(\"RGB\")\n            draw_bounding_box_on_image(\n                image_pil,\n                ymin,\n                xmin,\n                ymax,\n                xmax,\n                color,\n                font,\n                display_str_list=[display_str])\n            np.copyto(image, np.array(image_pil))\n    return image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Understanding the model"},{"metadata":{},"cell_type":"markdown","source":"We will be using a Single Shot MultiBox Detector (SSD) model with a MobileNet v2 as the backbone (SSD+MobileNetV2). If you are not familiar with the literature, check out [this article about SSD](https://towardsdatascience.com/review-ssd-single-shot-detector-object-detection-851a94607d11), and [this post explaining what's new with MobileNet v2](https://towardsdatascience.com/mobilenetv2-inverted-residuals-and-linear-bottlenecks-8a4362f4ffd5). On a high level, you can think of MobileNet as a lightweight CNN that extract features from the image, and SSD as a method to efficiently scale a set of default bounding boxes around the targets.\n\nThe model is trained on [Open Images V4](https://ai.googleblog.com/2018/04/announcing-open-images-v4-and-eccv-2018.html), which is the dataset used for last year's competition. Fortunately, the labels are still the same, so the outputs of the model can be directly submitted to this competition. Here's what the author says about the dataset:\n\n> Today, we are happy to announce Open Images V4, containing 15.4M bounding-boxes for 600 categories on 1.9M images, making it the largest existing dataset with object location annotations. The boxes have been largely manually drawn by professional annotators to ensure accuracy and consistency. The images are very diverse and often contain complex scenes with several objects (8 per image on average; visualizer). \n\nFor our implementation, it is important to note those following points:\n* This model does NOT support fine-tuning.\n* The model does NOT support batching, so the input has to be **ONE** image of shape `(1, height, width, 3)`.\n* It is recommended to run this module on GPU to get acceptable inference times.\n* The model is loaded directly from Tensorflow Hub, so this code might not work offline."},{"metadata":{},"cell_type":"markdown","source":"## Running the model on a Sample Image\n\nLet's start by running the model on a single image. We will go through each step of the process afterwards."},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"sample_image_path = \"../input/open-images-2019-object-detection/test/6beb79b52308112d.jpg\"\n\nwith tf.Graph().as_default():\n    # Create our inference graph\n    image_string_placeholder = tf.placeholder(tf.string)\n    decoded_image = tf.image.decode_jpeg(image_string_placeholder)\n    decoded_image_float = tf.image.convert_image_dtype(\n        image=decoded_image, dtype=tf.float32\n    )\n    # Expanding image from (height, width, 3) to (1, height, width, 3)\n    image_tensor = tf.expand_dims(decoded_image_float, 0)\n\n    # Load the model from tfhub.dev, and create a detector_output tensor\n    model_url = \"https://tfhub.dev/google/openimages_v4/ssd/mobilenet_v2/1\"\n    detector = hub.Module(model_url)\n    detector_output = detector(image_tensor, as_dict=True)\n    \n    # Initialize the Session\n    init_ops = [tf.global_variables_initializer(), tf.tables_initializer()]\n    sess = tf.Session()\n    sess.run(init_ops)\n\n    # Load our sample image into a binary string\n    with tf.gfile.Open(sample_image_path, \"rb\") as binfile:\n        image_string = binfile.read()\n\n    # Run the graph we just created\n    result_out, image_out = sess.run(\n        [detector_output, decoded_image],\n        feed_dict={image_string_placeholder: image_string}\n    )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see what it looks like:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"image_with_boxes = draw_boxes(\n    np.array(image_out), result_out[\"detection_boxes\"],\n    result_out[\"detection_class_entities\"], result_out[\"detection_scores\"]\n)\ndisplay_image(image_with_boxes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Step 1: Building a TF Graph\n\nLet's take a look at at how we are building the graph:\n\n```python\nimage_string_placeholder = tf.placeholder(tf.string)\ndecoded_image = tf.image.decode_jpeg(image_string_placeholder)\ndecoded_image_float = tf.image.convert_image_dtype(\n    image=decoded_image, dtype=tf.float32\n)\nimage_tensor = tf.expand_dims(decoded_image_float, 0)\n```\n\nFor those unfamiliar with the the dataflow paradigm, check out this [official guide](https://www.tensorflow.org/guide/graphs). Basically, whenever you use Tensorflow, you need to build and run a graph using the building blocks (i.e. the nodes of the graph) provided by the API. In our case, our model simply consists of the following blocks:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(image_string_placeholder)\nprint(decoded_image)\nprint(decoded_image_float)\nprint(image_tensor)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Each of those `tf.Tensor`s are passed to the next one, creating a [directed acyclic graph (DAG)](https://en.wikipedia.org/wiki/Directed_acyclic_graph), where the \"entry point\" is the variable `image_string_placeholder`.\n\nNote that afterwards, we are feeding `image_tensor` to `detector`:\n```python\nmodel_url = \"https://tfhub.dev/google/openimages_v4/ssd/mobilenet_v2/1\"\ndetector = hub.Module(model_url)\ndetector_output = detector(image_tensor, as_dict=True)\n```\n\nIn this case, `detector_output` is not a `tf.Tensor`, but a dictionary of `tf.Tensor`s:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"pprint(detector_output)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"However, this is not an issue since it will be accepted by `tf.Session`.\n\nAt this point, all we have is the \"blueprint\" of our model, but it has not been \"produced\" yet; that part is covered in Step 2 and 3."},{"metadata":{},"cell_type":"markdown","source":"## Step 2: Initializing the Session\n\n*If you are familiar with context managers and tf.Graph/tf.Session, feel free to skip this part.*\n\nIf you look at any piece of Tensorflow code, you will always see something along the lines of:\n\n```python\nwith tf.Graph().as_default():\n    with tf.Session() as sess:\n        ...\n        sess.run(...)\n        ...\n```\n\nBut what does the `with` keyword truly do? It lets you use what is called a *context manager*. You can often see it being used for opening documents, e.g.\n\n```python\nwith open('data.txt') as f:\n    print(f.read())\n```\n\nBasically, you can use a variable (in this case, `f`, which stands for file) in a constrained scope, as well as perform actions before and after the `with` code block is executed. For example, the built-in function `open()` will automatically perform a `f.close()` after you are done using it. [This tutorial](https://jeffknupp.com/blog/2016/03/07/python-with-context-managers/) goes more in-depth about how this works, and show how to build your own context manager.\n\nSo how does `tf.Session()` work then? According to the [official docs](https://www.tensorflow.org/api_docs/python/tf/Session), the following two examples are equivalent:\n\n```python\n# Using the `close()` method.\nsess = tf.Session()\nsess.run(...)\nsess.close()\n\n# Using the context manager.\nwith tf.Session() as sess:\n    sess.run(...)\n```\n\nThis means you can either create a context in which `sess` is active, or simply initialize `sess` and close it when you are done with all your Tensorflow computation. In our code, we use the second method:\n\n```python\ninit_ops = [tf.global_variables_initializer(), tf.tables_initializer()]\nsess = tf.Session()\nsess.run(init_ops)\n```\n\nIf you are confused about why we have those `init_ops` and need to run them before using `sess` for inference, check out this [StackOverflow thread](https://stackoverflow.com/questions/44433438/understanding-tf-global-variables-initializer). The [official docs section on variable initialization](https://www.tensorflow.org/guide/variables#initializing_variables) also gives a straightforward explanation:\n> Before you can use a variable, it must be initialized. If you are programming in the low-level TensorFlow API (that is, you are explicitly creating your own graphs and sessions), you must explicitly initialize the variables. Most high-level frameworks such as [...] Keras automatically initialize variables for you before training a model. \n> \n> Explicit initialization is otherwise useful because it allows you not to rerun potentially expensive initializers when reloading a model from a checkpoint as well as allowing determinism when randomly-initialized variables are shared in a distributed setting.\n> \n> To initialize all trainable variables in one go, before training starts, call tf.global_variables_initializer(). This function returns a single operation responsible for initializing all variables in the tf.GraphKeys.GLOBAL_VARIABLES collection. Running this operation initializes all variables.\n\nAs for `tf.tables_initializer()`, [this thread](https://stackoverflow.com/questions/54540018/what-does-tensorflows-tables-initializer-do) briefly explains why you need to use it, and the [official docs has a guide](https://www.tensorflow.org/guide/low_level_intro) showing `tf.tables_initializer()` being used in a simple example. For our purpose, think of it as being needed for initializing this particular type of graph, **and that it needs to be run after we load our model from TF Hub, or else you will run into a `FailedPreconditionError`.**\n\nLet's go back to `tf.Graph().as_default()`. The [Tensorflow docs on `tf.Graph()`](https://www.tensorflow.org/api_docs/python/tf/Graph) says:\n\n> [A] typical usage involves the `tf.Graph.as_default` context manager, which overrides the current default graph for the lifetime of the context:\n\n```python\ng = tf.Graph()\nwith g.as_default():\n    # Define operations and tensors in `g`.\n    c = tf.constant(30.0)\n    assert c.graph is g\n```\n\nThey also note:\n\n> This method should be used if you want to create multiple graphs in the same process. For convenience, a global default graph is provided, and all ops will be added to this graph if you do not create a new graph explicitly.\n\nSince we are using a notebook, it is good practice to call `as_default()`, since we might want to create other `tf.Graph` later on.\n\nOne last thing you might have noticed: we are not closing `sess` anywhere. This is because we will need to use the same session later to perform the inference on the entire test set."},{"metadata":{},"cell_type":"markdown","source":"## Step 3: Running the session\n\nAt this step, all there is left to do is to run `sess`:\n```python\n# Load our sample image into a binary string\nwith tf.gfile.Open(sample_image_path, \"rb\") as binfile:\n    image_string = binfile.read()\n\n# Run the graph we just created\nresult_out, image_out = sess.run(\n    [detector_output, decoded_image],\n    feed_dict={image_string_placeholder: image_string}\n)\n```\n\nThe [API docs](https://www.tensorflow.org/api_docs/python/tf/Session#run) indicates that `sess.run()` takes as input \n\n> * fetches: A single graph element, a list of graph elements, or a dictionary whose values are graph elements or lists of graph elements (described above).\n> * feed_dict: A dictionary that maps graph elements to values.\n\nIn our case, the `fetches` is `[detector_output, decoded_image]`, which are respectively the tensors containing the output dictionary with the predicted labels/boxes, and the image that we are currently decoding. Once `sess.run(...)` is computed, the tensors are resolved into a dictionary with the actual predictions, and the actual image (as a `np.array`):"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"result_out keys:\", result_out.keys())\nprint(\"First 10 detection scores:\", result_out['detection_scores'][:10])\nprint()\nprint(\"Shape of image_out\", image_out.shape)\nprint(\"Type of image_out:\", type(image_out))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference on Test Set"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission_df = pd.read_csv('../input/open-images-2019-object-detection/sample_submission.csv')\nimage_ids = sample_submission_df['ImageId']\npredictions = []\n\nfor image_id in tqdm(image_ids):\n    # Load the image string\n    image_path = f'../input/open-images-2019-object-detection/test/{image_id}.jpg'\n    with tf.gfile.Open(image_path, \"rb\") as binfile:\n        image_string = binfile.read()\n    \n    # Run our session\n    result_out = sess.run(\n        detector_output,\n        feed_dict={image_string_placeholder: image_string}\n    )\n    predictions.append(format_prediction_string(image_id, result_out))\n\nsess.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note above that `fetches` is not `[detector_output, decoded_image]` anymore, but instead only `detector_output`, since we are not interested in seeing what `decoded_image`/`image_out` contain anymore. `sess.run` accepts both as input a list of tensors, or a single tensor.\n\nYou can also notice that we are safely closing our `sess`, now that we have process all the images."},{"metadata":{},"cell_type":"markdown","source":"# Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_df = pd.DataFrame(predictions)\npred_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"import pandas as pd\nsample_submission = pd.read_csv(\"../input/open-images-2019-object-detection/sample_submission.csv\")","execution_count":0,"outputs":[]},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"import pandas as pd\nsample_submission = pd.read_csv(\"../input/open-images-2019-object-detection/sample_submission.csv\")","execution_count":0,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}