{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# General Description\nThis notebook implements a simple door detection method. We use the pretrained inception_resnet_v2 model which has been trained on the Open Images V4 dataset. This is a link to the Open Images V6 website where you can explore the dataset: https://storage.googleapis.com/openimages/web/visualizer/index.html?set=train&type=detection&c=%2Fm%2F02dgv. The Open Images V4 dataset is also available for download from that link.\n\nFor door detection on Google streetview images it is not enough to simply make predictions using inception_resnet_v2. The door makes up too small of a portion of the image and is thus ignored by the model. Instead we detect doors by first detecting a house in the image. If multiple houses exist in the image we use the house whose bounding box has the largest area. The reasoning behind this is that a house with a larger bounding box is closer to the camera and is more likely to contain the door we are interested in. Once a target house is identified we zoom into that portion of the image. Since doors are often small and occluded this zoomed in portion of the image is still too large for the model to detect a door. We next crop out small sections from this zoomed in image of the house and prediction on them. These sections are small enough to detect the door.\n\nHowever, these cropped sections must overlap to avoid cropping the door in half. This means that we will likely have multiple croppings which contain the door. To choose the best one we create a centrality metric. The cropping whose prediction is most central will be considered the best cropping and we will use the prediction which arises from this cropping as our final cropping.","metadata":{}},{"cell_type":"markdown","source":"# General Notes\n- This notebook is a modified verision of this code: https://www.kaggle.com/xhlulu/intro-to-tf-hub-for-object-detection\n- The images with bounding boxes overlayed do not show all of the model's predictions. It only shows the predictions with a confidence above a user-specificed threshold\n- get_croppings has a useful debug mode parameter. When active it will display additional information, the zoomed target image, and the cropping themselves. Otherwise nothing is displayed\n- predict_on_directory has three verbosity settings (0, 1, 2). These are useful for debugging and understanding where and how the model is making mistakes\n- It takes ~5mins to predict on the entire test set with a GPU and >1h with a CPU. This time can be significantly improved without great effort","metadata":{}},{"cell_type":"markdown","source":"# Improvement Notes\n- Batch computation is not supported so the gpu is not being fully utilized\n- The network currently takes strings in binary format as input. This should probably be changed to something more\n  convenient, like an array. This can be done by modifying the decoding step\n- Only using the door prediction which arises from the best cropping allows for the possibility of false positives. If we   instead consider the loction of the door (relative to the original image) in each cropping, we can get most consistent     results\n- When there are multiple houses in the image the wrong house is sometimes selected. Using a weighted average of: centrality, bounding box area, and prediction confidence will likely provide a good solution to this problem. Another option to fix this problem is to use GPS data. We can accurately select which house we want to target by knowing the camera's position and orientation. If we use GPS data we can also get multiple images of each house. Meaning we can get multiple predictions for multiple ground height calculations, which will result in a more accurate final calculation.\n- There is an excessive amount of repeated computation between on the cropped image section. This can be avoided by using a fully convolutional architecture. Or alternatively the sliding window size of the model (the object size which the model can detect) can be lowered to match the size of a door.","metadata":{}},{"cell_type":"markdown","source":"# Results","metadata":{}},{"cell_type":"markdown","source":"#### Clarification about the definition of accuracy within this notebook\nIn the tests below accuracy refers to the number of times a door was identified in an image. This does NOT mean that the correct door was found or that the bounding box for the door is correct! The reason we use this metric is because we do not have access to bounding box labled test data. Also, the pretrained model we are using generally provides great bounding box predictions, thus if a door is detected it is likely both the correct door and it likely has a good bounding box.","metadata":{}},{"cell_type":"markdown","source":"#### Results on the test set\nWe test the door detection algorithm on a test set of **24 images** named \"big-streetview-pics.\" These images are screenshots from google images. Some of the image have easy to detect doors, but there are a many images where the door is not easy to detect. However, that being said, it is reasonable to expect that a good model can detect a door in every image.\n\nThe algorithm gets a score of **58%**. However, in **9 images, (38%)** of the failure cases it failed because it selected the wrong house as the target house (the house which we zoom into). I believe this problem can be almost completely fixed by following the improvement note written above about target house selection. Additionally, the cropped sections that the model preforms predictions on are very blurry. I was unable to download the full resolution images from Google street view thus the dataset uses lower resolution screenshots. I think that if the full resolution images are used the accuracy will be greatly improved.\n\nTo test this claim I created a new dataset named \"big-streetview-pics-zoomed\" which contains the same houses as in the original test set, but I zoomed in to the correct house before taking a screenshot. This dataset simulates the algorithm selecting the correct house (which I think is very achievable) and simulates having a higher resolution image. On this new dataset the algorithm gets a score of **96%**. Therefore, we see that increasing resolution and selecting the correct target house greatly improves performance.\n\nBut this is a misleadingly high accuracy. As mentioned above the accuracy only refers to the number of times a door was detected in an image. It relies on the assumption that because the pretrained model is good, whenever a door is detected it is a good prediction. However, looking at the results we see that is not the case.\n\nThe model found a door in **23/24** of the images. In the one case the where the model was unable to identify a door, the door was only visible through the leaves of a tree. But out of the remaining images where a door was found it still made mistakes. In **1** of the images it identified a side door instead of the front door. This can be fixed by tweeking the method by which we select our final bounding box from among all the bounding boxes found within all the croppings. Finally, the model was confused by **5** images. In these images it mislabled objects as doors. The common mislabled objects were garage doors and windows. However, in most of these images it was still able to identify the door, it just so happened the that the mislabled object had a better centrality score causing it be selected as the best prediction. This can be fixed by editing the process we use to select our final door bounding box and by tuning the model to better distinguish between doors, garage doors, and windows. Note that the Open Images V4 dataset which was used to train this model contained very few training examples of house front doors. The doors in its training data were often very unusual (weird side ally doors, doors on ancient buildings, etc). Thus, some fine tuning on common house front doors will likely provide a good improvement.","metadata":{}},{"cell_type":"markdown","source":"#### Conclusion\nIn conclusion, I believe that this is a good method for detecting doors. It is not production ready but it can be greatly improved without too much effort. I believe that this approach is still worthwhile even if the model is modified (as mentions in the improvement notes) to be able to detect smaller object without the use of zooming or cropping. This is because I believe that by zooming into the target house we are greatly lowering the number of predictions we are considering and are thus greatly lowering the chances of getting a false positive door detection. Ideally, we would be able to modify the model to detect smaller objects and we would simply ignore all predictions which are not contained within the target house's bounding box. However, this is practically identical to our current method since both the challenge to idendentify the correct target house and the challenge to identify the correct door bounding box remain. The only difference would be that predictions on the bounding boxes are done simultaneously and without excess computations. Thus, it would provide a significant speed improvement.","metadata":{}},{"cell_type":"code","source":"import os\nfrom pprint import pprint\nfrom six import BytesIO\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nimport tensorflow_hub as hub\nfrom PIL import Image, ImageColor, ImageDraw, ImageFont, ImageOps\nfrom tqdm import tqdm","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-04T03:45:08.492450Z","iopub.execute_input":"2022-01-04T03:45:08.493239Z","iopub.status.idle":"2022-01-04T03:45:10.417412Z","shell.execute_reply.started":"2022-01-04T03:45:08.493177Z","shell.execute_reply":"2022-01-04T03:45:10.416581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import keras\nimport math","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:45:10.419151Z","iopub.execute_input":"2022-01-04T03:45:10.419488Z","iopub.status.idle":"2022-01-04T03:45:10.499268Z","shell.execute_reply.started":"2022-01-04T03:45:10.419369Z","shell.execute_reply":"2022-01-04T03:45:10.498503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import base64","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:45:10.500553Z","iopub.execute_input":"2022-01-04T03:45:10.500997Z","iopub.status.idle":"2022-01-04T03:45:10.504740Z","shell.execute_reply.started":"2022-01-04T03:45:10.500946Z","shell.execute_reply":"2022-01-04T03:45:10.503907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Utility Functions\n\nIn this section, we define a few functions that will be used for processing images and formatting the output prediction. You can safely skip this section and use the following functions as is:\n* `format_prediction_string(image_id, result)`: `image_id` is the ID of the test image you are trying to label. `result` is the dictionary created from running a `tf.Session`. The output is a formatted output row (i.e. `{Label Confidence XMin YMin XMax YMax},{...}`), so we need to modify the order from Tensorflow, which is by default `YMin XMin YMax XMax` (Thanks to [Nicolas for discovering this](https://www.kaggle.com/nhlr21/tf-hub-bounding-boxes-coordinates-corrected/notebook)).\n* `draw_boxes(image, boxes, class_names, scores, max_boxes=10, min_score=0.1)`: `image` is a numpy array representing an image, `boxes`, `class_names`, and `scores` are directly retrieved from the model predictions.\n* `display_image(image)`: Display a numpy array representing an `image`.","metadata":{}},{"cell_type":"code","source":"def format_prediction_string(image_id, result):\n    prediction_strings = []\n    \n    for i in range(len(result['detection_scores'])):\n        class_name = result['detection_class_names'][i].decode(\"utf-8\")\n        YMin,XMin,YMax,XMax = result['detection_boxes'][i]\n        score = result['detection_scores'][i]\n        \n        prediction_strings.append(\n            f\"{class_name} {score} {XMin} {YMin} {XMax} {YMax}\"\n        )\n        \n    prediction_string = \" \".join(prediction_strings)\n\n    return {\n        \"ImageID\": image_id,\n        \"PredictionString\": prediction_string\n    }","metadata":{"execution":{"iopub.status.busy":"2022-01-04T05:26:36.691317Z","iopub.execute_input":"2022-01-04T05:26:36.691783Z","iopub.status.idle":"2022-01-04T05:26:36.698420Z","shell.execute_reply.started":"2022-01-04T05:26:36.691570Z","shell.execute_reply":"2022-01-04T05:26:36.697650Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def display_image(image):\n    fig = plt.figure(figsize=(20, 15))\n    plt.grid(False)\n    plt.axis('off')\n    plt.imshow(image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Unhide the cell below to see how the intermediate function `draw_bounding_box_on_image` is constructed.","metadata":{}},{"cell_type":"code","source":"def draw_bounding_box_on_image(image,\n                               ymin,\n                               xmin,\n                               ymax,\n                               xmax,\n                               color,\n                               font,\n                               thickness=4,\n                               display_str_list=()):\n    \"\"\"Adds a bounding box to an image.\"\"\"\n    draw = ImageDraw.Draw(image)\n    im_width, im_height = image.size\n    (left, right, top, bottom) = (xmin * im_width, xmax * im_width,\n                                  ymin * im_height, ymax * im_height)\n    draw.line([(left, top), (left, bottom), (right, bottom), (right, top),\n               (left, top)],\n              width=thickness,\n              fill=color)\n\n    # If the total height of the display strings added to the top of the bounding\n    # box exceeds the top of the image, stack the strings below the bounding box\n    # instead of above.\n    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]\n    # Each display_str has a top and bottom margin of 0.05x.\n    total_display_str_height = (1 + 2 * 0.05) * sum(display_str_heights)\n\n    if top > total_display_str_height:\n        text_bottom = top\n    else:\n        text_bottom = bottom + total_display_str_height\n    # Reverse list and print from bottom to top.\n    for display_str in display_str_list[::-1]:\n        text_width, text_height = font.getsize(display_str)\n        margin = np.ceil(0.05 * text_height)\n        draw.rectangle([(left, text_bottom - text_height - 2 * margin),\n                        (left + text_width, text_bottom)],\n                       fill=color)\n        draw.text((left + margin, text_bottom - text_height - margin),\n                  display_str,\n                  fill=\"black\",\n                  font=font)\n        text_bottom -= text_height - 2 * margin","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-04T03:45:10.527405Z","iopub.execute_input":"2022-01-04T03:45:10.527884Z","iopub.status.idle":"2022-01-04T03:45:10.542121Z","shell.execute_reply.started":"2022-01-04T03:45:10.527836Z","shell.execute_reply":"2022-01-04T03:45:10.541273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def draw_boxes(image, boxes, class_names, scores, max_boxes=10, min_score=0.1):\n    \"\"\"Overlay labeled boxes on an image with formatted scores and label names.\"\"\"\n    colors = list(ImageColor.colormap.values())\n\n    try:\n        \n        #was originally simply set to 25\n        #I used height instead of width because generally the images are wider than they are long,\n        #but our cropped sections are squares\n        font_size = math.ceil(image.shape[0] /20)\n        \n        font = ImageFont.truetype(\n            \"/usr/share/fonts/truetype/liberation/LiberationSansNarrow-Regular.ttf\",\n            font_size)\n    except IOError:\n        print(\"Font not found, using default font.\")\n        font = ImageFont.load_default()\n\n    for i in range(min(boxes.shape[0], max_boxes)):\n        if scores[i] >= min_score:\n            ymin, xmin, ymax, xmax = tuple(boxes[i].tolist())\n            display_str = \"{}: {}%\".format(class_names[i].decode(\"ascii\"),\n                                           int(100 * scores[i]))\n            color = colors[hash(class_names[i]) % len(colors)]\n            image_pil = Image.fromarray(np.uint8(image)).convert(\"RGB\")\n            draw_bounding_box_on_image(\n                image_pil,\n                ymin,\n                xmin,\n                ymax,\n                xmax,\n                color,\n                font,\n                display_str_list=[display_str])\n            np.copyto(image, np.array(image_pil))\n    return image","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:45:10.543491Z","iopub.execute_input":"2022-01-04T03:45:10.543978Z","iopub.status.idle":"2022-01-04T03:45:10.556383Z","shell.execute_reply.started":"2022-01-04T03:45:10.543929Z","shell.execute_reply":"2022-01-04T03:45:10.555763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Understanding the model","metadata":{}},{"cell_type":"markdown","source":"We will be using a Single Shot MultiBox Detector (SSD) model with a MobileNet v2 as the backbone (SSD+MobileNetV2). If you are not familiar with the literature, check out [this article about SSD](https://towardsdatascience.com/review-ssd-single-shot-detector-object-detection-851a94607d11), and [this post explaining what's new with MobileNet v2](https://towardsdatascience.com/mobilenetv2-inverted-residuals-and-linear-bottlenecks-8a4362f4ffd5). On a high level, you can think of MobileNet as a lightweight CNN that extract features from the image, and SSD as a method to efficiently scale a set of default bounding boxes around the targets.\n\nThe model is trained on [Open Images V4](https://ai.googleblog.com/2018/04/announcing-open-images-v4-and-eccv-2018.html), which is the dataset used for last year's competition. Fortunately, the labels are still the same, so the outputs of the model can be directly submitted to this competition. Here's what the author says about the dataset:\n\n> Today, we are happy to announce Open Images V4, containing 15.4M bounding-boxes for 600 categories on 1.9M images, making it the largest existing dataset with object location annotations. The boxes have been largely manually drawn by professional annotators to ensure accuracy and consistency. The images are very diverse and often contain complex scenes with several objects (8 per image on average; visualizer). \n\nFor our implementation, it is important to note those following points:\n* This model does NOT support fine-tuning.\n* The model does NOT support batching, so the input has to be **ONE** image of shape `(1, height, width, 3)`.\n* It is recommended to run this module on GPU to get acceptable inference times.\n* The model is loaded directly from Tensorflow Hub, so this code might not work offline.","metadata":{}},{"cell_type":"markdown","source":"## Running the model on a Sample Image\n\nLet's start by running the model on a single image. We will go through each step of the process afterwards.","metadata":{}},{"cell_type":"code","source":"sample_image_path = \"../input/d/andreimuresanu/jpgstreetviewimages/51_prince_philip.jpg\"\n\nwith tf.Graph().as_default():\n    #the decoding step should probably be changed to use numpy or something\n    \n    # Create our inference graph\n    image_string_placeholder = tf.placeholder(tf.string)\n    decoded_image = tf.image.decode_jpeg(image_string_placeholder)\n    decoded_image_float = tf.image.convert_image_dtype(\n        image=decoded_image, dtype=tf.float32\n    )\n    # Expanding image from (height, width, 3) to (1, height, width, 3)\n    image_tensor = tf.expand_dims(decoded_image_float, 0)\n\n    # Load the model from tfhub.dev, and create a detector_output tensor\n    #model_url = \"https://tfhub.dev/google/openimages_v4/ssd/mobilenet_v2/1\"\n    model_url = \"https://tfhub.dev/google/faster_rcnn/openimages_v4/inception_resnet_v2/1\"\n    detector = hub.Module(model_url)\n    detector_output = detector(image_tensor, as_dict=True)\n    \n    # Initialize the Session\n    init_ops = [tf.global_variables_initializer(), tf.tables_initializer()]\n    sess = tf.Session()\n    sess.run(init_ops)\n\n    # Load our sample image into a binary string\n    with tf.gfile.Open(sample_image_path, \"rb\") as binfile:\n        image_string = binfile.read()\n\n    # Run the graph we just created\n    result_out, image_out = sess.run(\n        [detector_output, decoded_image],\n        feed_dict={image_string_placeholder: image_string}\n    )","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-04T03:45:10.559569Z","iopub.execute_input":"2022-01-04T03:45:10.559808Z","iopub.status.idle":"2022-01-04T03:47:13.245854Z","shell.execute_reply.started":"2022-01-04T03:45:10.559766Z","shell.execute_reply":"2022-01-04T03:47:13.244971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see what it looks like:","metadata":{}},{"cell_type":"code","source":"image_with_boxes = draw_boxes(\n    np.array(image_out), result_out[\"detection_boxes\"],\n    result_out[\"detection_class_entities\"], result_out[\"detection_scores\"]\n)\ndisplay_image(image_with_boxes)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-01-04T03:47:13.247578Z","iopub.execute_input":"2022-01-04T03:47:13.247894Z","iopub.status.idle":"2022-01-04T03:47:14.165976Z","shell.execute_reply.started":"2022-01-04T03:47:13.247845Z","shell.execute_reply":"2022-01-04T03:47:14.165185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Crop/zoom utility functions","metadata":{}},{"cell_type":"code","source":"def visible_print(text):\n    print(\"||||||||||\", text, \"||||||||||\")","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:47:14.167060Z","iopub.execute_input":"2022-01-04T03:47:14.167307Z","iopub.status.idle":"2022-01-04T03:47:14.172007Z","shell.execute_reply.started":"2022-01-04T03:47:14.167270Z","shell.execute_reply":"2022-01-04T03:47:14.171271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def angry_print(text):\n    print(\"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\")\n    print(text)\n    print(\"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\")","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:47:14.173302Z","iopub.execute_input":"2022-01-04T03:47:14.173783Z","iopub.status.idle":"2022-01-04T03:47:14.182304Z","shell.execute_reply.started":"2022-01-04T03:47:14.173736Z","shell.execute_reply":"2022-01-04T03:47:14.181625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def show_image(image, size=None):\n    '''\n    image is an array\n    size is a tuple (width, height)\n    '''\n    if size is not None: plt.rcParams[\"figure.figsize\"] = (size[0], size[1])\n        \n    plt.imshow(image, interpolation='nearest')\n    plt.axis(\"off\")\n    \n    plt.show()\n    \n    if size is not None: plt.rcParams['figure.figsize'] = plt.rcParamsDefault['figure.figsize'] #resetting the plot size","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:47:14.183572Z","iopub.execute_input":"2022-01-04T03:47:14.184075Z","iopub.status.idle":"2022-01-04T03:47:14.198879Z","shell.execute_reply.started":"2022-01-04T03:47:14.184025Z","shell.execute_reply":"2022-01-04T03:47:14.198216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def getCrops(base_image, predictions, shift=0.3, height=0.7, min_score=0.1, debug=False):\n    '''\n    the cropped sections will be square\n    base_image is a numpy array\n    \n    predictions['detection_boxes'][i] is in the format: ymin, xmin, ymax, xmax\n    and every value is a percentage\n    \n    when debug is true it will display additional information, the zoomed target image, and the cropping themselves\n    otherwise nothing is displayed\n    '''\n    \n    #we will only have a single target class\n    target_class = \"House\".encode('utf-8') #this is the class we will zoom into\n    \n    if target_class in predictions[\"detection_class_entities\"]:\n        if debug: print(\"base_image\")\n        if debug: show_image(base_image)\n\n        #we will only zoom into the target_class which takes up the most area\n        largest_area = -1\n        largest_area_idx = None\n        for i in range(len(predictions['detection_boxes'])):\n            if (predictions['detection_class_entities'][i] == target_class\n                    and predictions[\"detection_scores\"][i] >= min_score):\n                ymin, xmin, ymax, xmax = predictions['detection_boxes'][i]\n                cur_area = (ymax - ymin) * (xmax - xmin)\n                if cur_area > largest_area:\n                    largest_area = cur_area\n                    largest_area_idx = i\n        \n        if largest_area_idx is None:\n            raise Exception(target_class.decode('utf-8') + \" was not found with a confidence above \" + str(min_score))\n        \n        if debug: print(\"largest_area\", largest_area, \"largest_area_idx\", largest_area_idx)\n        \n        ymin, xmin, ymax, xmax = predictions['detection_boxes'][largest_area_idx]\n        target_image = np.array(base_image[int(ymin*base_image.shape[0]):int(ymax*base_image.shape[0]),\n                                           int(xmin*base_image.shape[1]):int(xmax*base_image.shape[1]),\n                                           :]) #[y, x, z]\n        \n        if debug: print(\"target_image, shape:\", target_image.shape)\n        if debug: show_image(target_image)\n        \n        #target_image.shape := [height, width, colour channel count] \n        target_h = target_image.shape[0]\n        target_w = target_image.shape[1]\n        \n        crop_h = int(target_h * height) #cropping height\n        shift_size = int(shift * crop_h)\n    \n        crop_count = int((target_w - target_h * height) // shift_size)\n    \n        if debug:\n            print(\"cropped images\")\n            print(\"cropping height:\", crop_h)\n            print(\"shift size:\", shift_size)\n            print(crop_count, \"croppings\")\n        \n        crop_img_list = []\n        for i in range(crop_count):\n            cur_xmin = i * shift_size\n            cur_xmax = cur_xmin + crop_h\n            \n            if debug: print(\"target_image\", target_image.shape)\n            if debug: print(\"xmin\", cur_xmin, \"xmax\", cur_xmax)\n            \n            crop_img = np.array(target_image[int((1-height)*target_image.shape[0]):, cur_xmin:cur_xmax, :]) #[y, x, z]\n\n            crop_img_list.append(crop_img)\n            if debug: show_image(crop_img)\n            \n        return crop_img_list, (ymin, xmin, ymax, xmax)\n        \n    else:\n        raise Exception(target_class.decode('utf-8') + \" was not found in the list of predictions\")","metadata":{"execution":{"iopub.status.busy":"2022-01-04T04:05:52.235027Z","iopub.execute_input":"2022-01-04T04:05:52.235318Z","iopub.status.idle":"2022-01-04T04:05:52.254261Z","shell.execute_reply.started":"2022-01-04T04:05:52.235267Z","shell.execute_reply":"2022-01-04T04:05:52.253383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Testing the cropping function on the sample image","metadata":{}},{"cell_type":"code","source":"#converting the previously selected and predicted on sample path to a numpy array\nsample_img_np = np.asarray(Image.open(sample_image_path))\n\ncrop_img_list = getCrops(sample_img_np, result_out, debug=True)","metadata":{"execution":{"iopub.status.busy":"2022-01-04T04:02:53.685762Z","iopub.execute_input":"2022-01-04T04:02:53.686061Z","iopub.status.idle":"2022-01-04T04:02:54.685201Z","shell.execute_reply.started":"2022-01-04T04:02:53.686011Z","shell.execute_reply":"2022-01-04T04:02:54.684362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predicting on the cropped image sections","metadata":{}},{"cell_type":"code","source":"def find_centrality(predictions, target_class=\"Door\", min_score=0.1):\n    '''\n    Given the list of prediction results, this function returns a measure of centrality for the target_class.\n    The closer to 0 the result is, the closer the predicted bounding box is to the center.\n    A large result means the predicted bounding box is far from the center.\n    If the result is -1, the target_class was not found in the list of predictions.\n    This measure does not give directional information.\n    \n    We ingore all predictions with a lower confidence than min_score\n    '''\n    \n    target_class = target_class.encode('utf-8')\n    \n    if target_class in predictions[\"detection_class_entities\"]:\n        \n        #this loops through all the bounding boxes\n        for i in range(len(predictions['detection_boxes'])):\n            if (predictions['detection_class_entities'][i] == target_class and\n                    predictions[\"detection_scores\"][i] >= min_score):\n                \n                #we will assume that only one door is detected\n                #(this means that if two doors are found we calculate centrality based on the first one)\n                \n                #min and max are percentages\n                ymin, xmin, ymax, xmax = predictions['detection_boxes'][i]\n                \n                predicted_center = ((xmax - xmin) /2 + xmin)\n                \n                return abs(0.5 - predicted_center) #because xmin and xmax are precentages, 0.5 is the image center\n            \n        return -1\n        \n    else:\n        return -1","metadata":{"execution":{"iopub.status.busy":"2022-01-04T04:02:54.954461Z","iopub.execute_input":"2022-01-04T04:02:54.954731Z","iopub.status.idle":"2022-01-04T04:02:54.962946Z","shell.execute_reply.started":"2022-01-04T04:02:54.954686Z","shell.execute_reply":"2022-01-04T04:02:54.961900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def predict_on_directory(directory, verbosity=1, one_prediction=False):\n    '''\n    if one_prediction is True it will only predict on the first image in the directory then stop\n    \n    verbosity = 0: only final accuracy is outputted\n    verbosity = 1: final accuracy, base image with base prediction overlayed, best cropped, best cropping with predictions\n                   overlayed\n    verbosity = 2: everything from verbosity 1, every cropping, every croppping with predictions overlayed\n    '''\n    \n    my_results = []\n\n    total_predictions = 0\n    total_correct = 0\n\n    for filename in os.listdir(directory):\n        total_predictions += 1\n\n        # Load the image string\n        image_path = os.path.join(directory, filename)\n        if verbosity > 0: print(image_path)\n        with tf.gfile.Open(image_path, \"rb\") as binfile:\n            image_string = binfile.read()\n\n\n        #Here we make a prediction to find where the house is located in the image\n        #We use this to know which portion of the image to zoom into\n\n        # Run our session\n        result_out = sess.run(\n            detector_output,\n            feed_dict={image_string_placeholder: image_string}\n        )\n\n\n        #We must check to see if a house was detected\n        #(we will only zoom/crop if there is a house in the image)\n        house_class = \"House\".encode('utf-8')\n\n        if house_class in result_out[\"detection_class_entities\"]:\n\n            sample_img_np = np.asarray(Image.open(image_path)) #get numpy version of the image\n            #zoom_range := (ymin, xmin, ymax, xmax)\n            crop_img_list, zoom_range = getCrops(sample_img_np, result_out) #get all the croppings\n\n\n            if verbosity > 0:\n                visible_print(\"base prediction\")\n\n                image_with_boxes = draw_boxes(\n                    sample_img_np.copy(), result_out[\"detection_boxes\"],\n                    result_out[\"detection_class_entities\"], result_out[\"detection_scores\"]\n                )\n                show_image(image_with_boxes, (16,10))\n\n\n                zoom_image = np.array(sample_img_np[int(zoom_range[0]*sample_img_np.shape[0]):\n                                                       int(zoom_range[2]*sample_img_np.shape[0]),\n                                                    int(zoom_range[1]*sample_img_np.shape[1]):\n                                                       int(zoom_range[3]*sample_img_np.shape[1]),\n                                                    :]) #[y, x, z]\n\n                visible_print(\"zoom image\")\n                show_image(zoom_image)    \n\n\n            best_centrality = -1 #-1 means that the object wasn't detected\n            best_centrality_idx = None\n            crop_res_list = []\n            for i, crop_img in enumerate(crop_img_list):\n\n                #convert the numpy cropping to the correct format\n                #Something clever can probably be done to avoid this saving\n                #But the best option would be to modify the input of graph (the decoding step)\n\n                crop_pil = Image.fromarray(crop_img)\n                crop_pil.save(\"temp_img.jpg\")\n\n                with tf.gfile.Open(\"./temp_img.jpg\", \"rb\") as binfile:\n                    crop_string = binfile.read()\n\n                # Run our session\n                crop_res_out = sess.run(\n                    detector_output,\n                    feed_dict={image_string_placeholder: crop_string}\n                )\n\n                crop_res_list.append(crop_res_out)\n\n                cur_centrality = find_centrality(crop_res_out)\n\n                if verbosity > 1:\n                    visible_print(\"detected classes for cropping \" + str(i+1) + \"/\" + str(len(crop_img_list)))\n                    print(\"centrality:\", cur_centrality)\n\n                if cur_centrality != -1 and (best_centrality == -1 or cur_centrality < best_centrality):\n                    best_centrality = cur_centrality\n                    best_centrality_idx = i\n\n\n                if verbosity > 1:\n                    show_image(crop_img)\n\n                    image_with_boxes = draw_boxes(\n                        crop_img.copy(), crop_res_out[\"detection_boxes\"],\n                        crop_res_out[\"detection_class_entities\"], crop_res_out[\"detection_scores\"]\n                    )\n                    show_image(image_with_boxes)\n\n\n\n            #finding the best cropping\n\n            if verbosity > 0: visible_print(\"best cropping\")\n\n            if best_centrality != -1:\n                if verbosity > 0:\n                    show_image(crop_img_list[best_centrality_idx])\n\n                    image_with_boxes = draw_boxes(\n                        crop_img_list[best_centrality_idx].copy(), crop_res_list[best_centrality_idx][\"detection_boxes\"],\n                        crop_res_list[best_centrality_idx][\"detection_class_entities\"], crop_res_list[best_centrality_idx][\"detection_scores\"]\n                    )\n                    show_image(image_with_boxes)\n\n                total_correct += 1\n\n            else:\n                if verbosity > 0: angry_print(\"no door found in \" + filename)\n\n        else:\n            if verbosity > 0: angry_print(\"no house found in \" + filename)\n\n        if one_prediction: break\n\n    angry_print(\"final accuracy: \" + str(100 * total_correct / total_predictions) + \"%\")","metadata":{"execution":{"iopub.status.busy":"2022-01-04T04:02:55.668468Z","iopub.execute_input":"2022-01-04T04:02:55.668782Z","iopub.status.idle":"2022-01-04T04:02:55.691919Z","shell.execute_reply.started":"2022-01-04T04:02:55.668732Z","shell.execute_reply":"2022-01-04T04:02:55.690849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Predicting on the cropped sections of the image for door detection. Using a small dataset","metadata":{}},{"cell_type":"code","source":"predict_on_directory(\"../input/raw-street-view\", verbosity=2)","metadata":{"execution":{"iopub.status.busy":"2022-01-04T04:06:05.690759Z","iopub.execute_input":"2022-01-04T04:06:05.691073Z","iopub.status.idle":"2022-01-04T04:06:32.164491Z","shell.execute_reply.started":"2022-01-04T04:06:05.691022Z","shell.execute_reply":"2022-01-04T04:06:32.163607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Testing on a larger dataset","metadata":{}},{"cell_type":"code","source":"predict_on_directory(\"../input/big-streetview-pics\", verbosity=1)","metadata":{"execution":{"iopub.status.busy":"2022-01-04T04:38:47.421507Z","iopub.execute_input":"2022-01-04T04:38:47.421833Z","iopub.status.idle":"2022-01-04T04:42:47.353667Z","shell.execute_reply.started":"2022-01-04T04:38:47.421782Z","shell.execute_reply":"2022-01-04T04:42:47.352701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Testing on a zoomed in version of the previous dataset\nWe do this to see if an increased resolution improves results. Zooming in improves resolution because we are not working with the original full images, we are only using screenshots of Google maps.","metadata":{}},{"cell_type":"code","source":"predict_on_directory(\"../input/big-streetview-pics-zoomed\", verbosity=1)","metadata":{"execution":{"iopub.status.busy":"2022-01-04T04:14:53.913633Z","iopub.execute_input":"2022-01-04T04:14:53.913960Z","iopub.status.idle":"2022-01-04T04:18:46.190718Z","shell.execute_reply.started":"2022-01-04T04:14:53.913903Z","shell.execute_reply":"2022-01-04T04:18:46.189861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## End the session","metadata":{}},{"cell_type":"code","source":"sess.close()","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:53:27.924353Z","iopub.status.idle":"2022-01-04T03:53:27.925031Z"},"trusted":true},"execution_count":null,"outputs":[]}]}