{"cells":[{"metadata":{},"cell_type":"markdown","source":"## References:\n* https://github.com/tensorflow/hub/blob/master/examples/colab/object_detection.ipynb\n* https://towardsdatascience.com/non-maximum-suppression-nms-93ce178e177c\n* https://github.com/justcallmewilliam/iccv19-silco/tree/master/cl_utils/mAP_lib\n* https://www.kaggle.com/vikramtiwari/baseline-predictions-using-inception-resnet-v2","execution_count":null},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true},"cell_type":"markdown","source":"# Object Detection Using TF HUB","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Content in this kernel\n1. Faster R-CNN\n2. Intersection Over Union (IOU)\n3. Precision\n4. Recall\n5. Non-maximum Suppression (NMS)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Faster R-CNN\nIt first extract feature maps from the input image using ConvNet and then pass those maps through a RPN which returns object proposals. Finally, these maps are classified and the bounding boxes are predicted.\n\n\n![](http://cdn.analyticsvidhya.com/wp-content/uploads/2018/10/Faster-rcnn.png)\n\n## Steps followed by a Faster R-CNN algorithm to detect objects in an image:\n1. Take an input image and pass it to the ConvNet which returns feature maps for the image.\n2. Apply Region Proposal Network (RPN) on these feature maps and get object proposals.\n3. Apply ROI pooling layer to bring down all the proposals to the same size.\n4. Finally, pass these proposals to a fully connected layer in order to classify any predict the bounding boxes for the image.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Intersection Over Union (IOU)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Intersection Over Union (IOU) is measure based on Jaccard Index that evaluates the overlap between two bounding boxes. It requires a ground truth bounding box and a predicted bounding box . By applying the IOU we can tell if a detection is valid (True Positive) or not (False Positive).\n\nIOU is given by the overlapping area between the predicted bounding box and the ground truth bounding box divided by the area of union between them:\n![](http://camo.githubusercontent.com/70d881e53ef692bc1c7c1cb3265d7b30a8818701/687474703a2f2f6c617465782e636f6465636f67732e636f6d2f6769662e6c617465783f25354374657874253742494f552537442532302533442532302535436672616325374225354374657874253742617265612537442532302535436c656674253238425f70253230253543636170253230425f2537426774253744253543726967687425323925374425374225354374657874253742617265612537442532302535436c656674253238425f70253230253543637570253230425f25374267742537442535437269676874253239253744)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The image below illustrates the IOU between a ground truth bounding box (in green) and a detected bounding box (in red).\n![](http://raw.githubusercontent.com/rafaelpadilla/Object-Detection-Metrics/master/aux_images/iou.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# True Positive, False Positive, False Negative and True Negative\n\nSome basic concepts used by the metrics:\n1. **True Positive (TP)**: A correct detection. Detection with IOU ≥ threshold\n2. **False Positive (FP)**: A wrong detection. Detection with IOU < threshold\n3. **False Negative (FN)**: A ground truth not detected\n4. **True Negative (TN)**: Does not apply. It would represent a corrected misdetection. In the object detection task there are many possible   bounding boxes that should not be detected within an image. Thus, TN would be all possible bounding boxes that were corrrectly not detected (so many possible boxes within an image). That's why it is not used by the metrics.\n\nthreshold: depending on the metric, it is usually set to 50%, 75% or 95%.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Precision\nPrecision is the ability of a model to identify only the relevant objects. It is the percentage of correct positive predictions and is given by:\n\n\n![](http://camo.githubusercontent.com/b1b6fdbeef01e93c1369e9d3e28fd7932e322852/687474703a2f2f6c617465782e636f6465636f67732e636f6d2f6769662e6c617465783f507265636973696f6e25323025334425323025354366726163253742545025374425374254502b465025374425334425354366726163253742545025374425374225354374657874253742616c6c253230646574656374696f6e73253744253744)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Recall\nRecall is the ability of a model to find all the relevant cases (all ground truth bounding boxes). It is the percentage of true positive detected among all relevant ground truths and is given by:\n\n![](http://camo.githubusercontent.com/3e4ced65f38c8177e5fed382ba409f357ecab0b6/687474703a2f2f6c617465782e636f6465636f67732e636f6d2f6769662e6c617465783f526563616c6c25323025334425323025354366726163253742545025374425374254502b464e25374425334425354366726163253742545025374425374225354374657874253742616c6c25323067726f756e64253230747275746873253744253744)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":" # Non-Maximum Suppression (NMS):\n  Non-Maximum Suppression (NMS), a post-processing algorithm responsible for merging all detections that belong to the same object.\n  \n  ## Input:\n   A list of Proposal boxes B, corresponding confidence scores S and overlap threshold N.\n  ## Output:\n  A list of filtered proposals D.\n  ## Algorithm:\n  \n1. Select the proposal with highest confidence score, remove it from B and add it to the final proposal list D. (Initially D is empty).\n2. Now compare this proposal with all the proposals — calculate the IOU (Intersection over Union) of this proposal with every other proposal. If the IOU is greater than the threshold N, remove that proposal from B.\n3. Again take the proposal with the highest confidence from the remaining proposals in B and remove it from B and add it to D.\n4. Once again calculate the IOU of this proposal with all the proposals in B and eliminate the boxes which have high IOU than threshold.\n5. This process is repeated until there are no more proposals left in B.\n\n![](http://miro.medium.com/max/1400/1*6d_D0ySg-kOvfrzIRwHIiA.png)\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport pandas as pd\nimport tensorflow as tf\nimport tensorflow_hub as hub\nimport matplotlib.pyplot as plt\nfrom six import BytesIO\nimport numpy as np\nimport xml.etree.ElementTree as et\nimport ast\nimport tqdm\nfrom itertools import chain\nfrom xml.dom import minidom\nfrom PIL import Image\nfrom PIL import ImageColor\nfrom PIL import ImageDraw\nfrom PIL import ImageFont\nfrom PIL import ImageOps\nimport cv2\nimport glob\nimport time","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Path of Image directory","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"path1='/kaggle/input/open-images-object-detection-rvc-2020/test/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = pd.read_csv(\"/kaggle/input/open-images-object-detection-rvc-2020/sample_submission.csv\")\nsample.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ids = []\nfor i in range(len(sample)):\n    ids.append(sample['ImageId'][i])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ids[0:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_data=[]\nfor i in range(len(sample)):\n    img_data.append(glob.glob('/kaggle/input/open-images-object-detection-rvc-2020/test/{0}.jpg'.format(ids[i])))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_data[0:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_data=list(chain.from_iterable(img_data))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_data[0:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_prediction_string(result):\n    with tf.device('/device:GPU:0'):\n        df = pd.DataFrame(columns=['Ymin','Xmin','Ymax', 'Xmax','Score','Label','Class_label','Class_name'])\n        min_score=0.01\n        for i in range(result['detection_boxes'].shape[0]):\n           if (result[\"detection_scores\"][i]) >= min_score:\n              df.loc[i]= tuple(result['detection_boxes'][i])+(result[\"detection_scores\"][i],)+(result[\"detection_class_labels\"][i],)+(result[\"detection_class_names\"][i],)+(result[\"detection_class_entities\"][i],)\n        return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow.compat.v1 as tf\ntf.disable_v2_behavior()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"module_handle = \"https://tfhub.dev/google/faster_rcnn/openimages_v4/inception_resnet_v2/1\"\nwith tf.device('/device:GPU:0'):\n    with tf.Graph().as_default():\n        detector = hub.Module(module_handle)\n        image_string_placeholder = tf.placeholder(tf.string)\n        decoded_image = tf.image.decode_jpeg(image_string_placeholder)\n        decoded_image_float = tf.image.convert_image_dtype(\n            image=decoded_image, dtype=tf.float32)\n        module_input = tf.expand_dims(decoded_image_float, 0)\n        result = detector(module_input, as_dict=True)\n        init_ops = [tf.global_variables_initializer(), tf.tables_initializer()]\n\n        session = tf.Session()\n        session.run(init_ops)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def nms(dets, thresh):\n    x1 = dets[:, 0]\n    y1 = dets[:, 1]\n    x2 = dets[:, 2]\n    y2 = dets[:, 3]\n    scores = dets[:, 4]\n\n    areas = (x2 - x1 + 1) * (y2 - y1 + 1)\n    order = scores.argsort()[::-1]\n\n    keep = []\n    while order.size > 0:\n        i = order[0]\n        keep.append(i)\n        xx1 = np.maximum(x1[i], x1[order[1:]])\n        yy1 = np.maximum(y1[i], y1[order[1:]])\n        xx2 = np.minimum(x2[i], x2[order[1:]])\n        yy2 = np.minimum(y2[i], y2[order[1:]])\n\n        w = np.maximum(0.0, xx2 - xx1 + 1)\n        h = np.maximum(0.0, yy2 - yy1 + 1)\n        inter = w * h\n        ovr = inter / (areas[i] + areas[order[1:]] - inter)\n\n        inds = np.where(ovr <= thresh)[0]\n        order = order[inds + 1]\n\n    return keep","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## First 20 Images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"image_paths = img_data[0:20]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images = []\nfor f in image_paths:\n    images.append(np.asarray(Image.open(f)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir deepak","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_id = sample['ImageId']\ndef format_prediction_string(image_id, result):\n    prediction_strings = []\n    \n    for i in range(len(result['Score'])):\n        class_name = result['Class_label'][i].decode(\"utf-8\")\n        YMin,XMin,YMax,XMax = result['Ymin'][i],result['Xmin'][i],result['Ymax'][i],result['Xmax'][i]\n        score = result['Score'][i]\n        \n        prediction_strings.append(\n            f\"{class_name} {score} {XMin} {YMin} {XMax} {YMax}\"\n        )\n        \n    prediction_string = \" \".join(prediction_strings)\n\n    return {\n        \"PredictionString\": prediction_string\n    }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"k =-1\npredictions = []\nwith tf.device('/device:GPU:0'):\n    for image_path in image_paths:\n        k=k+1\n        img_path = img_data[k]\n        img = cv2.imread(img_path)\n        with tf.gfile.Open(image_path, \"rb\") as binfile:\n            image_string = binfile.read()\n\n        inference_start_time = time.time()\n        result_out, image_out = session.run(\n            [result, decoded_image],\n            feed_dict={image_string_placeholder: image_string})\n        df1=get_prediction_string(result_out)\n        z1=nms(df1.values,0.68)\n        z=df1.iloc[z1]\n        z=z.reset_index()\n        predictions.append(format_prediction_string(image_id, z))\n        data1=z\n        COLORS = np.random.uniform(0, 255, size=(len(z['Class_name']), 3))\n        for m in range(len(data1)):\n            if data1['Score'][m] >=0.01:\n                img_class=data1.iloc[m].Class_name\n                img_xmax, img_ymax =images[k].shape[1],images[k].shape[0]\n                bbox_x_max, bbox_x_min = data1.Xmax[m] * img_xmax, data1.Xmin[m] * img_xmax\n                bbox_y_max ,bbox_y_min = data1.Ymax[m] * img_ymax, data1.Ymin[m] * img_ymax\n                xmin = int(bbox_x_min)\n                ymin = int(bbox_y_min)\n                xmax = int(bbox_x_max)\n                ymax = int(bbox_y_max)\n                width = xmax - xmin\n                height = ymax - ymin\n                label = str(data1['Class_name'][m])\n                color = COLORS[m]\n                cv2.rectangle(img, (xmin, ymax), (xmax, ymin), color, 2)\n                path1 = '/kaggle/working/deepak/'+str(k)+'.jpg'\n                img_path = path1\n                cv2.imwrite(path1, img)\n                cv2.putText(img, label, (xmax,ymin), cv2.FONT_HERSHEY_SIMPLEX, 0.9,color, 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_images(folder):\n    images = []\n    for filename in os.listdir(folder):\n        img = Image.open(os.path.join(folder, filename))\n        if img is not None:\n            images.append(img)\n    return images","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"z = load_images(\"/kaggle/working/deepak\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"z[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"z[3]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"z[4]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"z[6]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"z[9]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"z[10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"z[11]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"z[15]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"z[18]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## For Submission","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_df = pd.DataFrame(predictions)\npred_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample['PredictionString']= pred_df['PredictionString']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## If you like this notebook,please upvote.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}