{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"## TensorFlow Object Detection API + Apache Spark\n\nThe goal of this notebook is to utilize TensorFlow Object Detection API [1] and provide insight on how to prepare the training files using Apache Spark [2].\n\nTensorFlow Object Detection API is a research library maintained by Google that contains multiple pretrained, ready for transfer learning object detectors that provide different speed vs accuracy tradeoffs [3]. Examples include Faster R-CNN, YOLO and SSD."},{"metadata":{"_uuid":"d706975250df4e67071bd24961f1ee7ec3afa3ac","trusted":true},"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/object-detection-kite/kites_detections_output.jpg\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c098220660046bb15d123fd5217674f73853984f"},"cell_type":"markdown","source":"In the kite example image above we can see multiple objects detected at multiple scales. Object detectors enable us to capture rich information present in the image (when a single label is not informative enough) and is the key technology behind many applications such as visual search."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"markdown","source":"## Installation Notes\n1. git clone https://github.com/tensorflow/models.git\n2. have a look at installation instructions: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/installation.md  \nInstall / make sure you have 3.0.0 version of protobuf:  https://github.com/tensorflow/models/issues/4062  \nUpdated your ~/.bashrc to locate the protobuf binary and the required directories  \nexport PATH=PATH:<full-path-to-downloaded-protobuf-3.0.0>/bin/  \nexport PYTHONPATH=$PYTHONPATH:<full path to tensorflow>/models/research/:<full path to tensorflow>/models/research/slim/    \nprotoc object_detection/protos/*.proto --python_out=.   \n\n3. test your installation (from /tensorflow/models/research directory) should work without a problem:   \npython object_detection/builders/model_builder_test.py"},{"metadata":{"_uuid":"d4229d1f5bd9c8ac38b4632e957ce03fb73cf430"},"cell_type":"markdown","source":"## Preparing Data\nWe'll use the PASCAL VOC data format for storing images and annotations. For training, we need to generate a TFRecord file using the API script: create\\_pascal\\_tf\\_record.py . We'll also need to define the label map, e.g. similar to one in object\\_detection/data/pascal\\_label\\_map.pbtxt  \n\nFor more info on how to generate TFRecord train and validation files, have a look at: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/preparing_inputs.md"},{"metadata":{"_uuid":"a5099cb6acd9bc4ff3cca926853dc691549699fe"},"cell_type":"markdown","source":"## Selecting the Model\nThe model can be selected by modifying the corresponding config file located at: \nhttps://github.com/tensorflow/models/tree/master/research/object_detection/samples/configs\n\nIn particular, for transfer learning, we'd like to modify the number of classes and all the paths marked to be configured. We also need to download the check-point files from the model zoo, for a list of available pre-trained models and to download see:\nhttps://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md  \n"},{"metadata":{"_uuid":"affbc9138440bc365185b27a71b673b394a09dff"},"cell_type":"markdown","source":"## Training and Evaluation \nTo train and evaluate the model, we need to copy the following files (created above) into a training folder:  \n*dataset\\_train.record*, *dataset\\_val.record*, *model\\_name.config*, *label\\_map.pbtxt*, *model.ckpt* in addition to *train.py* and *eval.py*  \n    \nOnce we have all the files, we are ready to train and evaluate the model as follows:\n> python object_detection/train.py --logtostderr  --pipeline\\_config\\_path={PATH\\_TO\\_CONFIG}  \n--train\\_dir=${PATH\\_TO\\_TRAIN\\_DIR}  \n \n> python object_detection/eval.py --logtostderr --pipeline_config_path={PATH\\_TO\\_CONFIG} --checkpoint_dir={PATH\\_TO\\_TRAIN\\_DIR} --eval_dir={PATH\\_TO\\_EVAL\\_DIR}\n \n> tensorboard --logdir=${PATH\\_TO\\_MODEL\\_DIR}\n \n> python object_detection/export_inference_graph.py --input_type image_tensor --pipeline_config_path {PATH\\_TO\\_CONFIG}   \n--trained_checkpoint_prefix {PATH\\_TO\\_TRAIN\\_DIR}/model.ckpt-XXXX --output_directory {PATH\\_TO\\_TRAIN\\_DIR}/inference\\_graph\n\nNote that we can start training and evaluation simultaneously (in two different terminals or on two different machines / GPUs).  \nAfter the model has been trained, we need to export it using TensorFlow graph proto as described here:\nhttps://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/exporting_models.md\n\n"},{"metadata":{"_uuid":"990f153df98a401016d95dd62fc09fc6ffdc1413"},"cell_type":"markdown","source":"## Multi-GPU training\n\nTensorFlow Object Detection API supports multi-GPU training that can be enabled as follows (where num_clones  = number of GPUs):\n> python object\\_detection/train.py --logtostderr --pipeline\\_config\\_path={PATH\\_TO\\_CONFIG} --train\\_dir={PATH\\_TO\\_TRAIN\\_DIR}  \n    --num\\_clones=4 --ps\\_tasks=1"},{"metadata":{"_uuid":"7177f678386a345b2b2aa8b91cab20321de3a296"},"cell_type":"markdown","source":"## Notes on Evaluation\nThe metric used to evaluate object detection performance is **mean average precision** (mAP@IoU). mAP is computed by taking the average of AP over the object detection classes. Average Precision (AP) is computed per class by fixing the IoU threshold (e.g. 0.5) and calculating precision (fraction of bounding boxes that overlap with ground truth by at least the IoU threshold). To learn more about evaluation metrics for object detection, have a look at [4].  \n\nIn order to configure the IoU threshold for evaluation, you have to modify matching_iou_threshold parameter in object\\_detection/utils/object\\_detection\\_evaluation.py\n\nIn addition, in order to display detection results (especially during the beggining of training), we might want to lower the score threshold:\n\n> eval_config: {  \n  num_examples: 20000  \n  num_visualizations: 16  \n  min_score_threshold: 0.15  \n  max_evals: 1  \n}  \n\nBy default the evaluator is set to EVAL\\_DEFAULT\\_METRIC = 'pascal\\_voc\\_detection\\_metrics' in order to change it to use Open Image evaluation metrics you have to modify /models/research/object\\_detection/evaluator.py script and set EVAL\\_DEFAULT\\_METRIC = 'open\\_images\\_V2\\_detection\\_metrics'.  \n\n"},{"metadata":{"_uuid":"9233ad4dce0dada1f2cf89899a3d71156d12f5f7"},"cell_type":"markdown","source":"## Preparing Training Files"},{"metadata":{"_uuid":"ad01da616ca83fa0988764730f9262b2448c563c"},"cell_type":"markdown","source":"In order to train the object detector, we need to prepare the training files using the following structure:\n> VOC2012  \n    |\\_\\_\\_Annotations  \n    |\\_\\_\\_ImageSets/Main   \n    |\\_\\_\\_JPEGImages \n\nFor every image file we need to create an XML annotations files with information about different objects, their labels and bounding box coordinates. In addition, ImageSets/Main folder stores the list of training, validation and trainval files that contain indicators for when a particular object is present in an image. Finally, the actual images are stored in JPEGImages folder.\n    \n\n"},{"metadata":{"_uuid":"dc041a03bb7035963d8266a765015d5726403c67"},"cell_type":"markdown","source":"## Apache Spark: Preprocessing Data\n\nDue to large dataset size, it's good to parallelize computations using Apache Spark [2]. First, we'll need to pre-process the data and extract image dimensions (since the bounding box coordinates in the dataset are normalized between 0 and 1). We can do that by defining a User Defined Function (UDF) in pySpark:\n\n    from skimage.io import imread\n    def get_dims(filename):\n       img = imread(\"/path/to/VOC2012/JPEGImages/\" + filename + \".jpg\")\n       return img.shape[:2]\n    getDimsUDF = udf(get_dims, ArrayType(IntegerType()))`\n\nFinally, we can transform our dataframe to have absolute bounding box pixel corrdinates as follows:\n\n    dataframe_with_dims_unfiltered = (dataframe_with_dims\n    .withColumn(\"x1\", (col(\"XMin\") * col(\"width\")).cast(IntegerType()))\n    .withColumn(\"y1\", (col(\"YMin\") * col(\"height\")).cast(IntegerType()))\n    .withColumn(\"x2\", (col(\"XMax\") * col(\"width\")).cast(IntegerType()))\n    .withColumn(\"y2\", (col(\"YMax\") * col(\"height\")).cast(IntegerType())).cache()\n    )\n    dataframe_with_dims_unfiltered.createOrReplaceTempView(\"unfiltered_annotations\")\n\nHaving created a temporary view, we can now use SQL to pre-process the data:\n\n    widthThreshold = 0.001 \n    heightThreshold = 0.001\n    widthPadding = 1 \n    heightPadding = 1\n    maxSize = 6000\n    \n    spark.sql(\"\"\"\n            SELECT * FROM unfiltered_annotations \n             WHERE width < {maxSize} AND height < {maxSize} \n             AND (x2 - x1 > {widthThreshold} * width AND y2 - y1 > {heightThreshold} * height)\n             AND (x1 < x2 AND x1 > {widthPadding} AND x2 < width - {widthPadding})\n             AND (y1 < y2 AND y1 > {heightPadding} AND y2 < height - {heightPadding})\n            \"\"\".format(maxSize=maxSize, widthThreshold=widthThreshold, heightThreshold=heightThreshold, widthPadding=widthPadding, heightPadding=heightPadding)).createOrReplaceTempView(\"filtered_annotations\")\n\n\n"},{"metadata":{"_uuid":"f8e03f359d3ce785eb3315b3e41ee46e6a12ff96"},"cell_type":"markdown","source":"## Apache Spark: Generating Annotations\n\nWe can use Spark transformations and actions to generate annotations. We'll use the older resilient distributed dataset (RDD) API to extract features from our filtered dataframe. To generate the XML annotations we can use the xmltodict package. We'll first extract the bounding box coordinates and labels for each object and then merge multiple objects into a single XML file. We'll distribute the computation among multiple workers and write out the XML annotations file using a foreach action as shown below: \n\n    def obj2dict(row):\n        return  (row['ImageID'], (row['width'], row['height'],\n        OrderedDict([('name', row['image_group']),\n                          ('pose', 'Unspecified'),\n                          ('truncated', '0'),\n                          ('difficult', '0'),\n                          ('bndbox',\n                          OrderedDict([('xmin', row['x1']),\n                                       ('ymin', row['y1']),\n                                       ('xmax', row['x2']),\n                                       ('ymax', row['y2'])]))])))\n\n    def img2dict(image_id, img_info):\n        width, height, image_objects = img_info[0]\n        return  (image_id, OrderedDict([('annotation',\n                 OrderedDict([('folder', 'VOC2012'),\n                         ('filename', image_id + '.jpg'),\n                         ('source',\n                         OrderedDict([('database', 'open_image'),\n                                     ('annotation', 'open_image'),\n                                     ('image', 'open_image')])),\n                                     ('size',\n                                      OrderedDict([('width',  width),\n                                                   ('height', height),\n                                                   ('depth', '3')])),\n                                     ('segmented', '0'),\n                                     ('object', image_objects)]))]))\n\n    def dict2xml(item):\n        return xmltodict.unparse(item, pretty=True)\n  \n    def saveXML(imgId, data):\n      if not os.path.exists(\"/path/to/VOC2012/Annotations/\" + imgId + \".xml\"):\n        with open(\"/path/to/VOC2012/Annotations/\" + imgId + \".xml\", \"w\") as f:\n          f.write(data) . \n\n    xml_df = dataframe_merged.rdd.map(lambda row: obj2dict(row)).groupByKey().mapValues(list)\\\n                                       .map(lambda img_info: img2dict(img_info[0], img_info[1]))\\\n                                       .map(lambda item: (item[0], dict2xml(item[1])))\\\n                                       .map(lambda rec: (rec[0], \"\\n\".join(rec[1].split('\\n')[1:])))\\\n                                       .toDF([\"ImageID\", \"data\"])\\\n                                       .cache()\n                                       \n    xml_df.rdd.foreach(lambda r: saveXML(r[\"ImageID\"], r[\"data\"]))\n"},{"metadata":{"_uuid":"b70004c57c715c2024697b756db60ef998374c15"},"cell_type":"markdown","source":"## Apache Spark: Generating ImageSet File\n\nFinally, we need to generate the object indicator files. We can do that by diving the dataset into training and validation:\n \n    df_train, df_val = kaggle_annotations_merged.randomSplit([0.8, 0.2], seed=0)\n    \nAnd add a (+1/-1) indicator for (presence / absence) of a particular category:\n\n    joined = kaggle_annotations_merged.select(col(\"ImageID\"), col(\"Label\").alias(\"friendly\"))\n    grouped = joined.groupBy(\"ImageID\").agg(collect_set(col(\"friendly\")).alias(\"Classes\"))\n    categories = list(set(map(lambda r: r[0], joined.select(col(\"friendly\")).distinct().collect())))\n\n    def forCategory(category):\n      outputLocation = \"/path/to/VOC2012/ImageSets/Main/\" + category + \"_trainval.txt\"\n\n      print(category +  \" -> \"  + outputLocation)\n      result = (joined\n      .withColumn(\"Score\", when(col(\"friendly\") == category, lit(+1))\n      .otherwise(lit(-1))).select(\"ImageID\", \"Score\")\n      )\n \n      with open(\"/path/to/VOC2012/ImageSets/Main/\" + category + \"_trainval.txt\", \"w\") as f:\n        data = result.rdd.map(lambda r: (r[0], r[1])).collect()\n        for item in data:\n          f.write(str(item[0]) + \" \" + str(item[1]) + \"\\n\")\n        #end for\n      #end with\n      return result\n    \n    for cat in categories:\n       forCategory(cat)\n    \nIn a similar way, we can generate train and val indicator files.\n"},{"metadata":{"_uuid":"897fec1b798006a9c0b3d291c3256c810f3e513e"},"cell_type":"markdown","source":"## Conclusion\n\nIn conclusion, TensorFlow Object Detection API provides a really nice framework for evaluating multiple object detectors. Together with Spark and multi-GPU training, it's possible to scale to massive datasets and achieve high mean average precision."},{"metadata":{"_uuid":"84cfcc71453eac7bbc12b7296645789f82627bc7"},"cell_type":"markdown","source":"## References\n[1] TensorFlow Object Detection API: https://github.com/tensorflow/models/tree/master/research/object_detection  \n[2] Databricks Apache Spark: https://databricks.com/  \n[3] An article comparing different object detectors: https://medium.com/@jonathan_hui/object-detection-speed-and-accuracy-comparison-faster-r-cnn-r-fcn-ssd-and-yolo-5425656ae359  \n[4] An article discussing object detector evaluation metrics: https://medium.com/@timothycarlen/understanding-the-map-evaluation-metric-for-object-detection-a07fe6962cf3   \n[5] Open Images Dataset: https://storage.googleapis.com/openimages/web/index.html"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}