{"cells":[{"metadata":{"_uuid":"0c2c8a2028b69e288c615e2b5255cd14ec5d535c"},"cell_type":"markdown","source":"**Version 2:**\n\nIn the previous version I implemented a simple IoU function imagining a graph with the origin (0,0) located in the bottom left. In this version I've updated the IoU function imagining a numpy array with the origin (0,0) located at the top left corner. \n\nThis version can be a bit confusing. To understand it better keep in mind that:\n\nGiven bounding box: *[x, y, width, height, conf_score]*\n\nthen index selection in numpy will take the form *array[y,x]* i.e. *array[row, column]*."},{"metadata":{"_uuid":"0e8cb018b5351b9bafff359a256667bfb17f8be6"},"cell_type":"markdown","source":"#### Introduction"},{"metadata":{"_uuid":"1b67c0dd1f19e335a441e63b65c27469625c021c"},"cell_type":"markdown","source":"In this kernel I will quickly demonstrate a method for implementing a custom metric in Keras without having to use tensors to write it. A tensorflow wrapper function called tf.py_func makes this possible. The body of the function is written using numpy and then inserted into py_func. You can think of py_func as a kind of Trojan horse.\n\nWe will implement a basic IoU (Intersection Over Union) metric."},{"metadata":{"_uuid":"a3832e28fec0e2a04429d356ba5156c8e06b30c5"},"cell_type":"markdown","source":"### What is IoU?"},{"metadata":{"_uuid":"f7a68a17d676ea8ad2837c33a7d969ad2148fba4"},"cell_type":"markdown","source":"This is a good blog post that clearly explains IoU.\n\nhttps://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/"},{"metadata":{"_uuid":"4d96127f7bbfb7cb771d1545642df8eeb83490e9"},"cell_type":"markdown","source":"### What is the form of y, the target?"},{"metadata":{"_uuid":"a03d69ce5a20753b05dfcb3edfbca07946a6da4e"},"cell_type":"markdown","source":"We will be predicting one bounding box. The output has 5 columns. This is the form:\n\ny_true = [x, y, width, height, conf]<br>\ny_pred = [x, y, width, height, conf]\n\nx and y are the coordinates of the top left corner of the box and<br>\nconf is the confidence score."},{"metadata":{"_uuid":"66d196d7047747527bb05f9570baacdd2392f956"},"cell_type":"markdown","source":"### What output do we get from Keras?"},{"metadata":{"_uuid":"75181a3fa1aea5f1f323e9909cd85d511c0142bc"},"cell_type":"markdown","source":"As the model is running Keras outputs batches of y_pred and y_true as numpy arrays. So if y is as shown above and we have a batch size of 10, then y_pred and y_true will be numpy arrays with a shape (10,5) i.e. 10 rows and 5 colums."},{"metadata":{"_uuid":"3478cae0452db1de2354602bcf3b9f02e84d7c7f"},"cell_type":"markdown","source":"### The IoU Function"},{"metadata":{"_uuid":"9e5db7e55de97cd79a405d36edddd539d15908d6"},"cell_type":"markdown","source":"Our custon IoU function will take y_true and y_pred as inputs and output the average IoU score for the batch - as a scalar of type float32. This is what py_func needs to work its magic.\nSomething to note is that all caculations done within the function must be numpy. For example np.min() and np.max(). Python functions should not be used."},{"metadata":{"_uuid":"3f2fb758ed1e85813fa06d1c56e0278833e28d81"},"cell_type":"markdown","source":"This is my simple IoU implementation:"},{"metadata":{"trusted":true,"_uuid":"0eb3789262a90619bd4cefa8acaa2d24e3088952"},"cell_type":"code","source":"def calculate_iou(y_true, y_pred):\n    \n    \n    \"\"\"\n    Input:\n    Keras provides the input as numpy arrays with shape (batch_size, num_columns).\n    \n    Arguments:\n    y_true -- first box, numpy array with format [x, y, width, height, conf_score]\n    y_pred -- second box, numpy array with format [x, y, width, height, conf_score]\n    x any y are the coordinates of the top left corner of each box.\n    \n    Output: IoU of type float32. (This is a ratio. Max is 1. Min is 0.)\n    \n    \"\"\"\n\n    \n    results = []\n    \n    for i in range(0,y_true.shape[0]):\n    \n        # set the types so we are sure what type we are using\n        y_true = y_true.astype(np.float32)\n        y_pred = y_pred.astype(np.float32)\n\n\n        # boxTrue\n        x_boxTrue_tleft = y_true[0,0]  # numpy index selection\n        y_boxTrue_tleft = y_true[0,1]\n        boxTrue_width = y_true[0,2]\n        boxTrue_height = y_true[0,3]\n        area_boxTrue = (boxTrue_width * boxTrue_height)\n\n        # boxPred\n        x_boxPred_tleft = y_pred[0,0]\n        y_boxPred_tleft = y_pred[0,1]\n        boxPred_width = y_pred[0,2]\n        boxPred_height = y_pred[0,3]\n        area_boxPred = (boxPred_width * boxPred_height)\n\n\n        # calculate the bottom right coordinates for boxTrue and boxPred\n\n        # boxTrue\n        x_boxTrue_br = x_boxTrue_tleft + boxTrue_width\n        y_boxTrue_br = y_boxTrue_tleft + boxTrue_height # Version 2 revision\n\n        # boxPred\n        x_boxPred_br = x_boxPred_tleft + boxPred_width\n        y_boxPred_br = y_boxPred_tleft + boxPred_height # Version 2 revision\n\n\n        # calculate the top left and bottom right coordinates for the intersection box, boxInt\n\n        # boxInt - top left coords\n        x_boxInt_tleft = np.max([x_boxTrue_tleft,x_boxPred_tleft])\n        y_boxInt_tleft = np.max([y_boxTrue_tleft,y_boxPred_tleft]) # Version 2 revision\n\n        # boxInt - bottom right coords\n        x_boxInt_br = np.min([x_boxTrue_br,x_boxPred_br])\n        y_boxInt_br = np.min([y_boxTrue_br,y_boxPred_br]) \n\n        # Calculate the area of boxInt, i.e. the area of the intersection \n        # between boxTrue and boxPred.\n        # The np.max() function forces the intersection area to 0 if the boxes don't overlap.\n        \n        \n        # Version 2 revision\n        area_of_intersection = \\\n        np.max([0,(x_boxInt_br - x_boxInt_tleft)]) * np.max([0,(y_boxInt_br - y_boxInt_tleft)])\n\n        iou = area_of_intersection / ((area_boxTrue + area_boxPred) - area_of_intersection)\n\n\n        # This must match the type used in py_func\n        iou = iou.astype(np.float32)\n        \n        # append the result to a list at the end of each loop\n        results.append(iou)\n    \n    # return the mean IoU score for the batch\n    return np.mean(results)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"acadccd37d224602ccbb0aa79158d508572f6b0d"},"cell_type":"markdown","source":"### Define a function containing py_func"},{"metadata":{"_uuid":"bb0337e4b83eef091fe1478ba8f69e0af1c1407c"},"cell_type":"markdown","source":"The input arguments will be y_true, y_pred and the \"calculate_iou\" function we created above."},{"metadata":{"trusted":true,"_uuid":"95c1ce7da73a67d9cc6d68bc1aa756d95d26261f"},"cell_type":"code","source":"def IoU(y_true, y_pred):\n    \n    # Note: the type float32 is very important. It must be the same type as the output from\n    # the python function above or you too may spend many late night hours \n    # trying to debug and almost give up.\n    \n    iou = tf.py_func(calculate_iou, [y_true, y_pred], tf.float32)\n\n    return iou","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c511b5c0d152195331cbd96fb1d4ba200e54ae3"},"cell_type":"markdown","source":"### How to use this with Keras?"},{"metadata":{"_uuid":"167233a6e77d13ba591d978cf9275b51a9db7a87"},"cell_type":"markdown","source":"We need to simply enter the IoU function as a metric in the compile step, like this:"},{"metadata":{"trusted":true,"_uuid":"2bc291118ffc7f83ced26ec2c55181a099f5b30f"},"cell_type":"code","source":"# model.compile(optimizer='Adam', loss='mse', metrics=[IoU])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68193555c33743f2c3f064e1592417f737bde607"},"cell_type":"markdown","source":"#### and that's it. "},{"metadata":{"_uuid":"24d93ede26b40e4351b3ed2a59e438366cebaf5e"},"cell_type":"markdown","source":"<hr>"},{"metadata":{"_uuid":"263439480cfa7612e682d1a35ea86302c4dc48aa"},"cell_type":"markdown","source":"### Testing the functions"},{"metadata":{"_uuid":"b2713605968d9f926190d7d2d305ef09af4f7d2f"},"cell_type":"markdown","source":"This is a quick test to make sure that the functions we created are working as expected. The final answer should be:<br> IoU = 0.153846"},{"metadata":{"trusted":true,"_uuid":"40cbb4f6b2774b92e360d965277947052e2adc5b"},"cell_type":"code","source":"import numpy as np\nimport tensorflow as tf\n\n# create two inputs simulating a batch_size = 3\n# shape (3,5)\ny_true = np.array([[1,5,2,3,0.5], [1,5,2,3,0.5], [1,5,2,3,0.5]])\ny_pred = np.array([[2,4,3,3,0.7], [2,4,3,3,0.7], [2,4,3,3,0.7]])\n\n# call the first function\nresult = calculate_iou(y_true, y_pred)\n\nprint(result)\nprint(type(result))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d365b6fb25373a139d6c3492108637d046a6a967"},"cell_type":"code","source":"# call the second function\niou = IoU(y_true, y_pred)\n\nprint(type(iou))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e3e5afcc449d14b03bc7def04e61c0eb3216ab20"},"cell_type":"code","source":"# because iou is a tensor we can only print it inside a Tensorflow session\n\nwith tf.Session() as sess:\n    print(sess.run(iou))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"512917e5d202b80dacc06994ee6c96757a7e7e07"},"cell_type":"markdown","source":"<hr>"},{"metadata":{"_uuid":"9537f025687a058b814dca4de29a71e5a35af1fb"},"cell_type":"markdown","source":"### Resources\n\nThese are some resources that I found helpful when learning this subject:"},{"metadata":{"collapsed":true,"_uuid":"68981f6e1040e8c5300fa8c39735b566b1111919"},"cell_type":"markdown","source":"Tensorflow info on py_func:<br>\n https://www.tensorflow.org/api_docs/python/tf/py_func\n\nBlog post explaining IoU:<br>\n https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/\n\nKernel by @aglotero where I first saw py_func used:<br>\nhttps://www.kaggle.com/aglotero/another-iou-metric\n\n"},{"metadata":{"_uuid":"6a25f554af609ef63bac15e30b6621d97dc0d968"},"cell_type":"markdown","source":"<hr>\n\nThank you for reading."},{"metadata":{"trusted":true,"_uuid":"b2a1cfff853f80e08db2729a1269c52d11560cc1"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}