{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Augmented Reality\n\nIn this notebook, we'll augment an image by adding a 3D wireframe overlay on cars. 😊\n\n## References\n\n- Code for pin-hole camera projection is borrowed from **Li Linye (@zstusnoopy)**'s [\"Visualize the location and 3d bounding box of car\"](https://www.kaggle.com/zstusnoopy/visualize-the-location-and-3d-bounding-box-of-car) notebook\n- Code to load 3D car model is from my notebook [\"Load a 3D car model\"](https://www.kaggle.com/ebouteillon/load-a-3d-car-model)"},{"metadata":{"trusted":true},"cell_type":"code","source":"import json\nfrom math import sin, cos\nimport matplotlib.pylab as plt\nimport numpy as np\nimport pandas as pd\nimport cv2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load an image\nimg_name = 'ID_0ca978538'  # 'ID_0a0980d15'\nimg = cv2.imread(f'../input/pku-autonomous-driving/train_images/{img_name}.jpg',cv2.COLOR_BGR2RGB)[:,:,::-1]\nplt.figure(figsize=(20, 20))\nplt.imshow(img);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/pku-autonomous-driving/train.csv')\npred_string = train[train.ImageId == img_name].PredictionString.iloc[0]\nitems = pred_string.split(' ')\nmodel_types, yaws, pitches, rolls, xs, ys, zs = [items[i::7] for i in range(7)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# k is camera instrinsic matrix\nk = np.array([[2304.5479, 0,  1686.2379],\n           [0, 2305.8757, 1354.9849],\n           [0, 0, 1]], dtype=np.float32)\n\n# convert euler angle to rotation matrix\ndef euler_to_Rot(yaw, pitch, roll):\n    Y = np.array([[cos(yaw), 0, sin(yaw)],\n                  [0, 1, 0],\n                  [-sin(yaw), 0, cos(yaw)]])\n    P = np.array([[1, 0, 0],\n                  [0, cos(pitch), -sin(pitch)],\n                  [0, sin(pitch), cos(pitch)]])\n    R = np.array([[cos(roll), -sin(roll), 0],\n                  [sin(roll), cos(roll), 0],\n                  [0, 0, 1]])\n    return np.dot(Y, np.dot(P, R))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def draw_obj(image, vertices, triangles):\n    for t in triangles:\n        coord = np.array([vertices[t[0]][:2], vertices[t[1]][:2], vertices[t[2]][:2]], dtype=np.int32)\n#         cv2.fillConvexPoly(image, coord, (0,0,255))\n        cv2.polylines(image, np.int32([coord]), 1, (0,0,255))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load a 3D model of a car\nwith open('../input/pku-autonomous-driving/car_models_json/mazida-6-2015.json') as json_file:\n    data = json.load(json_file)\nvertices = np.array(data['vertices'])\nvertices[:, 1] = -vertices[:, 1]\ntriangles = np.array(data['faces']) - 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"overlay = np.zeros_like(img)\nfor yaw, pitch, roll, x, y, z in zip(yaws, pitches, rolls, xs, ys, zs):\n    yaw, pitch, roll, x, y, z = [float(x) for x in [yaw, pitch, roll, x, y, z]]\n    # I think the pitch and yaw should be exchanged\n    yaw, pitch, roll = -pitch, -yaw, -roll\n    Rt = np.eye(4)\n    t = np.array([x, y, z])\n    Rt[:3, 3] = t\n    Rt[:3, :3] = euler_to_Rot(yaw, pitch, roll).T\n    Rt = Rt[:3, :]\n    P = np.ones((vertices.shape[0],vertices.shape[1]+1))\n    P[:, :-1] = vertices\n    P = P.T\n    img_cor_points = np.dot(k, np.dot(Rt, P))\n    img_cor_points = img_cor_points.T\n    img_cor_points[:, 0] /= img_cor_points[:, 2]\n    img_cor_points[:, 1] /= img_cor_points[:, 2]\n    draw_obj(overlay, img_cor_points, triangles)\n\nalpha = .5\nimg = np.array(img)\ncv2.addWeighted(overlay, alpha, img, 1 - alpha, 0, img)\nplt.figure(figsize=(20,20))\nplt.imshow(img);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Conclusion\n\nThis simple image augmentation gives the image a great futuristic look. It feels like we can see through the black car on the left: look at the details of the four wheels!\n\nThanks for reading, and don't forget to upvote this notebook if you like it. 👍"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}