{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This is a starter folded from @Ruslan's kernel https://www.kaggle.com/hocop1/centernet-baseline .<br>\nI made some modifications:\n1. use test_size=0.1 for close to  LB score \n2. add the mAP valid score(from @tito's kernel:https://www.kaggle.com/its7171/metrics-evaluation-script)\n3. mask, regr = get_mask_and_regr(img0, labels, flip=flip). It seems better than flip=False\n","metadata":{}},{"cell_type":"markdown","source":"# EDA + CenterNet Baseline\n\nReferences:\n* Took 3D visualization code from https://www.kaggle.com/zstusnoopy/visualize-the-location-and-3d-bounding-box-of-car\n* CenterNet paper https://arxiv.org/pdf/1904.07850.pdf\n* CenterNet repository https://github.com/xingyizhou/CenterNet\n\nChange log:\n* v14: better inference: added `optimize_xy` function | LB 0.093\n* v15: horizontal flip augmentation | ERROR\n* v16: faster training (made smaller padding) | LB 0.089\n* v17: smaller image size and better image proportions | LB 0.066\n* v18: image size back; changed flip probability; **new visualizations** | LB ?","metadata":{}},{"cell_type":"markdown","source":"# What is this competition about?\n1. You are given the images taken from the roof of a car\n    * ~4k training images\n    * Always the same car and the same camera\n2. You are asked to detect other cars on that image\n    * There can be many cars\n    * You need to predict their positions\n![](https://i.ibb.co/7RJ2Wbs/results-33-2.png)\n\n## What is in this notebook?\n* Data distributions: 1D, 2D and 3D\n* Functions to transform between camera coordinates and road coordinates\n* Simple CenterNet baseline\n\n## CenterNet\nThis architecture predicts centers of objects as a heatmap.  \nIt predicts sizes of the boxes as a regression task.  \n![](https://github.com/xingyizhou/CenterNet/raw/master/readme/fig2.png)\n\nIt is also used for pose estimation:\n![](https://raw.githubusercontent.com/xingyizhou/CenterNet/master/readme/pose3.png)\n*(images from the [original repository](https://github.com/xingyizhou/CenterNet))*  \nCoordinates of human joints are also predicted using regression.  \n\nI use this idea to predict `x, y, z` coordinates of the vehicle and also `yaw, pitch_cos, pitch_sin, roll` angles.  \nFor `pitch` I predict sin and cos, because, as we will see, this angle can be both near 0 and near 3.14.  \nThese 7 parameters are my regression target variables instead of `shift_x, shift_y, size_x, size_y`.  ","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport cv2\nfrom tqdm import tqdm#_notebook as tqdm\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom functools import reduce\nimport os\nfrom scipy.optimize import minimize\nimport plotly.express as px\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.metrics import mean_absolute_error, mean_squared_error\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim import lr_scheduler\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import models\nfrom torchvision import transforms, utils\nimport sys\n\nsys.path.append('../input/sam-optimizer-pytorch')\n#from sam import SAM\n\n\nPATH = '../input/pku-autonomous-driving/'\nos.listdir(PATH)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-05-29T17:53:53.449569Z","iopub.execute_input":"2021-05-29T17:53:53.450052Z","iopub.status.idle":"2021-05-29T17:53:57.277687Z","shell.execute_reply.started":"2021-05-29T17:53:53.449878Z","shell.execute_reply":"2021-05-29T17:53:57.276876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load data","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(PATH + 'train.csv')\ntest = pd.read_csv(PATH + 'sample_submission.csv')\n\n# From camera.zip\ncamera_matrix = np.array([[2304.5479, 0,  1686.2379],\n                          [0, 2305.8757, 1354.9849],\n                          [0, 0, 1]], dtype=np.float32)\ncamera_matrix_inv = np.linalg.inv(camera_matrix)\n\ntrain.head()","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:53:57.280983Z","iopub.execute_input":"2021-05-29T17:53:57.281218Z","iopub.status.idle":"2021-05-29T17:53:57.398377Z","shell.execute_reply.started":"2021-05-29T17:53:57.281173Z","shell.execute_reply":"2021-05-29T17:53:57.397566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**ImageId** column contains names of images:","metadata":{}},{"cell_type":"code","source":"def imread(path, fast_mode=False):\n    img = cv2.imread(path)\n    if not fast_mode and img is not None and len(img.shape) == 3:\n        img = np.array(img[:, :, ::-1])\n    return img\n\nimg = imread(PATH + 'train_images/ID_8a6e65317' + '.jpg')\nIMG_SHAPE = img.shape\n\nplt.figure(figsize=(15,8))\nplt.imshow(img);","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:53:57.399767Z","iopub.execute_input":"2021-05-29T17:53:57.400254Z","iopub.status.idle":"2021-05-29T17:53:58.499530Z","shell.execute_reply.started":"2021-05-29T17:53:57.400063Z","shell.execute_reply":"2021-05-29T17:53:58.498699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**PredictionString** column contains pose information about all cars  \n\nFrom the data description:\n> The primary data is images of cars and related pose information. The pose information is formatted as strings, as follows:  \n>\n> `model type, yaw, pitch, roll, x, y, z`  \n>\n> A concrete example with two cars in the photo:  \n>\n> `5 0.5 0.5 0.5 0.0 0.0 0.0 32 0.25 0.25 0.25 0.5 0.4 0.7`  \n\nWe will need a function to extract these values:","metadata":{}},{"cell_type":"code","source":"def str2coords(s, names=['id', 'yaw', 'pitch', 'roll', 'x', 'y', 'z']):\n    '''\n    Input:\n        s: PredictionString (e.g. from train dataframe)\n        names: array of what to extract from the string\n    Output:\n        list of dicts with keys from `names`\n    '''\n    coords = []\n    for l in np.array(s.split()).reshape([-1, 7]):\n        coords.append(dict(zip(names, l.astype('float'))))\n        if 'id' in coords[-1]:\n            coords[-1]['id'] = int(coords[-1]['id'])\n    return coords","metadata":{"execution":{"iopub.status.busy":"2021-05-29T17:53:58.500611Z","iopub.execute_input":"2021-05-29T17:53:58.500850Z","iopub.status.idle":"2021-05-29T17:53:58.509816Z","shell.execute_reply.started":"2021-05-29T17:53:58.500814Z","shell.execute_reply":"2021-05-29T17:53:58.508982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"inp = train['PredictionString'][0]\nprint('Example input:\\n', inp)\nprint()\nprint('Output:\\n', str2coords(inp))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:53:58.513047Z","iopub.execute_input":"2021-05-29T17:53:58.513514Z","iopub.status.idle":"2021-05-29T17:53:58.528470Z","shell.execute_reply.started":"2021-05-29T17:53:58.513467Z","shell.execute_reply":"2021-05-29T17:53:58.527635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"points_df = pd.DataFrame()\nfor col in ['x', 'y', 'z', 'yaw', 'pitch', 'roll']:\n    arr = []\n    for ps in train['PredictionString']:\n        coords = str2coords(ps)\n        arr += [c[col] for c in coords]\n    points_df[col] = arr\n\nprint('len(points_df)', len(points_df))\npoints_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-05-29T17:53:58.531733Z","iopub.execute_input":"2021-05-29T17:53:58.531965Z","iopub.status.idle":"2021-05-29T17:54:02.699492Z","shell.execute_reply.started":"2021-05-29T17:53:58.531924Z","shell.execute_reply":"2021-05-29T17:54:02.698706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"points_df = pd.DataFrame()\nfor col in ['x', 'y', 'z', 'yaw', 'pitch', 'roll']:\n    arr = []\n    for ps in train['PredictionString']:\n        coords = str2coords(ps)\n        arr += [c[col] for c in coords]\n    points_df[col] = arr\n\nprint('len(points_df)', len(points_df))\npoints_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-05-29T17:54:02.700824Z","iopub.execute_input":"2021-05-29T17:54:02.701121Z","iopub.status.idle":"2021-05-29T17:54:06.763982Z","shell.execute_reply.started":"2021-05-29T17:54:02.701077Z","shell.execute_reply":"2021-05-29T17:54:06.763356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def rotate(x, angle):\n    x = x + angle\n    x = x - (x + np.pi) // (2 * np.pi) * 2 * np.pi\n    return x\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:06.765306Z","iopub.execute_input":"2021-05-29T17:54:06.765593Z","iopub.status.idle":"2021-05-29T17:54:06.770506Z","shell.execute_reply.started":"2021-05-29T17:54:06.765547Z","shell.execute_reply":"2021-05-29T17:54:06.769558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2D Visualization","metadata":{}},{"cell_type":"code","source":"def get_img_coords(s):\n    '''\n    Input is a PredictionString (e.g. from train dataframe)\n    Output is two arrays:\n        xs: x coordinates in the image (row)\n        ys: y coordinates in the image (column)\n    '''\n    coords = str2coords(s)\n    xs = [c['x'] for c in coords]\n    ys = [c['y'] for c in coords]\n    zs = [c['z'] for c in coords]\n    P = np.array(list(zip(xs, ys, zs))).T\n    img_p = np.dot(camera_matrix, P).T\n    img_p[:, 0] /= img_p[:, 2]\n    img_p[:, 1] /= img_p[:, 2]\n    img_xs = img_p[:, 0]\n    img_ys = img_p[:, 1]\n    img_zs = img_p[:, 2] # z = Distance from the camera\n    return img_xs, img_ys\n\nplt.figure(figsize=(14,14))\nplt.imshow(imread(PATH + 'train_images/' + train['ImageId'][2217] + '.jpg'))\nplt.scatter(*get_img_coords(train['PredictionString'][2217]), color='red', s=100);","metadata":{"execution":{"iopub.status.busy":"2021-05-29T17:54:06.771927Z","iopub.execute_input":"2021-05-29T17:54:06.772340Z","iopub.status.idle":"2021-05-29T17:54:07.980777Z","shell.execute_reply.started":"2021-05-29T17:54:06.772175Z","shell.execute_reply":"2021-05-29T17:54:07.979752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One point is out of image!","metadata":{}},{"cell_type":"markdown","source":"Let's look at the distribution of all points. Image is here just for reference.","metadata":{}},{"cell_type":"code","source":"xs, ys = [], []\n\nfor ps in train['PredictionString']:\n    x, y = get_img_coords(ps)\n    xs += list(x)\n    ys += list(y)\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:07.982233Z","iopub.execute_input":"2021-05-29T17:54:07.982792Z","iopub.status.idle":"2021-05-29T17:54:08.763628Z","shell.execute_reply.started":"2021-05-29T17:54:07.982491Z","shell.execute_reply":"2021-05-29T17:54:08.762905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Many points are outside!","metadata":{}},{"cell_type":"markdown","source":"Let's look at this distribution \"from the sky\"","metadata":{}},{"cell_type":"code","source":"# Road points\nroad_width = 3\nroad_xs = [-road_width, road_width, road_width, -road_width, -road_width]\nroad_ys = [0, 0, 500, 500, 0]\n\nplt.figure(figsize=(16,16))\nplt.axes().set_aspect(1)\nplt.xlim(-50,50)\nplt.ylim(0,100)\n\n# View road\nplt.fill(road_xs, road_ys, alpha=0.2, color='gray')\nplt.plot([road_width/2,road_width/2], [0,100], alpha=0.4, linewidth=4, color='white', ls='--')\nplt.plot([-road_width/2,-road_width/2], [0,100], alpha=0.4, linewidth=4, color='white', ls='--')\n# View cars\nplt.scatter(points_df['x'], np.sqrt(points_df['z']**2 + points_df['y']**2), color='red', s=10, alpha=0.1);","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:08.765273Z","iopub.execute_input":"2021-05-29T17:54:08.765712Z","iopub.status.idle":"2021-05-29T17:54:09.740088Z","shell.execute_reply.started":"2021-05-29T17:54:08.765538Z","shell.execute_reply":"2021-05-29T17:54:09.739152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"3d distribution of points:","metadata":{}},{"cell_type":"markdown","source":"1) `x` is measured from left to right  \n2) I thought that `y` is the distance from the car and `z` is height above the road. Looks like this is not the case.  ","metadata":{}},{"cell_type":"markdown","source":"Let's look how good these points lay in one plane  \nTry to predict `y` knowing `x, z`:","metadata":{}},{"cell_type":"code","source":"zy_slope = LinearRegression()\nX = points_df[['z']]\ny = points_df['y']\nzy_slope.fit(X, y)\nprint('MAE without x:', mean_absolute_error(y, zy_slope.predict(X)))\n\n# Will use this model later\nxzy_slope = LinearRegression()\nX = points_df[['x', 'z']]\ny = points_df['y']\nxzy_slope.fit(X, y)\nprint('MAE with x:', mean_absolute_error(y, xzy_slope.predict(X)))\n\nprint('\\ndy/dx = {:.3f}\\ndy/dz = {:.3f}'.format(*xzy_slope.coef_))","metadata":{"execution":{"iopub.status.busy":"2021-05-29T17:54:09.741492Z","iopub.execute_input":"2021-05-29T17:54:09.741913Z","iopub.status.idle":"2021-05-29T17:54:09.797338Z","shell.execute_reply.started":"2021-05-29T17:54:09.741726Z","shell.execute_reply":"2021-05-29T17:54:09.796548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,16))\nplt.xlim(0,500)\nplt.ylim(0,100)\nplt.scatter(points_df['z'], points_df['y'], label='Real points')\nX_line = np.linspace(0,500, 10)\nplt.plot(X_line, zy_slope.predict(X_line.reshape(-1, 1)), color='orange', label='Regression')\nplt.legend()\nplt.xlabel('z coordinate')\nplt.ylabel('y coordinate');","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:09.798627Z","iopub.execute_input":"2021-05-29T17:54:09.799095Z","iopub.status.idle":"2021-05-29T17:54:11.114447Z","shell.execute_reply.started":"2021-05-29T17:54:09.799044Z","shell.execute_reply":"2021-05-29T17:54:11.113807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3D Visualization\nUsed code from https://www.kaggle.com/zstusnoopy/visualize-the-location-and-3d-bounding-box-of-car, but made it one function","metadata":{}},{"cell_type":"code","source":"from math import sin, cos\n\n# convert euler angle to rotation matrix\ndef euler_to_Rot(yaw, pitch, roll):\n    Y = np.array([[cos(yaw), 0, sin(yaw)],\n                  [0, 1, 0],\n                  [-sin(yaw), 0, cos(yaw)]])\n    P = np.array([[1, 0, 0],\n                  [0, cos(pitch), -sin(pitch)],\n                  [0, sin(pitch), cos(pitch)]])\n    R = np.array([[cos(roll), -sin(roll), 0],\n                  [sin(roll), cos(roll), 0],\n                  [0, 0, 1]])\n    return np.dot(Y, np.dot(P, R))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:11.115710Z","iopub.execute_input":"2021-05-29T17:54:11.116013Z","iopub.status.idle":"2021-05-29T17:54:11.124932Z","shell.execute_reply.started":"2021-05-29T17:54:11.115957Z","shell.execute_reply":"2021-05-29T17:54:11.123106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def draw_line(image, points):\n    color = (255, 0, 0)\n    cv2.line(image, tuple(points[0][:2]), tuple(points[3][:2]), color, 16)\n    cv2.line(image, tuple(points[0][:2]), tuple(points[1][:2]), color, 16)\n    cv2.line(image, tuple(points[1][:2]), tuple(points[2][:2]), color, 16)\n    cv2.line(image, tuple(points[2][:2]), tuple(points[3][:2]), color, 16)\n    return image\n\n\ndef draw_points(image, points):\n    for (p_x, p_y, p_z) in points:\n        cv2.circle(image, (p_x, p_y), int(1000 / p_z), (0, 255, 0), -1)\n#         if p_x > image.shape[1] or p_y > image.shape[0]:\n#             print('Point', p_x, p_y, 'is out of image with shape', image.shape)\n    return image","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:11.126522Z","iopub.execute_input":"2021-05-29T17:54:11.127107Z","iopub.status.idle":"2021-05-29T17:54:11.138999Z","shell.execute_reply.started":"2021-05-29T17:54:11.126776Z","shell.execute_reply":"2021-05-29T17:54:11.138007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def visualize(img, coords):\n    # You will also need functions from the previous cells\n    x_l = 1.02\n    y_l = 0.80\n    z_l = 2.31\n    \n    img = img.copy()\n    for point in coords:\n        # Get values\n        x, y, z = point['x'], point['y'], point['z']\n        yaw, pitch, roll = -point['pitch'], -point['yaw'], -point['roll']\n        # Math\n        Rt = np.eye(4)\n        t = np.array([x, y, z])\n        Rt[:3, 3] = t\n        Rt[:3, :3] = euler_to_Rot(yaw, pitch, roll).T\n        Rt = Rt[:3, :]\n        P = np.array([[x_l, -y_l, -z_l, 1],\n                      [x_l, -y_l, z_l, 1],\n                      [-x_l, -y_l, z_l, 1],\n                      [-x_l, -y_l, -z_l, 1],\n                      [0, 0, 0, 1]]).T\n        img_cor_points = np.dot(camera_matrix, np.dot(Rt, P))\n        img_cor_points = img_cor_points.T\n        img_cor_points[:, 0] /= img_cor_points[:, 2]\n        img_cor_points[:, 1] /= img_cor_points[:, 2]\n        img_cor_points = img_cor_points.astype(int)\n        # Drawing\n        img = draw_line(img, img_cor_points)\n        img = draw_points(img, img_cor_points[-1:])\n    \n    return img","metadata":{"execution":{"iopub.status.busy":"2021-05-29T17:54:11.140701Z","iopub.execute_input":"2021-05-29T17:54:11.141314Z","iopub.status.idle":"2021-05-29T17:54:11.154739Z","shell.execute_reply.started":"2021-05-29T17:54:11.141248Z","shell.execute_reply":"2021-05-29T17:54:11.153513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_rows = 6\n\nfor idx in range(n_rows):\n    fig, axes = plt.subplots(1, 2, figsize=(20,20))\n    img = imread(PATH + 'train_images/' + train['ImageId'].iloc[idx] + '.jpg')\n    axes[0].imshow(img)\n    img_vis = visualize(img, str2coords(train['PredictionString'].iloc[idx]))\n    axes[1].imshow(img_vis)\n    plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:11.156313Z","iopub.execute_input":"2021-05-29T17:54:11.156766Z","iopub.status.idle":"2021-05-29T17:54:20.424514Z","shell.execute_reply.started":"2021-05-29T17:54:11.156654Z","shell.execute_reply":"2021-05-29T17:54:20.423340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Image preprocessing","metadata":{}},{"cell_type":"code","source":"IMG_WIDTH = 1024\nIMG_HEIGHT = IMG_WIDTH // 16 * 5\nMODEL_SCALE = 8\n\ndef _regr_preprocess(regr_dict, flip=False):\n    if flip:\n        for k in ['x', 'pitch', 'roll']:\n            regr_dict[k] = -regr_dict[k]\n    for name in ['x', 'y', 'z']:\n        regr_dict[name] = regr_dict[name] / 100\n    regr_dict['roll'] = rotate(regr_dict['roll'], np.pi)\n    regr_dict['pitch_sin'] = sin(regr_dict['pitch'])\n    regr_dict['pitch_cos'] = cos(regr_dict['pitch'])\n    regr_dict.pop('pitch')\n    regr_dict.pop('id')\n    return regr_dict\n\ndef _regr_back(regr_dict):\n    for name in ['x', 'y', 'z']:\n        regr_dict[name] = regr_dict[name] * 100\n    regr_dict['roll'] = rotate(regr_dict['roll'], -np.pi)\n    \n    pitch_sin = regr_dict['pitch_sin'] / np.sqrt(regr_dict['pitch_sin']**2 + regr_dict['pitch_cos']**2)\n    pitch_cos = regr_dict['pitch_cos'] / np.sqrt(regr_dict['pitch_sin']**2 + regr_dict['pitch_cos']**2)\n    regr_dict['pitch'] = np.arccos(pitch_cos) * np.sign(pitch_sin)\n    return regr_dict\n\ndef preprocess_image(img, flip=False):\n    img = img[img.shape[0] // 2:]\n    bg = np.ones_like(img) * img.mean(1, keepdims=True).astype(img.dtype)\n    bg = bg[:, :img.shape[1] // 6]\n    img = np.concatenate([bg, img, bg], 1)\n    img = cv2.resize(img, (IMG_WIDTH, IMG_HEIGHT))\n    if flip:\n        img = img[:,::-1]\n    return (img / 255).astype('float32')\n\ndef get_mask_and_regr(img, labels, flip=False):\n    mask = np.zeros([IMG_HEIGHT // MODEL_SCALE, IMG_WIDTH // MODEL_SCALE], dtype='float32')\n    regr_names = ['x', 'y', 'z', 'yaw', 'pitch', 'roll']\n    regr = np.zeros([IMG_HEIGHT // MODEL_SCALE, IMG_WIDTH // MODEL_SCALE, 7], dtype='float32')\n    coords = str2coords(labels)\n    xs, ys = get_img_coords(labels)\n    for x, y, regr_dict in zip(xs, ys, coords):\n        x, y = y, x\n        x = (x - img.shape[0] // 2) * IMG_HEIGHT / (img.shape[0] // 2) / MODEL_SCALE\n        x = np.round(x).astype('int')\n        y = (y + img.shape[1] // 6) * IMG_WIDTH / (img.shape[1] * 4/3) / MODEL_SCALE\n        y = np.round(y).astype('int')\n        if x >= 0 and x < IMG_HEIGHT // MODEL_SCALE and y >= 0 and y < IMG_WIDTH // MODEL_SCALE:\n            mask[x, y] = 1\n            regr_dict = _regr_preprocess(regr_dict, flip)\n            regr[x, y] = [regr_dict[n] for n in sorted(regr_dict)]\n    if flip:\n        mask = np.array(mask[:,::-1])\n        regr = np.array(regr[:,::-1])\n    return mask, regr","metadata":{"execution":{"iopub.status.busy":"2021-05-29T17:54:20.426051Z","iopub.execute_input":"2021-05-29T17:54:20.426562Z","iopub.status.idle":"2021-05-29T17:54:20.452145Z","shell.execute_reply.started":"2021-05-29T17:54:20.426310Z","shell.execute_reply":"2021-05-29T17:54:20.451061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img0 = imread(PATH + 'train_images/' + train['ImageId'][0] + '.jpg')\nimg = preprocess_image(img0)\n\nmask, regr = get_mask_and_regr(img0, train['PredictionString'][0])\n\nprint('img.shape', img.shape, 'std:', np.std(img))\nprint('mask.shape', mask.shape, 'std:', np.std(mask))\nprint('regr.shape', regr.shape, 'std:', np.std(regr))\n\nplt.figure(figsize=(16,16))\nplt.title('Processed image')\nplt.imshow(img)\nplt.show()\n\nplt.figure(figsize=(16,16))\nplt.title('Detection Mask')\nplt.imshow(mask)\nplt.show()\n\nplt.figure(figsize=(16,16))\nplt.title('Yaw values')\nplt.imshow(regr[:,:,-2])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-05-29T17:54:20.453698Z","iopub.execute_input":"2021-05-29T17:54:20.454143Z","iopub.status.idle":"2021-05-29T17:54:21.742217Z","shell.execute_reply.started":"2021-05-29T17:54:20.453969Z","shell.execute_reply":"2021-05-29T17:54:21.741243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Define functions to convert back from 2d map to 3d coordinates and angles","metadata":{}},{"cell_type":"code","source":"DISTANCE_THRESH_CLEAR = 2\n\ndef convert_3d_to_2d(x, y, z, fx = 2304.5479, fy = 2305.8757, cx = 1686.2379, cy = 1354.9849):\n    # stolen from https://www.kaggle.com/theshockwaverider/eda-visualization-baseline\n    return x * fx / z + cx, y * fy / z + cy\n\ndef optimize_xy(r, c, x0, y0, z0, flipped=False):\n    def distance_fn(xyz):\n        x, y, z = xyz\n        xx = -x if flipped else x\n        slope_err = (xzy_slope.predict([[xx,z]])[0] - y)**2\n        x, y = convert_3d_to_2d(x, y, z)\n        y, x = x, y\n        x = (x - IMG_SHAPE[0] // 2) * IMG_HEIGHT / (IMG_SHAPE[0] // 2) / MODEL_SCALE\n        y = (y + IMG_SHAPE[1] // 6) * IMG_WIDTH / (IMG_SHAPE[1] * 4 / 3) / MODEL_SCALE\n        return max(0.2, (x-r)**2 + (y-c)**2) + max(0.4, slope_err)\n    \n    res = minimize(distance_fn, [x0, y0, z0], method='Powell')\n    x_new, y_new, z_new = res.x\n    return x_new, y_new, z_new\n\ndef clear_duplicates(coords):\n    for c1 in coords:\n        xyz1 = np.array([c1['x'], c1['y'], c1['z']])\n        for c2 in coords:\n            xyz2 = np.array([c2['x'], c2['y'], c2['z']])\n            distance = np.sqrt(((xyz1 - xyz2)**2).sum())\n            if distance < DISTANCE_THRESH_CLEAR:\n                if c1['confidence'] < c2['confidence']:\n                    c1['confidence'] = -1\n    return [c for c in coords if c['confidence'] > 0]\n\ndef extract_coords(prediction, flipped=False):\n    logits = prediction[0]\n    regr_output = prediction[1:]\n    points = np.argwhere(logits > 0)\n    col_names = sorted(['x', 'y', 'z', 'yaw', 'pitch_sin', 'pitch_cos', 'roll'])\n    coords = []\n    for r, c in points:\n        regr_dict = dict(zip(col_names, regr_output[:, r, c]))\n        coords.append(_regr_back(regr_dict))\n        coords[-1]['confidence'] = 1 / (1 + np.exp(-logits[r, c]))\n        coords[-1]['x'], coords[-1]['y'], coords[-1]['z'] = \\\n                optimize_xy(r, c,\n                            coords[-1]['x'],\n                            coords[-1]['y'],\n                            coords[-1]['z'], flipped)\n    coords = clear_duplicates(coords)\n    return coords\n\ndef coords2str(coords, names=['yaw', 'pitch', 'roll', 'x', 'y', 'z', 'confidence']):\n    s = []\n    for c in coords:\n        for n in names:\n            s.append(str(c.get(n, 0)))\n    return ' '.join(s)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:21.743696Z","iopub.execute_input":"2021-05-29T17:54:21.744109Z","iopub.status.idle":"2021-05-29T17:54:21.774726Z","shell.execute_reply.started":"2021-05-29T17:54:21.744050Z","shell.execute_reply":"2021-05-29T17:54:21.773833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ensure that all the forward and back transformations work consistently","metadata":{}},{"cell_type":"code","source":"for idx in range(2):\n    fig, axes = plt.subplots(1, 2, figsize=(20,20))\n    \n    for ax_i in range(2):\n        img0 = imread(PATH + 'train_images/' + train['ImageId'].iloc[idx] + '.jpg')\n        if ax_i == 1:\n            img0 = img0[:,::-1]\n        img = preprocess_image(img0, ax_i==1)\n        mask, regr = get_mask_and_regr(img0, train['PredictionString'][idx], ax_i==1)\n        regr = np.rollaxis(regr, 2, 0)\n        coords = extract_coords(np.concatenate([mask[None], regr], 0), ax_i==1)\n        \n        axes[ax_i].set_title('Flip = {}'.format(ax_i==1))\n        axes[ax_i].imshow(visualize(img0, coords))\n    plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:21.776349Z","iopub.execute_input":"2021-05-29T17:54:21.777038Z","iopub.status.idle":"2021-05-29T17:54:26.749499Z","shell.execute_reply.started":"2021-05-29T17:54:21.776814Z","shell.execute_reply":"2021-05-29T17:54:26.746038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# PyTorch Dataset","metadata":{}},{"cell_type":"code","source":"class CarDataset(Dataset):\n    \"\"\"Car dataset.\"\"\"\n\n    def __init__(self, dataframe, root_dir, training=True, transform=None,hasIDs=False):\n        self.df = dataframe\n        self.root_dir = root_dir\n        self.transform = transform\n        self.training = training\n        self.hasIDs=hasIDs\n\n    def __len__(self):\n        return len(self.df)\n\n    def __getitem__(self, idx):\n        if torch.is_tensor(idx):\n            idx = idx.tolist()\n        \n        # Get image name\n        idx, labels = self.df.values[idx]\n        img_name = self.root_dir.format(idx)\n        \n        # Augmentation\n        flip = False\n        if self.training:\n            flip = np.random.randint(10) == 1\n        \n        # Read image\n        img0 = imread(img_name, True)\n        img = preprocess_image(img0, flip=flip)\n        img = np.rollaxis(img, 2, 0)\n        \n        # Get mask and regression maps\n        mask, regr = get_mask_and_regr(img0, labels, flip=flip)\n        regr = np.rollaxis(regr, 2, 0)\n        if self.hasIDs:\n            return [idx,img]\n        return [img, mask, regr]","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:26.750986Z","iopub.execute_input":"2021-05-29T17:54:26.751454Z","iopub.status.idle":"2021-05-29T17:54:26.766167Z","shell.execute_reply.started":"2021-05-29T17:54:26.751265Z","shell.execute_reply":"2021-05-29T17:54:26.765308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_images_dir = PATH + 'train_images/{}.jpg'\ntest_images_dir = PATH + 'test_images/{}.jpg'\n\ndf_train, df_dev = train_test_split(train, test_size=0.01, random_state=42)\ndf_test = test\n\n# Create dataset objects\ntrain_dataset = CarDataset(df_train, train_images_dir, training=True)\ndev_dataset = CarDataset(df_dev, train_images_dir, training=False)\ndev_dataset2 = CarDataset(df_dev, train_images_dir, training=False,hasIDs=True)\ntest_dataset = CarDataset(df_test, test_images_dir, training=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:26.767862Z","iopub.execute_input":"2021-05-29T17:54:26.768311Z","iopub.status.idle":"2021-05-29T17:54:26.785147Z","shell.execute_reply.started":"2021-05-29T17:54:26.768129Z","shell.execute_reply":"2021-05-29T17:54:26.783957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Show some generated examples","metadata":{}},{"cell_type":"code","source":"img, mask, regr = train_dataset[0]\n\nplt.figure(figsize=(16,16))\nplt.imshow(np.rollaxis(img, 0, 3))\nplt.show()\n\nplt.figure(figsize=(16,16))\nplt.imshow(mask)\nplt.show()\n\nplt.figure(figsize=(16,16))\nplt.imshow(regr[-2])\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:26.786956Z","iopub.execute_input":"2021-05-29T17:54:26.787401Z","iopub.status.idle":"2021-05-29T17:54:27.780404Z","shell.execute_reply.started":"2021-05-29T17:54:26.787210Z","shell.execute_reply":"2021-05-29T17:54:27.779594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"BATCH_SIZE = 4\nval_batch_size=6\n# Create data generators - they will produce batches\ntrain_loader = DataLoader(dataset=train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)\ndev_loader = DataLoader(dataset=dev_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=0)\ndev_loader2 = DataLoader(dataset=dev_dataset2, batch_size=val_batch_size, shuffle=False, num_workers=4)\ntest_loader = DataLoader(dataset=test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=0)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:27.781767Z","iopub.execute_input":"2021-05-29T17:54:27.782066Z","iopub.status.idle":"2021-05-29T17:54:27.791760Z","shell.execute_reply.started":"2021-05-29T17:54:27.782019Z","shell.execute_reply":"2021-05-29T17:54:27.790934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# PyTorch Model","metadata":{}},{"cell_type":"code","source":"!pip install efficientnet-pytorch","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:27.793327Z","iopub.execute_input":"2021-05-29T17:54:27.793998Z","iopub.status.idle":"2021-05-29T17:54:35.733021Z","shell.execute_reply.started":"2021-05-29T17:54:27.793770Z","shell.execute_reply":"2021-05-29T17:54:35.732188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from efficientnet_pytorch import EfficientNet","metadata":{"execution":{"iopub.status.busy":"2021-05-29T17:54:35.734707Z","iopub.execute_input":"2021-05-29T17:54:35.735024Z","iopub.status.idle":"2021-05-29T17:54:35.746163Z","shell.execute_reply.started":"2021-05-29T17:54:35.734976Z","shell.execute_reply":"2021-05-29T17:54:35.745392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class double_conv(nn.Module):\n    '''(conv => BN => ReLU) * 2'''\n    def __init__(self, in_ch, out_ch):\n        super(double_conv, self).__init__()\n        self.conv = nn.Sequential(\n            nn.Conv2d(in_ch, out_ch, 3, padding=1),\n            nn.BatchNorm2d(out_ch),\n            nn.ReLU(inplace=True),\n            nn.Conv2d(out_ch, out_ch, 3, padding=1),\n            nn.BatchNorm2d(out_ch),\n            nn.ReLU(inplace=True)\n        )\n\n    def forward(self, x):\n        x = self.conv(x)\n        return x\n\nclass up(nn.Module):\n    def __init__(self, in_ch, out_ch, bilinear=True):\n        super(up, self).__init__()\n\n        #  would be a nice idea if the upsampling could be learned too,\n        #  but my machine do not have enough memory to handle all those weights\n        if bilinear:\n            self.up = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)\n        else:\n            self.up = nn.ConvTranspose2d(in_ch//2, in_ch//2, 2, stride=2)\n\n        self.conv = double_conv(in_ch, out_ch)\n\n    def forward(self, x1, x2=None):\n        x1 = self.up(x1)\n        \n        # input is CHW\n        diffY = x2.size()[2] - x1.size()[2]\n        diffX = x2.size()[3] - x1.size()[3]\n\n        x1 = F.pad(x1, (diffX // 2, diffX - diffX//2,\n                        diffY // 2, diffY - diffY//2))\n        \n        # for padding issues, see \n        # https://github.com/HaiyongJiang/U-Net-Pytorch-Unstructured-Buggy/commit/0e854509c2cea854e247a9c615f175f76fbb2e3a\n        # https://github.com/xiaopeng-liao/Pytorch-UNet/commit/8ebac70e633bac59fc22bb5195e513d5832fb3bd\n        \n        if x2 is not None:\n            x = torch.cat([x2, x1], dim=1)\n        else:\n            x = x1\n        x = self.conv(x)\n        return x\n\ndef get_mesh(batch_size, shape_x, shape_y):\n    mg_x, mg_y = np.meshgrid(np.linspace(0, 1, shape_y), np.linspace(0, 1, shape_x))\n    mg_x = np.tile(mg_x[None, None, :, :], [batch_size, 1, 1, 1]).astype('float32')\n    mg_y = np.tile(mg_y[None, None, :, :], [batch_size, 1, 1, 1]).astype('float32')\n    mesh = torch.cat([torch.tensor(mg_x).to(device), torch.tensor(mg_y).to(device)], 1)\n    return mesh","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:35.748022Z","iopub.execute_input":"2021-05-29T17:54:35.748368Z","iopub.status.idle":"2021-05-29T17:54:35.767781Z","shell.execute_reply.started":"2021-05-29T17:54:35.748315Z","shell.execute_reply":"2021-05-29T17:54:35.766800Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class MyUNet(nn.Module):\n    '''Mixture of previous classes'''\n    def __init__(self, n_classes):\n        super(MyUNet, self).__init__()\n        self.base_model = EfficientNet.from_pretrained('efficientnet-b0')\n        \n        self.conv0 = double_conv(5, 64)\n        self.conv1 = double_conv(64, 128)\n        self.conv2 = double_conv(128, 512)\n        self.conv3 = double_conv(512, 1024)\n        \n        self.mp = nn.MaxPool2d(2)\n        \n        self.up1 = up(1282 + 1024, 512)\n        self.up2 = up(512 + 512, 256)\n        self.outc = nn.Conv2d(256, n_classes, 1)\n\n    def forward(self, x):\n        batch_size = x.shape[0]\n        mesh1 = get_mesh(batch_size, x.shape[2], x.shape[3])\n        x0 = torch.cat([x, mesh1], 1)\n        x1 = self.mp(self.conv0(x0))\n        x2 = self.mp(self.conv1(x1))\n        x3 = self.mp(self.conv2(x2))\n        x4 = self.mp(self.conv3(x3))\n        \n        x_center = x[:, :, :, IMG_WIDTH // 8: -IMG_WIDTH // 8]\n        feats = self.base_model.extract_features(x_center)\n        bg = torch.zeros([feats.shape[0], feats.shape[1], feats.shape[2], feats.shape[3] // 8]).to(device)\n        feats = torch.cat([bg, feats, bg], 3)\n        \n        # Add positional info\n        mesh2 = get_mesh(batch_size, feats.shape[2], feats.shape[3])\n        feats = torch.cat([feats, mesh2], 1)\n        \n        x = self.up1(feats, x4)\n        x = self.up2(x, x3)\n        x = self.outc(x)\n        return x","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:35.769673Z","iopub.execute_input":"2021-05-29T17:54:35.770185Z","iopub.status.idle":"2021-05-29T17:54:35.785891Z","shell.execute_reply.started":"2021-05-29T17:54:35.769974Z","shell.execute_reply":"2021-05-29T17:54:35.785070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class SAM(torch.optim.Optimizer):\n    def __init__(self, params, base_optimizer, rho=0.05, **kwargs):\n        assert rho >= 0.0, f\"Invalid rho, should be non-negative: {rho}\"\n\n        defaults = dict(rho=rho, **kwargs)\n        super(SAM, self).__init__(params, defaults)\n\n        self.base_optimizer = base_optimizer(self.param_groups, **kwargs)\n        self.param_groups = self.base_optimizer.param_groups\n\n    @torch.no_grad()\n    def first_step(self, zero_grad=False):\n        grad_norm = self._grad_norm()\n        for group in self.param_groups:\n            scale = group[\"rho\"] / (grad_norm + 1e-12)\n\n            for p in group[\"params\"]:\n                if p.grad is None: continue\n                e_w = p.grad * scale.to(p)\n                p.add_(e_w)  # climb to the local maximum \"w + e(w)\"\n                self.state[p][\"e_w\"] = e_w\n\n        if zero_grad: self.zero_grad()\n\n    @torch.no_grad()\n    def second_step(self, zero_grad=False):\n        for group in self.param_groups:\n            for p in group[\"params\"]:\n                if p.grad is None: continue\n                p.sub_(self.state[p][\"e_w\"])  # get back to \"w\" from \"w + e(w)\"\n\n        self.base_optimizer.step()  # do the actual \"sharpness-aware\" update\n\n        if zero_grad: self.zero_grad()\n\n    @torch.no_grad()\n    def step(self, closure=None):\n        assert closure is not None, \"Sharpness Aware Minimization requires closure, but it was not provided\"\n        closure = torch.enable_grad()(closure)  # the closure should do a full forward-backward pass\n\n        self.first_step(zero_grad=True)\n        closure()\n        self.second_step()\n\n    def _grad_norm(self):\n        shared_device = self.param_groups[0][\"params\"][0].device  # put everything on the same device, in case of model parallelism\n        norm = torch.norm(\n                    torch.stack([\n                        p.grad.norm(p=2).to(shared_device)\n                        for group in self.param_groups for p in group[\"params\"]\n                        if p.grad is not None\n                    ]),\n                    p=2\n               )\n        return norm","metadata":{"execution":{"iopub.status.busy":"2021-05-29T17:54:35.787341Z","iopub.execute_input":"2021-05-29T17:54:35.788448Z","iopub.status.idle":"2021-05-29T17:54:35.805355Z","shell.execute_reply.started":"2021-05-29T17:54:35.787615Z","shell.execute_reply":"2021-05-29T17:54:35.804592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Gets the GPU if there is one, otherwise the cpu\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(device)\n\nn_epochs = 10\n\nmodel = MyUNet(8).to(device)\n#base_optimizer = optim.Adam(model.parameters(), lr=0.001)\nbase_optimizer = optim.Adam\noptimizer = SAM(model.parameters(), base_optimizer, lr=0.001)\nexp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=max(n_epochs, 10) * len(train_loader) // 3, gamma=0.1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:35.806734Z","iopub.execute_input":"2021-05-29T17:54:35.807064Z","iopub.status.idle":"2021-05-29T17:54:40.779708Z","shell.execute_reply.started":"2021-05-29T17:54:35.807006Z","shell.execute_reply":"2021-05-29T17:54:40.778840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{}},{"cell_type":"code","source":"def criterion(prediction, mask, regr, size_average=True):\n    # Binary mask loss\n    pred_mask = torch.sigmoid(prediction[:, 0])\n#     mask_loss = mask * (1 - pred_mask)**2 * torch.log(pred_mask + 1e-12) + (1 - mask) * pred_mask**2 * torch.log(1 - pred_mask + 1e-12)\n    mask_loss = mask * torch.log(pred_mask + 1e-12) + (1 - mask) * torch.log(1 - pred_mask + 1e-12)\n    mask_loss = -mask_loss.mean(0).sum()\n    \n    # Regression L1 loss\n    pred_regr = prediction[:, 1:]\n    regr_loss = (torch.abs(pred_regr - regr).sum(1) * mask).sum(1).sum(1) / mask.sum(1).sum(1)\n    regr_loss = regr_loss.mean(0)\n    \n    # Sum\n    loss = mask_loss + regr_loss\n    if not size_average:\n        loss *= prediction.shape[0]\n    return loss","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:40.784641Z","iopub.execute_input":"2021-05-29T17:54:40.784887Z","iopub.status.idle":"2021-05-29T17:54:40.793974Z","shell.execute_reply.started":"2021-05-29T17:54:40.784831Z","shell.execute_reply":"2021-05-29T17:54:40.792948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#copy from https://www.kaggle.com/its7171/metrics-evaluation-script\nfrom math import sqrt, acos, pi, sin, cos\nfrom scipy.spatial.transform import Rotation as R\nfrom sklearn.metrics import average_precision_score\ndef expand_df(df, PredictionStringCols):\n    df = df.dropna().copy()\n    df['NumCars'] = [int((x.count(' ')+1)/7) for x in df['PredictionString']]\n    #fix nan bug\n    df['PredictionString'].fillna('0.15699866 0.029135812202187722 -3.0876481888168534 15.061413706416168 4.49235176722995 27.384573221206665 0.6447297909095615',inplace=True)\n    image_id_expanded = [item for item, count in zip(df['ImageId'], df['NumCars']) for i in range(count)]\n    prediction_strings_expanded = df['PredictionString'].str.split(' ',expand = True).values.reshape(-1,7).astype(float)\n    prediction_strings_expanded = prediction_strings_expanded[~np.isnan(prediction_strings_expanded).all(axis=1)]\n    df = pd.DataFrame(\n        {\n            'ImageId': image_id_expanded,\n            PredictionStringCols[0]:prediction_strings_expanded[:,0],\n            PredictionStringCols[1]:prediction_strings_expanded[:,1],\n            PredictionStringCols[2]:prediction_strings_expanded[:,2],\n            PredictionStringCols[3]:prediction_strings_expanded[:,3],\n            PredictionStringCols[4]:prediction_strings_expanded[:,4],\n            PredictionStringCols[5]:prediction_strings_expanded[:,5],\n            PredictionStringCols[6]:prediction_strings_expanded[:,6]\n        })\n    return df\n\ndef str2coords2(s, names):\n    coords = []\n    for l in np.array(s.split()).reshape([-1, 7]):\n        coords.append(dict(zip(names, l.astype('float'))))\n    return coords\n\ndef TranslationDistance(p,g, abs_dist = False):\n    dx = p['x'] - g['x']\n    dy = p['y'] - g['y']\n    dz = p['z'] - g['z']\n    diff0 = (g['x']**2 + g['y']**2 + g['z']**2)**0.5\n    diff1 = (dx**2 + dy**2 + dz**2)**0.5\n    if abs_dist:\n        diff = diff1\n    else:\n        diff = diff1/diff0\n    return diff\n\ndef RotationDistance(p, g):\n    true=[ g['pitch'] ,g['yaw'] ,g['roll'] ]\n    pred=[ p['pitch'] ,p['yaw'] ,p['roll'] ]\n    q1 = R.from_euler('xyz', true)\n    q2 = R.from_euler('xyz', pred)\n    diff = R.inv(q2) * q1\n    W = np.clip(diff.as_quat()[-1], -1., 1.)\n    \n    # in the official metrics code:\n    # https://www.kaggle.com/c/pku-autonomous-driving/overview/evaluation\n    #   return Object3D.RadianToDegree( Math.Acos(diff.W) )\n    # this code treat θ and θ+2π differntly.\n    # So this should be fixed as follows.\n    W = (acos(W)*360)/pi\n    if W > 180:\n        W = 180 - W\n    return W\n\ndef check_match(valid_df, train_df, thre_tr_dist, thre_ro_dist,keep_gt=False):\n    train_dict = {imgID:str2coords2(s, names=['carid_or_score', 'pitch', 'yaw', 'roll', 'x', 'y', 'z']) for imgID,s in zip(train_df['ImageId'],train_df['PredictionString'])}\n    valid_dict = {imgID:str2coords2(s, names=['pitch', 'yaw', 'roll', 'x', 'y', 'z', 'carid_or_score']) for imgID,s in zip(valid_df['ImageId'],valid_df['PredictionString'])}\n    result_flg = [] # 1 for TP, 0 for FP\n    scores = []\n    MAX_VAL = 10**10\n    for img_id in valid_dict:\n        for pcar in sorted(valid_dict[img_id], key=lambda x: -x['carid_or_score']):\n            # find nearest GT\n            min_tr_dist = MAX_VAL\n            min_idx = -1\n            for idx, gcar in enumerate(train_dict[img_id]):\n                tr_dist = TranslationDistance(pcar,gcar)\n                if tr_dist < min_tr_dist:\n                    min_tr_dist = tr_dist\n                    min_ro_dist = RotationDistance(pcar,gcar)\n                    min_idx = idx\n                    \n            # set the result\n            if min_tr_dist < thre_tr_dist and min_ro_dist < thre_ro_dist:\n                if not keep_gt:\n                    train_dict[img_id].pop(min_idx)\n                result_flg.append(1)\n            else:\n                result_flg.append(0)\n            scores.append(pcar['carid_or_score'])\n    \n    return result_flg, scores\n\n\n\ndef calc_map_df(valid_df, nrows=None):\n#     valid_df = pd.read_csv(infile, nrows=nrows)\n    expanded_valid_df = expand_df(valid_df, ['pitch','yaw','roll','x','y','z','Score'])\n#     valid_df = valid_df.fillna('')\n\n    train_df = pd.read_csv('../input/pku-autonomous-driving/train.csv')\n    train_df = train_df[train_df.ImageId.isin(valid_df.ImageId.unique())]\n    # data description page says, The pose information is formatted as\n    # model type, yaw, pitch, roll, x, y, z\n    # but it doesn't, and it should be\n    # model type, pitch, yaw, roll, x, y, z\n    expanded_train_df = expand_df(train_df, ['model_type','pitch','yaw','roll','x','y','z'])\n    n_gt = len(expanded_train_df)\n\n    thres_ro_list = [50, 45, 40, 35, 30, 25, 20, 15, 10, 5]\n    thres_tr_list = [0.1, 0.09, 0.08, 0.07, 0.06, 0.05, 0.04, 0.03, 0.02, 0.01]\n    ap_list = []\n    for thre_ro_dist,thre_tr_dist in tqdm(zip(thres_ro_list,thres_tr_list)):\n        abs_dist = False\n        result_flg, scores = check_match(valid_df, train_df, thre_tr_dist, thre_ro_dist)\n        n_tp = np.sum(result_flg)\n        recall = n_tp/n_gt\n        ap = average_precision_score(result_flg, scores)*recall\n        ap_list.append(ap)\n    return np.mean(ap_list)","metadata":{"execution":{"iopub.status.busy":"2021-05-29T17:54:40.795775Z","iopub.execute_input":"2021-05-29T17:54:40.796218Z","iopub.status.idle":"2021-05-29T17:54:40.831891Z","shell.execute_reply.started":"2021-05-29T17:54:40.796042Z","shell.execute_reply":"2021-05-29T17:54:40.830925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#For validating,we fill some fixed codes in PredictionString field when it's nan\ndef fill_str(x):\n    if type(x)==float or len(x)<2:\n        return '0.15511163 0.025993261021686774 -3.1062442382150373 -15.10751805129461 12.073826862286817 70.47340792740864 0.5496648404696726'\n    return x\n\n\ndef evaluate_model_metrics(epoch=None, history=None):\n    model.eval()\n    ids=[]\n    preds=[]\n    \n\n    \n    with torch.no_grad():\n        for ids_batch,img_batch in tqdm(dev_loader2):\n            img_batch = img_batch.to(device)\n            output = model(img_batch).cpu().numpy()\n            ids.extend(ids_batch)\n            for out in output:\n                predictions=[]\n                coords = extract_coords(out,0.)#0.2\n                s = coords2str(coords)\n                predictions.append(s)\n                preds.append(' '.join(predictions))\n        \n        \n#     for idx in tqdm(range(len(dev_dataset))):\n#         predictions=[]\n#         img, mask, regr = dev_dataset[idx]    \n#         output = model(torch.tensor(img[None]).to(device)).data.cpu().numpy()\n#         imgId=df_dev['ImageId'].iloc[idx]\n#         ids.append(imgId)\n#         for out in output:\n#             coords = extract_coords(out,0.)#0.2\n#             s = coords2str(coords)\n#             predictions.append(s)\n#         preds.append(' '.join(predictions))\n\n    torch.cuda.empty_cache()  \n    validation_prediction='valid_preds.csv'\n    sub1=pd.DataFrame()\n    sub1['ImageId']=ids\n    sub1['PredictionString']=preds\n    #fix nan bug\n    sub1['PredictionString']=sub1['PredictionString'].apply(fill_str)\n#     sub1['PredictionString'].fillna('0.15511163 0.025993261021686774 -3.1062442382150373 -15.10751805129461 12.073826862286817 70.47340792740864 0.5496648404696726',inplace=True)\n    sub1.to_csv(validation_prediction,index=False)\n#     map = calc_map(validation_prediction, nrows=None)\n    map=calc_map_df(sub1,nrows=None)\n    print('map:', map) ","metadata":{"execution":{"iopub.status.busy":"2021-05-29T17:54:40.833166Z","iopub.execute_input":"2021-05-29T17:54:40.833493Z","iopub.status.idle":"2021-05-29T17:54:40.845380Z","shell.execute_reply.started":"2021-05-29T17:54:40.833417Z","shell.execute_reply":"2021-05-29T17:54:40.844336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_model(epoch, history=None):\n    model.train()\n    def closure():\n        loss = loss_function(output, model(input))\n        loss.backward()\n        return loss\n  \n    for batch_idx, (img_batch, mask_batch, regr_batch) in enumerate(tqdm(train_loader)):\n        img_batch = img_batch.to(device)\n        mask_batch = mask_batch.to(device)\n        regr_batch = regr_batch.to(device)\n        \n        \n        output = model(img_batch)\n        loss = criterion(output, mask_batch, regr_batch)\n        if history is not None:\n            history.loc[epoch + batch_idx / len(train_loader), 'train_loss'] = loss.data.cpu().numpy()\n        \n        loss.backward()\n        optimizer.first_step(zero_grad=True)\n        \n\n        preds_second = model(img_batch)\n        loss_second = criterion(preds_second, mask_batch, regr_batch)\n            \n        loss_second.backward()\n        optimizer.second_step(zero_grad=True)\n\n\n        #optimizer.step()\n        exp_lr_scheduler.step()\n    \n    print('Train Epoch: {} \\tLR: {:.6f}\\tLoss: {:.6f}'.format(\n        epoch,\n        optimizer.state_dict()['param_groups'][0]['lr'],\n        loss.data))\n\ndef evaluate_model(epoch, history=None):\n    model.eval()\n    loss = 0\n    \n    with torch.no_grad():\n        for img_batch, mask_batch, regr_batch in dev_loader:\n            img_batch = img_batch.to(device)\n            mask_batch = mask_batch.to(device)\n            regr_batch = regr_batch.to(device)\n\n            output = model(img_batch)\n\n            loss += criterion(output, mask_batch, regr_batch, size_average=False).data\n    \n    loss /= len(dev_loader.dataset)\n    \n    if history is not None:\n        history.loc[epoch, 'dev_loss'] = loss.cpu().numpy()\n    \n    print('Dev loss: {:.4f}'.format(loss))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:40.847051Z","iopub.execute_input":"2021-05-29T17:54:40.847590Z","iopub.status.idle":"2021-05-29T17:54:40.864567Z","shell.execute_reply.started":"2021-05-29T17:54:40.847426Z","shell.execute_reply":"2021-05-29T17:54:40.863960Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nimport gc\n\nhistory = pd.DataFrame()\n\nfor epoch in range(n_epochs):\n    torch.cuda.empty_cache()\n    gc.collect()\n    train_model(epoch, history)\n#     evaluate_model(epoch, history)\n    evaluate_model_metrics(epoch, history)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-05-29T17:54:40.865913Z","iopub.execute_input":"2021-05-29T17:54:40.866908Z","iopub.status.idle":"2021-05-29T18:13:52.077458Z","shell.execute_reply.started":"2021-05-29T17:54:40.866265Z","shell.execute_reply":"2021-05-29T18:13:52.075396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# evaluate_model_metrics(epoch, history)","metadata":{"execution":{"iopub.status.busy":"2021-05-29T18:13:52.078999Z","iopub.execute_input":"2021-05-29T18:13:52.079291Z","iopub.status.idle":"2021-05-29T18:13:52.084585Z","shell.execute_reply.started":"2021-05-29T18:13:52.079241Z","shell.execute_reply":"2021-05-29T18:13:52.083632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#torch.save(model.state_dict(), './model.pth')","metadata":{"execution":{"iopub.status.busy":"2021-05-29T18:13:52.085981Z","iopub.execute_input":"2021-05-29T18:13:52.086483Z","iopub.status.idle":"2021-05-29T18:13:52.099238Z","shell.execute_reply.started":"2021-05-29T18:13:52.086281Z","shell.execute_reply":"2021-05-29T18:13:52.098476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history['train_loss'].iloc[100:].plot();","metadata":{"execution":{"iopub.status.busy":"2021-05-29T18:13:52.101035Z","iopub.execute_input":"2021-05-29T18:13:52.101494Z","iopub.status.idle":"2021-05-29T18:13:52.265396Z","shell.execute_reply.started":"2021-05-29T18:13:52.101306Z","shell.execute_reply":"2021-05-29T18:13:52.264271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# series = history.dropna()['dev_loss']\n# plt.scatter(series.index, series);","metadata":{"execution":{"iopub.status.busy":"2021-05-29T18:13:52.266932Z","iopub.execute_input":"2021-05-29T18:13:52.267374Z","iopub.status.idle":"2021-05-29T18:13:52.271381Z","shell.execute_reply.started":"2021-05-29T18:13:52.267189Z","shell.execute_reply":"2021-05-29T18:13:52.270298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualize predictions","metadata":{}},{"cell_type":"code","source":"img, mask, regr = dev_dataset[0]\n\nplt.figure(figsize=(16,16))\nplt.title('Input image')\nplt.imshow(np.rollaxis(img, 0, 3))\nplt.show()\n\nplt.figure(figsize=(16,16))\nplt.title('Ground truth mask')\nplt.imshow(mask)\nplt.show()\n\noutput = model(torch.tensor(img[None]).to(device))\nlogits = output[0,0].data.cpu().numpy()\n\nplt.figure(figsize=(16,16))\nplt.title('Model predictions')\nplt.imshow(logits)\nplt.show()\n\nplt.figure(figsize=(16,16))\nplt.title('Model predictions thresholded')\nplt.imshow(logits > 0)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-29T18:13:52.272850Z","iopub.execute_input":"2021-05-29T18:13:52.273293Z","iopub.status.idle":"2021-05-29T18:13:53.513824Z","shell.execute_reply.started":"2021-05-29T18:13:52.273122Z","shell.execute_reply":"2021-05-29T18:13:53.512903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"torch.cuda.empty_cache()\ngc.collect()\n\nfor idx in range(8):\n    img, mask, regr = dev_dataset[idx]\n    \n    output = model(torch.tensor(img[None]).to(device)).data.cpu().numpy()\n    coords_pred = extract_coords(output[0])\n    coords_true = extract_coords(np.concatenate([mask[None], regr], 0))\n    \n    img = imread(train_images_dir.format(df_dev['ImageId'].iloc[idx]))\n    \n    fig, axes = plt.subplots(1, 2, figsize=(30,30))\n    axes[0].set_title('Ground truth')\n    axes[0].imshow(visualize(img, coords_true))\n    axes[1].set_title('Prediction')\n    axes[1].imshow(visualize(img, coords_pred))\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-05-29T18:13:53.515117Z","iopub.execute_input":"2021-05-29T18:13:53.515433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Make submission","metadata":{}},{"cell_type":"code","source":"predictions = []\n\ntest_loader = DataLoader(dataset=test_dataset, batch_size=4, shuffle=False, num_workers=4)\n\nmodel.eval()\n\nfor img, _, _ in tqdm(test_loader):\n    with torch.no_grad():\n        output = model(img.to(device))\n    output = output.data.cpu().numpy()\n    for out in output:\n        coords = extract_coords(out)\n        s = coords2str(coords)\n        predictions.append(s)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = pd.read_csv(PATH + 'sample_submission.csv')\ntest['PredictionString'] = predictions\ntest.to_csv('predictions.csv', index=False)\ntest.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}