{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Peking University/Baidu - Autonomous Driving Competition\n## References:\n- Papers : </br>\n     - Centernet paper https://arxiv.org/pdf/1904.07850.pdf </br>\n     - U-net Paper https://arxiv.org/pdf/1505.04597.pdf </br>\n     - 3D Bounding Box Estimation Using Deep Learning and Geometry Paper https://arxiv.org/pdf/1612.00496.pdf</br>\n- Kernels :</br>\n     - https://www.kaggle.com/mobassir/baidu-autonomous-driving-eda </br>\n     - https://www.kaggle.com/mobassir/center-resnext50-baseline </br>\n     - https://www.kaggle.com/diegojohnson/centernet-objects-as-points </br>\n     - https://www.kaggle.com/hocop1/centernet-baseline </br>\n\n     \n## what is this competition about ? </br>\nYou are given The dataset contains photos of streets, taken from the roof of a car.</br>\n- ~4k training images\n- We're attempting to predict the position and orientation of all un-masked cars in the test images. we should also provide a confidence score indicating how sure we are of your prediction.\n\n\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n'''\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n'''\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This libarary is for image augmentation Link: https://github.com/albumentations-team/albumentations"},{"metadata":{"trusted":true},"cell_type":"code","source":"#!pip install -U git+https://github.com/albu/albumentations\n!pip install -U albumentations --user \n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport cv2\nfrom tqdm import tqdm#_notebook as tqdm\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom functools import reduce\nimport os\nfrom scipy.optimize import minimize\nimport plotly.express as px\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.metrics import mean_absolute_error, mean_squared_error\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim import lr_scheduler\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import models\nfrom torchvision import transforms, utils\n\nPATH = '../input/pku-autonomous-driving/'\nos.listdir(PATH)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If you want to understand Camera Matrix,the Intrinsic Matrix\nLink : http://ksimek.github.io/2013/08/13/intrinsic/"},{"metadata":{},"cell_type":"markdown","source":"## Loading the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(PATH + 'train.csv')\ntest = pd.read_csv(PATH + 'sample_submission.csv')\n\n# From camera.zip\ncamera_matrix = np.array([[2304.5479, 0,  1686.2379],\n                          [0, 2305.8757, 1354.9849],\n                          [0, 0, 1]], dtype=np.float32)\ncamera_matrix_inv = np.linalg.inv(camera_matrix)\n\ntrain.head()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def imread(path, fast_mode=False):\n    img = cv2.imread(path)\n    if not fast_mode and img is not None and len(img.shape) == 3:\n        img = np.array(img[:, :, ::-1])\n    return img\n\nimg = imread(PATH + 'train_images/ID_5f4dac207' + '.jpg')\nIMG_SHAPE = img.shape\n\nplt.figure(figsize=(15,8))\nplt.imshow(img);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Extracting data \n PredictionString column contains pose information about all cars\n From the data description:\n The primary data is images of cars and related pose information. The pose information is formatted as    strings, as follows:\n\nmodel type, yaw, pitch, roll, x, y, z\n\nThis function extracts these values:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def str2coords(s, names=['id', 'yaw', 'pitch', 'roll', 'x', 'y', 'z']):\n    '''\n    Input:\n        s: PredictionString (e.g. from train dataframe)\n        names: array of what to extract from the string\n    Output:\n        list of dicts with keys from `names`\n    '''\n    coords = []\n    for l in np.array(s.split()).reshape([-1, 7]):\n        coords.append(dict(zip(names, l.astype('float'))))\n        if 'id' in coords[-1]:\n            coords[-1]['id'] = int(coords[-1]['id'])\n    return coords","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lens = [len(str2coords(s)) for s in train['PredictionString']]\n\nplt.figure(figsize=(15,6))\nsns.countplot(lens);\nplt.xlabel('Number of cars in image');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"points_df = pd.DataFrame()\nfor col in ['x', 'y', 'z', 'yaw', 'pitch', 'roll']:\n    arr = []\n    for ps in train['PredictionString']:\n        coords = str2coords(ps)\n        arr += [c[col] for c in coords]\n    points_df[col] = arr\n\nprint('len(points_df)', len(points_df))\npoints_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plot position information distribution\nto gain insight into the dataset Let's plot distribution of the position information "},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,6))\nsns.distplot(points_df['x'], bins=500);\nplt.xlabel('x')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,6))\nsns.distplot(points_df['y'], bins=500);\nplt.xlabel('y')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,6))\nsns.distplot(points_df['z'], bins=500);\nplt.xlabel('z')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,6))\nsns.distplot(points_df['yaw'], bins=500);\nplt.xlabel('yaw')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,6))\nsns.distplot(points_df['pitch'], bins=500);\nplt.xlabel('pitch')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,6))\nsns.distplot(points_df['roll'], bins=500);\nplt.xlabel('roll rotated by pi')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def rotate(x, angle):\n    x = x + angle\n    x = x - (x + np.pi) // (2 * np.pi) * 2 * np.pi\n    return x\n\nplt.figure(figsize=(15,6))\nsns.distplot(points_df['roll'].map(lambda x: rotate(x, np.pi)), bins=500);\nplt.xlabel('roll rotated by pi')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Some notes about the data \n- You notice something strange in the data. the pitch and yaw are mixed up in this dataset. Pitch cannot take thoses big values. That would mean that cars are upside down.\n- We use function rotate in roll angle because rotation by Pi is the same as rotation by - pi"},{"metadata":{},"cell_type":"markdown","source":"# projecting 3d position on 2d image \nRefer to this discussion to understand how to project 3d position on 2d image :https://www.kaggle.com/c/pku-autonomous-driving/discussion/120083"},{"metadata":{},"cell_type":"markdown","source":"# 2D Visualization"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_img_coords(s):\n    '''\n    Input is a PredictionString (e.g. from train dataframe)\n    Output is two arrays:\n        xs: x coordinates in the image\n        ys: y coordinates in the image\n    '''\n    coords = str2coords(s)\n    xs = [c['x'] for c in coords]\n    ys = [c['y'] for c in coords]\n    zs = [c['z'] for c in coords]\n    P = np.array(list(zip(xs, ys, zs))).T\n    img_p = np.dot(camera_matrix, P).T\n    img_p[:, 0] /= img_p[:, 2]\n    img_p[:, 1] /= img_p[:, 2]\n    #img_p[:, 0] /= zs\n    #img_p[:, 1] /= zs\n    \n    img_xs = img_p[:, 0]\n    img_ys = img_p[:, 1]\n    img_zs = img_p[:, 2] # z = Distance from the camera\n    return img_xs, img_ys\n\nplt.figure(figsize=(14,14))\nplt.imshow(imread(PATH + 'train_images/' + train['ImageId'][500] + '.jpg'))\nplt.scatter(*get_img_coords(train['PredictionString'][500]), color='red', s=100);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"xs, ys = [], []\n\nfor ps in train['PredictionString']:\n    x, y = get_img_coords(ps)\n    xs += list(x)\n    ys += list(y)\n\nplt.figure(figsize=(18,18))\nplt.imshow(imread(PATH + 'train_images/' + train['ImageId'][500] + '.jpg'), alpha=0.3)\nplt.scatter(xs, ys, color='red', s=10, alpha=0.2);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"zy_slope = LinearRegression()\nX = points_df[['z']]\ny = points_df['y']\nzy_slope.fit(X, y)\nprint('MAE without x:', mean_absolute_error(y, zy_slope.predict(X)))\n\n# Will use this model later\nxzy_slope = LinearRegression()\nX = points_df[['x', 'z']]\ny = points_df['y']\nxzy_slope.fit(X, y)\nprint('MAE with x:', mean_absolute_error(y, xzy_slope.predict(X)))\n\nprint('\\ndy/dx = {:.3f}\\ndy/dz = {:.3f}'.format(*xzy_slope.coef_))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,16))\nplt.xlim(0,500)\nplt.ylim(0,100)\nplt.scatter(points_df['z'], points_df['y'], label='Real points')\nX_line = np.linspace(0,500, 10)\nplt.plot(X_line, zy_slope.predict(X_line.reshape(-1, 1)), color='orange', label='Regression')\nplt.legend()\nplt.xlabel('z coordinate')\nplt.ylabel('y coordinate');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3D Visualization"},{"metadata":{},"cell_type":"markdown","source":"## Rotation matrix \nRefer to this link to understand the transformation from euler angles (yaw,pith,roll) to Rotation matrix : https://www.phas.ubc.ca/~berciu/TEACHING/PHYS206/LECTURES/FILES/euler.pdf"},{"metadata":{"trusted":true},"cell_type":"code","source":"from math import sin, cos\n\n# convert euler angle to rotation matrix\ndef euler_to_Rot(yaw, pitch, roll):\n    Y = np.array([[cos(yaw), 0, sin(yaw)],\n                  [0, 1, 0],\n                  [-sin(yaw), 0, cos(yaw)]])\n    P = np.array([[1, 0, 0],\n                  [0, cos(pitch), -sin(pitch)],\n                  [0, sin(pitch), cos(pitch)]])\n    R = np.array([[cos(roll), -sin(roll), 0],\n                  [sin(roll), cos(roll), 0],\n                  [0, 0, 1]])\n    return np.dot(Y, np.dot(P, R))\n\ndef draw_line(image, points):\n    color = (255, 0, 0)\n    cv2.line(image, tuple(points[0][:2]), tuple(points[3][:2]), color, 16)\n    cv2.line(image, tuple(points[0][:2]), tuple(points[1][:2]), color, 16)\n    cv2.line(image, tuple(points[1][:2]), tuple(points[2][:2]), color, 16)\n    cv2.line(image, tuple(points[2][:2]), tuple(points[3][:2]), color, 16)\n    return image\n\n\ndef draw_points(image, points):\n    for (p_x, p_y, p_z) in points:\n        cv2.circle(image, (p_x, p_y), int(1000 / p_z), (0, 255, 0), -1)\n#         if p_x > image.shape[1] or p_y > image.shape[0]:\n#             print('Point', p_x, p_y, 'is out of image with shape', image.shape)\n    return image\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize(img, coords):\n    # You will also need functions from the previous cells\n    x_l = 1.02\n    y_l = 0.80\n    z_l = 2.31\n    \n    img = img.copy()\n    for point in coords:\n        # Get values\n        x, y, z = point['x'], point['y'], point['z']\n        yaw, pitch, roll = -point['pitch'], -point['yaw'], -point['roll']\n        # Math\n        center_point = np.array([x, y, z]).reshape([1,3])\n        Rotation_matrix = euler_to_Rot(yaw, pitch, roll).T#Rotation matrix to transform from car coordinate frame to camera coordinate frame\n        bounding_box = np.array([[x_l, -y_l, -z_l],\n                      [x_l, -y_l, z_l],\n                      [-x_l, -y_l, z_l],\n                      [-x_l, -y_l, -z_l],\n                     ]).T\n        img_cor_points = np.dot(camera_matrix, np.dot(Rotation_matrix,bounding_box)+center_point.T)\n        img_cor_points = img_cor_points.T\n        img_cor_points[:, 0] /= img_cor_points[:, 2]\n        img_cor_points[:, 1] /= img_cor_points[:, 2]\n        img_cor_points = img_cor_points.astype(int)\n        # Drawing\n        img = draw_line(img, img_cor_points)\n        img_point=np.dot(camera_matrix, center_point.T).T\n        img_point[:, 0] /= img_point[:, 2]\n        img_point[:, 1] /=img_point[:, 2]\n        img = draw_points(img,img_point.astype(int))\n    \n    return img\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_rows = 6\n\nfor idx in range(n_rows):\n    fig, axes = plt.subplots(1, 2, figsize=(20,20))\n    img = imread(PATH + 'train_images/' + train['ImageId'].iloc[10+idx] + '.jpg')\n    axes[0].imshow(img)\n    img_vis = visualize(img, str2coords(train['PredictionString'].iloc[10+idx]))\n    axes[1].imshow(img_vis)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Image preprocessing\n- regr_preprocess: process the x,y,z,yaw,pitch,roll, e.g. x = x/100, before feed into the network\n- regr_back undo the process, e.g. x = x*100 after get prediction from network\n- extract_coords: extract information from network\n- x = (x - img.shape[0] // 2) * IMGHEIGHT / (img.shape[0] // 2) / MODELSCALE\n- if you look carefully, the output mask is different dimension from the input image, so this calculation  map coordinate from larger image to smaller mask\nThis image decribes that \n![GitHub Logo](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F114413%2F4dc7064479fa77f077c657bb7fc184db%2FUntitled.png?generation=1573531132825427&alt=media)"},{"metadata":{"trusted":true},"cell_type":"code","source":"IMG_WIDTH = 1024\nIMG_HEIGHT = IMG_WIDTH // 16 * 5\nMODEL_SCALE = 8\n\ndef rotate(x, angle):\n    x = x + angle\n    x = x - (x + np.pi) // (2 * np.pi) * 2 * np.pi\n    return x\n\n\ndef _regr_preprocess(regr_dict, flip=False):\n    if flip:\n        for k in ['x', 'pitch', 'roll']:\n            regr_dict[k] = -regr_dict[k]\n    for name in ['x', 'y', 'z']:\n        regr_dict[name] = regr_dict[name] / 100\n    regr_dict['roll'] = rotate(regr_dict['roll'], np.pi)\n    regr_dict['pitch_sin'] = sin(regr_dict['pitch'])\n    regr_dict['pitch_cos'] = cos(regr_dict['pitch'])\n    regr_dict.pop('pitch')\n    regr_dict.pop('id')\n    return regr_dict\n\ndef _regr_back(regr_dict):\n    for name in ['x', 'y', 'z']:\n        regr_dict[name] = regr_dict[name] * 100\n    regr_dict['roll'] = rotate(regr_dict['roll'], -np.pi)\n    \n    pitch_sin = regr_dict['pitch_sin'] / np.sqrt(regr_dict['pitch_sin']**2 + regr_dict['pitch_cos']**2)\n    pitch_cos = regr_dict['pitch_cos'] / np.sqrt(regr_dict['pitch_sin']**2 + regr_dict['pitch_cos']**2)\n    regr_dict['pitch'] = np.arccos(pitch_cos) * np.sign(pitch_sin)\n    return regr_dict\n\ndef preprocess_image(img, flip=False):\n    img = img[img.shape[0] // 2:]\n    bg = np.ones_like(img) * img.mean(1, keepdims=True).astype(img.dtype)\n    bg = bg[:, :img.shape[1] // 6]\n    img = np.concatenate([bg, img, bg], 1)\n    img = cv2.resize(img, (IMG_WIDTH, IMG_HEIGHT))\n    if flip:\n        img = img[:,::-1]\n    return (img / 255).astype('float32')\n\ndef get_mask_and_regr(img, labels, flip=False):\n    mask = np.zeros([IMG_HEIGHT // MODEL_SCALE, IMG_WIDTH // MODEL_SCALE], dtype='float32')\n    regr_names = ['x', 'y', 'z', 'yaw', 'pitch', 'roll']\n    regr = np.zeros([IMG_HEIGHT // MODEL_SCALE, IMG_WIDTH // MODEL_SCALE, 7], dtype='float32')\n    coords = str2coords(labels)\n    xs, ys = get_img_coords(labels)\n    for x, y, regr_dict in zip(xs, ys, coords):\n        x, y = y, x\n        #print(x,img.shape[0] // 2,y, img.shape[1] // 6)\n        x = (x - img.shape[0] // 2) * IMG_HEIGHT / (img.shape[0] // 2) / MODEL_SCALE\n        #x=(x*1/2)*(IMG_HEIGHT / MODEL_SCALE)/(img.shape[0] // 2)\n        x = np.round(x).astype('int')\n        y = (y + img.shape[1] // 6) * IMG_WIDTH / (img.shape[1] * 4/3) / MODEL_SCALE\n        #y=(y* 4/3)*(IMG_WIDTH / MODEL_SCALE)/((img.shape[1] * 3/4) )\n\n        y = np.round(y).astype('int')\n        #print(x,y)\n\n        if x >= 0 and x < IMG_HEIGHT // MODEL_SCALE and y >= 0 and y < IMG_WIDTH // MODEL_SCALE:\n            mask[x, y] = 1\n            regr_dict = _regr_preprocess(regr_dict, flip)\n            regr[x, y] = [regr_dict[n] for n in sorted(regr_dict)]\n    if flip:\n        mask = np.array(mask[:,::-1])\n        regr = np.array(regr[:,::-1])\n    return mask, regr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img0 = imread(PATH + 'train_images/' + train['ImageId'][500] + '.jpg')\nimg = preprocess_image(img0,flip=True)\n\nmask, regr = get_mask_and_regr(img0, train['PredictionString'][0],flip=True)\n\nprint('img.shape', img.shape, 'std:', np.std(img))\nprint('mask.shape', mask.shape, 'std:', np.std(mask))\nprint('regr.shape', regr.shape, 'std:', np.std(regr))\nprint(img[:,::-1].shape)\n\nplt.figure(figsize=(16,16))\nplt.title('Processed image')\nplt.imshow(img)\nplt.show()\n\nplt.figure(figsize=(16,16))\nplt.title('Processed flip image')\nplt.imshow(img[:,::-1])\nplt.show()\n\nplt.figure(figsize=(16,16))\nplt.title('Detection Mask')\nplt.imshow(mask)\nplt.show()\n\nplt.figure(figsize=(16,16))\nplt.title('Yaw values')\nplt.imshow(regr[:,:,-2])\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Define functions to convert back from 2d map to 3d coordinates and angles"},{"metadata":{"trusted":true},"cell_type":"code","source":"DISTANCE_THRESH_CLEAR = 2\n\ndef convert_3d_to_2d(x, y, z, fx = 2304.5479, fy = 2305.8757, cx = 1686.2379, cy = 1354.9849):\n    # stolen from https://www.kaggle.com/theshockwaverider/eda-visualization-baseline\n    return x * fx / z + cx, y * fy / z + cy\n\ndef optimize_xy(r, c, x0, y0, z0, flipped=False):\n    def distance_fn(xyz):\n        x, y, z = xyz\n        xx = -x if flipped else x\n        slope_err = (xzy_slope.predict([[xx,z]])[0] - y)**2\n        x, y = convert_3d_to_2d(x, y, z)\n        y, x = x, y\n        x = (x - IMG_SHAPE[0] // 2) * IMG_HEIGHT / (IMG_SHAPE[0] // 2) / MODEL_SCALE\n        y = (y + IMG_SHAPE[1] // 6) * IMG_WIDTH / (IMG_SHAPE[1] * 4 / 3) / MODEL_SCALE\n        return max(0.2, (x-r)**2 + (y-c)**2) + max(0.4, slope_err)\n    \n    res = minimize(distance_fn, [x0, y0, z0], method='Powell')\n    x_new, y_new, z_new = res.x\n    return x_new, y_new, z_new\n\ndef clear_duplicates(coords):\n    for c1 in coords:\n        xyz1 = np.array([c1['x'], c1['y'], c1['z']])\n        for c2 in coords:\n            xyz2 = np.array([c2['x'], c2['y'], c2['z']])\n            distance = np.sqrt(((xyz1 - xyz2)**2).sum())\n            if distance < DISTANCE_THRESH_CLEAR:\n                if c1['confidence'] < c2['confidence']:\n                    c1['confidence'] = -1\n    return [c for c in coords if c['confidence'] > 0]\n\ndef extract_coords(prediction, flipped=False):\n    logits = prediction[0]\n    \n    regr_output = prediction[1:]\n    points = np.argwhere(logits > 0)\n    col_names = sorted(['x', 'y', 'z', 'yaw', 'pitch_sin', 'pitch_cos', 'roll'])\n    coords = []\n    for r, c in points:\n        regr_dict = dict(zip(col_names, regr_output[:, r, c]))\n        coords.append(_regr_back(regr_dict))\n        coords[-1]['confidence'] = 1 / (1 + np.exp(-logits[r, c]))\n        coords[-1]['x'], coords[-1]['y'], coords[-1]['z'] = \\\n                optimize_xy(r, c,\n                            coords[-1]['x'],\n                            coords[-1]['y'],\n                            coords[-1]['z'], flipped)\n    coords = clear_duplicates(coords)\n    return coords\n\ndef coords2str(coords, names=['yaw', 'pitch', 'roll', 'x', 'y', 'z', 'confidence']):\n    s = []\n    for c in coords:\n        for n in names:\n            s.append(str(c.get(n, 0)))\n    return ' '.join(s)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for idx in range(2):\n    fig, axes = plt.subplots(1, 2, figsize=(20,20))\n    \n    for ax_i in range(2):\n        img0 = imread(PATH + 'train_images/' + train['ImageId'].iloc[10+idx] + '.jpg')\n        if ax_i == 1:\n            img0 = img0[:,::-1]\n        img = preprocess_image(img0, ax_i==1)\n        mask, regr = get_mask_and_regr(img0, train['PredictionString'][10+idx], ax_i==1)\n        \n        regr = np.rollaxis(regr, 2, 0)\n        coords = extract_coords(np.concatenate([mask[None], regr], 0), ax_i==1)\n        \n        axes[ax_i].set_title('Flip = {}'.format(ax_i==1))\n        axes[ax_i].imshow(visualize(img0, coords))\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating Dataset and Dataloader for the dataset \nCreating Dataset and Dataloader for the dataset  to feed it to the Pytorch model </br>\nRefer to this link : https://pytorch.org/tutorials/beginner/data_loading_tutorial.html"},{"metadata":{"trusted":true},"cell_type":"code","source":"import albumentations as A\n\nclass CarDataset(Dataset):\n    \"\"\"Car dataset.\"\"\"\n\n    def __init__(self, dataframe, root_dir, training=True, transform=None):\n        self.df = dataframe\n        self.root_dir = root_dir\n        self.transform = transform\n        self.training = training\n\n    def __len__(self):\n        return len(self.df)\n\n    def __getitem__(self, idx):\n        if torch.is_tensor(idx):\n            idx = idx.tolist()\n        \n        # Get image name\n        idx, labels = self.df.values[idx]\n        img_name = self.root_dir.format(idx)\n        \n        # Augmentation\n        flip = False\n        if self.training:\n            flip = np.random.randint(2) == 1\n        \n        # Read image\n        img0 = imread(img_name, True)\n        if self.training and self.transform:\n           sample = self.transform(image=img0, mask=None)\n           img0=sample['image']\n        img = preprocess_image(img0, flip=flip)\n\n        img = np.rollaxis(img, 2, 0)\n        \n        # Get mask and regression maps\n        mask, regr = get_mask_and_regr(img0, labels, flip=flip)\n        regr = np.rollaxis(regr, 2, 0)\n        \n        \n\n        return [img, mask, regr]\n\ndef get_training_augmentation():\n    train_transform = [\n\n        #A.HorizontalFlip(p=0.5),\n\n        #A.ShiftScaleRotate(scale_limit=0.5, rotate_limit=0, shift_limit=0.1, p=1, border_mode=0),\n\n        #A.PadIfNeeded(min_height=320, min_width=320, always_apply=True, border_mode=0),\n        #A.RandomCrop(height=320, width=320, always_apply=True),\n\n        A.IAAAdditiveGaussianNoise(p=0.2),\n        A.IAAPerspective(p=0.5),\n        A.OneOf(\n            [\n                A.CLAHE(p=1),\n                A.RandomBrightness(p=1),\n                A.RandomGamma(p=1),\n            ],\n            p=0.9,\n        ),\n        A.OneOf(\n            [\n                A.IAASharpen(p=1),\n                A.Blur(blur_limit=3, p=1),\n                A.MotionBlur(blur_limit=3, p=1),\n            ],\n            p=0.9,\n        ),\n\n        \n \n\n    ]\n    return A.Compose(train_transform) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_images_dir = PATH + 'train_images/{}.jpg'\ntest_images_dir = PATH + 'test_images/{}.jpg'\n\ndf_train, df_dev = train_test_split(train, test_size=0.01, random_state=42)\ndf_test = test\n\n# Create dataset objects\ntrain_dataset = CarDataset(df_train, train_images_dir, training=True,transform=get_training_augmentation())\ndev_dataset = CarDataset(df_dev, train_images_dir, training=False)\ntest_dataset = CarDataset(df_test, test_images_dir, training=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img, mask, regr = train_dataset[500]\n\nplt.figure(figsize=(16,16))\nplt.imshow(np.rollaxis(img, 0, 3))\nplt.show()\n\nplt.figure(figsize=(16,16))\nplt.imshow(mask)\nplt.show()\n\nplt.figure(figsize=(16,16))\nplt.imshow(regr[-2])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 4\n\n# Create data generators - they will produce batches\ntrain_loader = DataLoader(dataset=train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)\ndev_loader = DataLoader(dataset=dev_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=0)\ntest_loader = DataLoader(dataset=test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating the Pytorch model\nthe model is U-net model in Pytorch </br>\nRefer to this link for implementation detials: https://github.com/milesial/Pytorch-UNet\nthis main idea of the model to use U-net to predict the center point of the car and regress the rotation angles ."},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install efficientnet-pytorch\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from efficientnet_pytorch import EfficientNet\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class double_conv(nn.Module):\n    '''(conv => BN => ReLU) * 2'''\n    def __init__(self, in_ch, out_ch):\n        super(double_conv, self).__init__()\n        self.conv = nn.Sequential(\n            nn.Conv2d(in_ch, out_ch, 3, padding=1),\n            nn.BatchNorm2d(out_ch),\n            nn.ReLU(inplace=True),\n            nn.Conv2d(out_ch, out_ch, 3, padding=1),\n            nn.BatchNorm2d(out_ch),\n            nn.ReLU(inplace=True)\n        )\n\n    def forward(self, x):\n        x = self.conv(x)\n        return x\n\nclass up(nn.Module):\n    def __init__(self, in_ch, out_ch, bilinear=True):\n        super(up, self).__init__()\n\n        #  would be a nice idea if the upsampling could be learned too,\n        #  but my machine do not have enough memory to handle all those weights\n        if bilinear:\n            self.up = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)\n        else:\n            self.up = nn.ConvTranspose2d(in_ch//2, in_ch//2, 2, stride=2)\n\n        self.conv = double_conv(in_ch, out_ch)\n\n    def forward(self, x1, x2=None):\n        x1 = self.up(x1)\n        \n        # input is CHW\n        diffY = x2.size()[2] - x1.size()[2]\n        diffX = x2.size()[3] - x1.size()[3]\n\n        x1 = F.pad(x1, (diffX // 2, diffX - diffX//2,\n                        diffY // 2, diffY - diffY//2))\n        \n        # for padding issues, see \n        # https://github.com/HaiyongJiang/U-Net-Pytorch-Unstructured-Buggy/commit/0e854509c2cea854e247a9c615f175f76fbb2e3a\n        # https://github.com/xiaopeng-liao/Pytorch-UNet/commit/8ebac70e633bac59fc22bb5195e513d5832fb3bd\n        \n        if x2 is not None:\n            x = torch.cat([x2, x1], dim=1)\n        else:\n            x = x1\n        x = self.conv(x)\n        return x\n\ndef get_mesh(batch_size, shape_x, shape_y):\n    mg_x, mg_y = np.meshgrid(np.linspace(0, 1, shape_y), np.linspace(0, 1, shape_x))\n    mg_x = np.tile(mg_x[None, None, :, :], [batch_size, 1, 1, 1]).astype('float32')\n    mg_y = np.tile(mg_y[None, None, :, :], [batch_size, 1, 1, 1]).astype('float32')\n    mesh = torch.cat([torch.tensor(mg_x).to(device), torch.tensor(mg_y).to(device)], 1)\n    return mesh\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class MyUNet(nn.Module):\n    '''Mixture of previous classes'''\n    def __init__(self, n_classes):\n        super(MyUNet, self).__init__()\n        self.base_model = EfficientNet.from_pretrained('efficientnet-b0')\n        \n        self.conv0 = double_conv(5, 64)\n        self.conv1 = double_conv(64, 128)\n        self.conv2 = double_conv(128, 512)\n        self.conv3 = double_conv(512, 1024)\n        \n        self.mp = nn.MaxPool2d(2)\n        \n        self.up1 = up(1282 + 1024, 512)\n        self.up2 = up(512 + 512, 256)\n        self.outc = nn.Conv2d(256, n_classes, 1)\n\n    def forward(self, x):\n        batch_size = x.shape[0]\n        mesh1 = get_mesh(batch_size, x.shape[2], x.shape[3])\n        x0 = torch.cat([x, mesh1], 1)\n        x1 = self.mp(self.conv0(x0))\n        x2 = self.mp(self.conv1(x1))\n        x3 = self.mp(self.conv2(x2))\n        x4 = self.mp(self.conv3(x3))\n        \n        x_center = x[:, :, :, IMG_WIDTH // 8: -IMG_WIDTH // 8]\n        feats = self.base_model.extract_features(x_center)\n        bg = torch.zeros([feats.shape[0], feats.shape[1], feats.shape[2], feats.shape[3] // 8]).to(device)\n        feats = torch.cat([bg, feats, bg], 3)\n        \n        # Add positional info\n        mesh2 = get_mesh(batch_size, feats.shape[2], feats.shape[3])\n        feats = torch.cat([feats, mesh2], 1)\n        \n        x = self.up1(feats, x4)\n        x = self.up2(x, x3)\n        x = self.outc(x)\n        return x\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(device)\n\nn_epochs = 10\n\nmodel = MyUNet(8).to(device)\noptimizer = optim.Adam(model.parameters(), lr=0.001)\nexp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=max(n_epochs, 10) * len(train_loader) // 3, gamma=0.1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The Loss function\nif you wana know more about custom loss in Pytorch  refer to this link : https://cs230.stanford.edu/blog/pytorch/"},{"metadata":{"trusted":true},"cell_type":"code","source":"def criterion(prediction, mask, regr, size_average=True):\n    # Binary mask loss\n    pred_mask = torch.sigmoid(prediction[:, 0])\n#     mask_loss = mask * (1 - pred_mask)**2 * torch.log(pred_mask + 1e-12) + (1 - mask) * pred_mask**2 * torch.log(1 - pred_mask + 1e-12)\n    mask_loss = mask * torch.log(pred_mask + 1e-12) + (1 - mask) * torch.log(1 - pred_mask + 1e-12)\n    mask_loss = -mask_loss.mean(0).sum()\n    \n    # Regression L1 loss\n    pred_regr = prediction[:, 1:]\n    regr_loss = (torch.abs(pred_regr - regr).sum(1) * mask).sum(1).sum(1) / mask.sum(1).sum(1)\n    regr_loss = regr_loss.mean(0)\n    \n    # Sum\n    loss = mask_loss + regr_loss\n    if not size_average:\n        loss *= prediction.shape[0]\n    return loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_model(epoch, history=None):\n    model.train()\n\n    for batch_idx, (img_batch, mask_batch, regr_batch) in enumerate(tqdm(train_loader)):\n        img_batch = img_batch.to(device)\n        mask_batch = mask_batch.to(device)\n        regr_batch = regr_batch.to(device)\n        \n        optimizer.zero_grad()\n        output = model(img_batch)\n        loss = criterion(output, mask_batch, regr_batch)\n        if history is not None:\n            history.loc[epoch + batch_idx / len(train_loader), 'train_loss'] = loss.data.cpu().numpy()\n        \n        loss.backward()\n        \n        optimizer.step()\n        exp_lr_scheduler.step()\n    \n    print('Train Epoch: {} \\tLR: {:.6f}\\tLoss: {:.6f}'.format(\n        epoch,\n        optimizer.state_dict()['param_groups'][0]['lr'],\n        loss.data))\n\ndef evaluate_model(epoch, history=None):\n    model.eval()\n    loss = 0\n    \n    with torch.no_grad():\n        for img_batch, mask_batch, regr_batch in dev_loader:\n            img_batch = img_batch.to(device)\n            mask_batch = mask_batch.to(device)\n            regr_batch = regr_batch.to(device)\n\n            output = model(img_batch)\n\n            loss += criterion(output, mask_batch, regr_batch, size_average=False).data\n    \n    loss /= len(dev_loader.dataset)\n    \n    if history is not None:\n        history.loc[epoch, 'dev_loss'] = loss.cpu().numpy()\n    \n    print('Dev loss: {:.4f}'.format(loss))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"checkpoint = torch.load('/kaggle/input/trained-model-pku-competition/model_06.pth')\nmodel.load_state_dict(checkpoint['state_dict'])\noptimizer.load_state_dict(checkpoint['optimizer'])\nexp_lr_scheduler.load_state_dict(checkpoint['lr_scheduler'])\nepoch = checkpoint['epoch']\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training the Model \nTraing the model took more than 15 hrs not continous by saving checkpoints and retrain on Colab for more than 30 ephocs and increasing the augmentation ratio with every retraining of the model  "},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nimport gc\nn_epochs=10\nhistory = pd.DataFrame()\n\nfor epoch in range(n_epochs):\n    torch.cuda.empty_cache()\n    gc.collect()\n    train_model(epoch, history)\n    evaluate_model(epoch, history)\n    state = {\n    'epoch': epoch,\n    'state_dict': model.state_dict(),\n    'optimizer': optimizer.state_dict(),\n    'lr_scheduler':exp_lr_scheduler.state_dict()\n    }\n    torch.save(state, '/content/drive/My Drive/model_06.pth')\n'''    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#history['train_loss'].iloc[100:].plot();\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#series = history.dropna()['dev_loss']\n#plt.scatter(series.index, series);\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualize some predictions of the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nimg, mask, regr = dev_dataset[10]\n\nplt.figure(figsize=(16,16))\nplt.title('Input image')\nplt.imshow(np.rollaxis(img, 0, 3))\nplt.show()\n\nplt.figure(figsize=(16,16))\nplt.title('Ground truth mask')\nplt.imshow(mask)\nplt.show()\n\noutput = model(torch.tensor(img[None]).to(device))\nlogits = output[0,0].data.cpu().numpy()\nplt.figure(figsize=(16,16))\nplt.title('Model predictions')\nplt.imshow(logits)\nplt.show()\n\nplt.figure(figsize=(16,16))\nplt.title('Model predictions thresholded')\nplt.imshow(logits > 0)\nplt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\ntorch.cuda.empty_cache()\ngc.collect()\n\nfor idx in range(8):\n    img, mask, regr = dev_dataset[idx]\n    \n    output = model(torch.tensor(img[None]).to(device)).data.cpu().numpy()\n    coords_pred = extract_coords(output[0])\n    coords_true = extract_coords(np.concatenate([mask[None], regr], 0))\n    \n    img = imread(train_images_dir.format(df_dev['ImageId'].iloc[idx]))\n    \n    fig, axes = plt.subplots(1, 2, figsize=(30,30))\n    axes[0].set_title('Ground truth')\n    axes[0].imshow(visualize(img, coords_true))\n    axes[1].set_title('Prediction')\n    axes[1].imshow(visualize(img, coords_pred))\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Make submission file"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\npredictions = []\n\ntest_loader = DataLoader(dataset=test_dataset, batch_size=4, shuffle=False, num_workers=4)\n\nmodel.eval()\n\nfor img, _, _ in tqdm(test_loader):\n    with torch.no_grad():\n        output = model(img.to(device))\n    output = output.data.cpu().numpy()\n    for out in output:\n        coords = extract_coords(out)\n        s = coords2str(coords)\n        predictions.append(s)\n'''        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\ntest = pd.read_csv(PATH + 'sample_submission.csv')\ntest['PredictionString'] = predictions\ntest.to_csv('predictions.csv', index=False)\ntest.head()\n'''","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}