{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Car Yaw prediction\n\nIn this kernel, a NN is built to, given a car image (cropped from the original, using YOLO, for example), predict its yaw. Hope this kernel/model will serve those of you who are using a separated stages to solve the problem.\n\n## 1. Intuition\n\nTo describe the rotation of an object we need three things: Yaw, Pitch and Roll.\n\n![](http://www.formula1-dictionary.net/Images/motion_yaw_pitch_roll.jpg)\n\nAs shown by the data analysis, roll is 0 always. This makes sense as there aren't any cars dumped (in this dataset), so we can suppose roll always $0$.\n\nWith respect to Pitch (elevation of the nose of the car), the distribution of this datasets shows a gaussian with $\\mu \\approx 0.1$ and a low $\\sigma$. The intuition tells us that because of a car is parallel to the ground, this should be 0. The deviation of this dataset seems to be due to the annotation/generation failures. Notwithstanding, we can suppose that Pitch will be always $\\mu = 0.15$.\n\nSo, only Yaw is unknown. This will be our regression target. \n\nIt is expressed as a rotation with range $(+\\pi, -\\pi)$ (in radians), taking $0$ as facing frontwards. The cars are, in most of the cases, facing frontwards or backwards (they are on our lane or on the opposite).\n\nThis way of expressing the target has a problem: wrapping arround edges. That is, if we predict a car is rotated $3.13$ radians and the real target is 0, in reality, we are very close to the target, but the loss would be huge. We need a way to transform the target so we can eliminate the wrapping.\n\nThere is a paper (https://arxiv.org/pdf/1612.00496.pdf) which solution I found clever and very simple. I will explain it here.\n\nFor fixing the problem of wrapping, we first have to convert the rotation to \"global angle\", that is $+\\pi = 180, -\\pi = 180 $ (the negative angles are $360 - \\hat{a}$). Once done that, we divide the space in $N$ bins shifted a little bit (as shown in the image) so that we can exploit the advantage that most of the cars have 0 and 180 degrees.\n\n![](https://i.imgur.com/wNUjHhx.png)\n\nNow, we represent a point as the bin it belongs to plus an offset in degrees. This way, we face a regression + classification task which seems easier. Then, $\\hat{P} = Bin_{nb} * bin_{width} + \\text{offset} + bin_{width}/2$\n\nThe last thing to overview is the \"local view problem\" (following image, taken from the paper). This means that due to camera perspective, a car apparently shows different rotation even thouth its global orientation is constant (0º in the case of the image).\n\n![](https://i.imgur.com/czogKDy.png)\n\nBecause of that, we need to include in our representation the \"ray angle\" of the car, which can be easily calculated as we already have the camera properties (the following image, taken from the paper, ilustrates that). Said this, we will regress the \"local angle\" of the car and later, we will transform it to the global using the following equation: $\\theta_g = \\theta_l + \\text{ray}_{angle}$\n\n> Please, note that in the paper the origin (0 degrees) is taken as right from the camera, I will take it as the front direction (0 degrees will mean that a car is facing frontwards).\n\n![](https://i.imgur.com/r5VjQxS.png)\n\nWith this strategy our regression task becomes way easier as we are operating with data distributed much closer to normal distributions instead of multi modal ones (as the original is).\n\n## 2. Dataset\n\nI built a dataset automatically from the data provided. The dataset contains crops of detected cars from the original images and the information of the ray angles and rotations.\n\nTo build it, I ran a pretrained YOLO on the images and then looked for matches between the extracted crops and the original points of the cars. \n\nYou are free to re-use it for your own approach.\n\nHope this will help you to improve some of your predictions :)\n\n## 3. Model\nThe main idea of the model is to use a \"common part\" which takes the image crop and then, two separate heads: one for the bin classification and other for the angle offset.\n\n> I previously used transfer learning for the common part (VGG, Mobilenet and ResNet), but I got similar results with a smaller custom network (defined below). The dataset is too small to take advantage of these architectures.\n\n## TODO\n\n- Pass the Ray-Angle to the network and see if it improves performance"},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nfrom absl import flags, app\n\nfrom tensorflow.keras import layers, models, optimizers\n\nimport numpy as np\nimport pandas as pd\nimport os\nimport math\nimport uuid\nfrom tqdm import tqdm\nfrom sklearn.metrics import r2_score, confusion_matrix\n\nfrom matplotlib import pyplot as plt\nimport seaborn as sns\n\nimport cv2\nfrom PIL import Image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Camera information + I will use 8 total bins "},{"metadata":{"trusted":true},"cell_type":"code","source":"CAMERA_fx = 2304.5479\nCAMERA_fy = 2305.8757\nCAMERA_cx = 1686.2379\nCAMERA_cy = 1354.9849\n\nCAMERA_FOV = (CAMERA_cx / CAMERA_fx)\nRADS_PIXEL_X = CAMERA_FOV / 3384\n\nNUM_BINS = 8\nIMAGE_INP_SIZE = 64\n\nPATH_DATAFRAME = \"../input/car-rotation-crops/rotation-dataset/data.csv\"\nPATH_IMGS = \"../input/car-rotation-crops/rotation-dataset/img/\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some utility functions to manage data"},{"metadata":{"trusted":true},"cell_type":"code","source":"def to_angle(radian):\n    \"\"\"Takes a +-π radian and transforms in [0, 2π) range\"\"\"\n    if radian < 0:\n        radian = (2*math.pi) + radian\n    return radian\n\ndef to_rotation(radian):\n    \"\"\"Takes an angle and transforms it in +-π range \"\"\"\n    base_angle = 0\n    \n    if math.sin(radian) < 0:\n        base_angle = -(2 * math.pi - radian)\n    else:\n        base_angle = radian - base_angle\n    \n    return base_angle\n\ndef get_local_rot(ray_angle, global_angle):\n    return global_angle - ray_angle\n\ndef get_global_rot(ray_angle, local_angle):\n    return ray_angle + local_angle\n\ndef get_bin(angle):\n    \"\"\"Gets bin nb and offset from that number.\n    params: \n        - angle: Angle in radians [0, 2π)\n    \"\"\"\n    bin_size = 360 / NUM_BINS\n    total_bins = 360//bin_size\n    \n    degrees = math.degrees(angle) + bin_size/2  #Shift the bins\n    bin_number = (degrees // bin_size) % total_bins\n    offset = (degrees - (bin_number*bin_size))\n    \n    if degrees > 360:  #Correct offset if in last semi bin (8 == 0)\n        offset = degrees - ((total_bins) * bin_size)\n    \n    offset = math.radians(offset)\n\n    return bin_number, offset\n\ndef prediction_to_yaw(bin_nb, offset, ray_angle):\n    \"\"\" Takes bin + offset and using the ray angle \n    returns the global rotation of the car \"\"\"\n    bin_size = 2*math.pi / NUM_BINS\n    \n    # Local rotation of the car in [0, 2π)\n    angle = bin_nb * bin_size + offset - bin_size/2  # shift bins\n    \n    # Global rotation of the car (taking into account the camera ray angle)\n    angle = get_global_rot(ray_angle, angle)\n    \n    if angle < 0:\n        angle = 2*math.pi + angle\n    \n    # Represent the angle as a rotation of [0, π] or [0, -π)\n    #angle = to_rotation(angle)\n    \n    return angle\n\ndef angle_distance(angle1, angle2):\n    \"\"\"Returns the shortest distance in degrees between two angles.\n    Parameters:\n        - angle1, angle2: (Degrees)\n    \"\"\"\n    diff = ( angle2 - angle1 + 180 ) % 360 - 180;\n    diff =  diff + 360  if diff < -180 else diff\n    return diff","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Process data"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = pd.read_csv(PATH_DATAFRAME).drop('Unnamed: 0', axis=1)\ndataset['Name'] = dataset.apply(lambda r: PATH_IMGS + r['Name'] +'.jpeg', axis=1)\n\n# Get global angle expressed in [0, 2π] range\ndataset['Global_angle'] = dataset.apply(lambda r: to_angle(r['Global']), axis=1)\n\n# Get ray angle expressed in [0, 2π] range\ndataset['Ray_Angle_angle'] = dataset.apply(lambda r: to_angle(r['Ray_Angle']), axis=1)\n\n# Calculate Local rotation\ndataset['Local'] = dataset['Global_angle'] - dataset['Ray_Angle_angle']\n# Correct the local angle in case the substraction is < 0 (express it in range 0,2π)\ndataset['Local_corrected'] = dataset.apply(lambda r: to_angle(r['Local']), axis=1)\n\n# Get Bins + Offsets\ndataset[['Bin_nb', 'Bin_offset']] = dataset.apply(lambda r: pd.Series(get_bin(r['Local_corrected'])), axis=1)\ndataset['Bin_nb'] = dataset['Bin_nb'].astype('int')\n\n#Normalize bin offset\nmax_off = dataset['Bin_offset'].max()\ndataset['Bin_offset_norm'] = dataset['Bin_offset'] / max_off# if max_off > 1 else dataset['Bin_offset']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The dataset is pretty unbalanced, so in order to mitigate (a little bit) that, I'm going to introduce an intentional bias towards the underrepresented classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Balance bin number to reduce (as much as possible) biases. Cars in 0/4 (facing frontwards/backwards) are the majority\nbins0 = dataset.loc[dataset['Bin_nb'] == 0].head(5000)\nbins4 = dataset.loc[dataset['Bin_nb'] == 4].head(5000)\nothers = dataset.loc[(dataset['Bin_nb'] != 0) & (dataset['Bin_nb'] != 4)]\n\n# Over represent the classes\nnew_dataset_rows = [bins0, bins4]\nnew_dataset_rows.extend([others] * 10)\n\ndataset = pd.concat(new_dataset_rows).sample(frac=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A crop example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# A crop example\nImage.open(dataset.iloc[115,0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(3, figsize=(12,7))\n\nsns.distplot(dataset['Ray_Angle_angle'], ax=ax[0], bins=360)\nax[0].set_title(\"Ray_Angle_angle\")\n\nsns.distplot(dataset['Bin_nb'], ax=ax[1], kde=False)\nax[1].set_title(\"Bin_nb\")\n\nsns.distplot(dataset['Bin_offset_norm'], ax=ax[2])\nax[2].set_title(\"Bin_offset_norm\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train / Validation / Test split\ntrain_mask = np.random.rand(len(dataset)) < 0.8\n\ndf_train = dataset[train_mask]\ndf_test = dataset[~train_mask]\n\nvalid_mask = np.random.rand(len(df_train)) < 0.1\ndf_val = df_train[valid_mask]\ndf_train = df_train[~valid_mask]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**TF.Data conversion for speed**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def proc_image(path, bin_nb, offset):\n    image = tf.io.read_file(path)\n    oh_bin = tf.one_hot(bin_nb, NUM_BINS)\n    #offset /= 2 #Normalize\n    \n    #image = tf.image.decode_jpeg(image, channels=3)\n    image = tf.image.decode_jpeg(image, channels=1)\n    image = tf.image.convert_image_dtype(image, tf.float32)\n    # resize the image to the desired size.\n    image = tf.image.resize(image, [IMAGE_INP_SIZE, IMAGE_INP_SIZE])\n    \n    return image, (oh_bin, offset)\n\ndataset = tf.data.Dataset.from_tensor_slices((df_train['Name'].values, df_train['Bin_nb'].values, df_train['Bin_offset_norm'].values))\ndataset = dataset.map(proc_image, num_parallel_calls=tf.data.experimental.AUTOTUNE)\ndataset = dataset.batch(8).repeat().prefetch(buffer_size=tf.data.experimental.AUTOTUNE)\n\n\ndataset_valid = tf.data.Dataset.from_tensor_slices((df_val['Name'].values, df_val['Bin_nb'].values, df_val['Bin_offset_norm'].values))\ndataset_valid = dataset_valid.map(proc_image, num_parallel_calls=tf.data.experimental.AUTOTUNE)\ndataset_valid = dataset_valid.batch(8).repeat().prefetch(buffer_size=tf.data.experimental.AUTOTUNE)\n\nsteps_per_epoch_train = len(df_train) // 8\nvalidation_steps = len(df_val) // 8","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_test = tf.data.Dataset.from_tensor_slices((df_test['Name'].values, df_test['Bin_nb'].values, df_test['Bin_offset_norm'].values))\ndata_test = data_test.map(proc_image, num_parallel_calls=tf.data.experimental.AUTOTUNE)\ndata_test = data_test.batch(8).repeat().prefetch(buffer_size=tf.data.experimental.AUTOTUNE)\n\nsteps_per_epoch_test = len(df_test) // 8","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tf.keras.backend.clear_session()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I will use a simple ResNet approach. The speed must be a requirement for a Autonomous driving application."},{"metadata":{"trusted":true},"cell_type":"code","source":"def res_block(previous):\n    conv1 = layers.Conv2D(256, (1,1), activation='relu', kernel_regularizer='l2', padding='same')(previous)\n    conv2 = layers.Conv2D(128, (3,3), activation='relu', kernel_regularizer='l2', padding='same')(conv1)\n    conv3 = layers.Conv2D(128, (1,1), activation='relu', kernel_regularizer='l2', padding='same')(conv2)\n    concat = layers.Concatenate()([previous, conv3])\n    concat = layers.BatchNormalization()(concat)\n    return concat\n\ninp = layers.Input(shape=(IMAGE_INP_SIZE,IMAGE_INP_SIZE,1))\nconv1 = layers.Conv2D(128, (4,4), activation='relu', strides=2)(inp)\npool1 = layers.MaxPool2D()(conv1)\nblock1 = res_block(pool1)\npool2 = layers.MaxPool2D((2,2))(block1)\nblock2 = res_block(pool2)\npool3 = layers.MaxPool2D((2,2))(block2)\n#block3 = res_block(pool3)\n\nflat = layers.Flatten()(pool3)\nmodel_output = layers.Dense(500, activation='linear')(flat)\n\nout_offset = layers.Dense(500, activation='relu')(model_output)\nout_offset = layers.Dropout(.2)(out_offset)\nout_offset = layers.Dense(300, activation='linear')(out_offset)\nout_offset = layers.Dense(1, activation='sigmoid', name='out_offset')(out_offset)\n\nout_bin = layers.Dense(500, activation='relu')(model_output)\nout_bin = layers.Dropout(.2)(out_bin)\nout_bin = layers.Dense(200, activation='linear')(out_bin)\nout_bin = layers.Dense(NUM_BINS, activation='softmax', name='out_bin')(out_bin)\n\n\nmodel = models.Model(inputs=[inp], outputs=[out_bin, out_offset])\n\ncallbacks = [\n    tf.keras.callbacks.TerminateOnNaN(),\n    tf.keras.callbacks.EarlyStopping(monitor='val_out_bin_acc', patience=2, min_delta=0.001, mode='max', restore_best_weights=True)\n]\n\nmetrics = {\n    'out_bin': ['acc', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()]\n}\n\nmodel.compile(optimizers.Adam(lr=1e-4), loss=['categorical_crossentropy', 'mse'], loss_weights=[0.5, 0.5], \n              metrics=metrics)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hist = model.fit(dataset,\n                 epochs = 20,\n                 callbacks = callbacks,\n                 steps_per_epoch=steps_per_epoch_train, \n                 validation_data=dataset_valid, \n                 validation_steps=validation_steps)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2, figsize=(8,5))\nax[0].plot(hist.history['val_out_bin_acc'], label='val_acc')\nax[0].plot(hist.history['out_bin_acc'], label='train_acc')\nax[0].grid(True)\nax[0].legend()\n\nax[1].plot(hist.history['val_out_offset_loss'], label='val_offset')\nax[1].plot(hist.history['out_offset_loss'], label='train_offset')\nax[1].grid(True)\nax[1].legend()\n\nfig.suptitle(\"Train history\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Testing**"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_results = model.predict(data_test, steps=steps_per_epoch_test, verbose=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Denormalize offset \ntest_results[1] *= max_off","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Predict example"},{"metadata":{"trusted":true},"cell_type":"code","source":"index = 29\nimage = Image.open(df_test.iloc[index]['Name'])\np_off = test_results[1][index]\np_bin = test_results[0][index]\n\nreal_off = df_test.iloc[index]['Bin_offset']\nreal_bin = df_test.iloc[index]['Bin_nb']\n\n\nprint(f\"Real offset: {real_off}\")\nprint(f\"Real bin: {real_bin}\")\nprint(f\"REAL ANGLE {math.degrees(df_test.iloc[index]['Global_angle'])}\")\n\nprint(\"-\"*30)\nbin_nb = np.argmax(p_bin)\noffset = p_off\nray_angle = df_test.iloc[index]['Ray_Angle']\n\nprint(f\"Predicted bin: {bin_nb}\")\nprint(f\"Predicted offset: {offset}\")\nprint(f\"PREDICTED ANGLE {math.degrees(prediction_to_yaw(bin_nb, offset, ray_angle)[0])}\")\n\ndisplay(image)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Evaluate the model with the global (ground truth) angles"},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_angles = []\nreal_angles = []\n\nreal_bins = []\npredicted_bins = []\n\nfor i in range(len(test_results[1])):\n    p_off = test_results[1][i]\n    p_bin = test_results[0][i]\n    ray_angle = df_test.iloc[i]['Ray_Angle']\n    \n    bin_nb = np.argmax(p_bin)\n    offset = p_off\n    \n    ang = math.degrees(prediction_to_yaw(bin_nb, offset, ray_angle)[0])\n    \n    pred_angles.append(ang)\n    real_angles.append(math.degrees(df_test.iloc[i]['Global_angle']))\n    \n    real_bins.append(df_test.iloc[i]['Bin_nb'])\n    predicted_bins.append(bin_nb)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Confussion matrix of the bins"},{"metadata":{"trusted":true},"cell_type":"code","source":"cm = confusion_matrix(real_bins, predicted_bins)\ndf_cm = pd.DataFrame(cm)\ndf_cm.index.name = 'Actual'\ndf_cm.columns.name = 'Predicted'\n\nfig, ax = plt.subplots(figsize=(8, 8))\nax.set_title('Confusion Matrix')\nsns.heatmap(cm, cmap=\"Blues\", annot=True, ax=ax, annot_kws={\"size\": 9})\nax.set_ylabel('Actual bin')\nax.set_xlabel('Predicted bin');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1)\nax.scatter(real_angles, pred_angles, alpha=0.5, s=2);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above plots show that the model is biased because of the data. This is normal as most of the cars are placed on two directions (frontwards/backwards).\n\nCalculate the error distribution."},{"metadata":{"trusted":true},"cell_type":"code","source":"errors = [angle_distance(a1, a2) for a1,a2 in zip(pred_angles, real_angles)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, figsize=(11,8))\nsns.distplot(errors, ax=ax)\nax.set_title(\"Error distribution\")\nax.set_xlabel(\"Degrees\")\n\nprint(f\"Avg. error: {np.mean(np.abs(errors))}\")\nprint(f\"Median error: {np.median(errors)}\")\nprint(f\"Max. error: {np.max(np.abs(errors))}\")\nprint(f\"SD error: {np.std(errors)}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Save model if needed\nmodel.save('yaw.h5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Weaknesses"},{"metadata":{},"cell_type":"markdown","source":"* This approach might not work well with car occlusion\n* The original training data containes noise. Parallel cars often have different yaws or their associated yaws are wrong."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"}},"nbformat":4,"nbformat_minor":1}