{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":15768,"databundleVersionId":700263,"sourceType":"competition"}],"dockerImageVersionId":31153,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Project: Lyft Competition\n\nThis project explores annotated object properties (location and dimensions) in the Lyft Level 5 dataset for autonomous driving, focusing on their distributions by class (car, truck, bus, bicycle, pedestrian, etc.).\n\n## Overview\n\nThe dataset contains 3D bounding box annotations for multiple object types detected by autonomous vehicle sensors. Through visualizations including KDE joint plots, boxplots, and violin plots, we analyze spatial positions (center_x, center_y, center_z) and object dimensions (width, length, height) for each class. Heavy vehicles (truck, bus, other_vehicle) show greater spread and larger typical dimensions than cars, while pedestrians and bicycles are distinctively smaller and occupy tighter spatial regions. These patterns reflect real-world road structure and validate annotation quality.\n\n## Key Research Questions\n\n* How do object spatial positions (center_x, center_y, center_z) vary by class in autonomous driving scenes?\n* What are the typical and outlier dimensions (width, length, height) for each annotated class?\n* Are data distributions for each class consistent with real-world expectations?\n* Can these statistics inform model design, class separability, and data quality assurance?\n* How can multimodality and outliers in these features be used for robust detection and scene understanding?\n\n## Credits and Citations\n\n### 1. Theory\n\n* https://github.com/lyft/nuscenes-devkit\n* https://en.wikipedia.org/wiki/Lidar\n* https://www.youtube.com/watch?v=tlThdr3O5Qo&t=4s\n\n### 2. Kaggle Notebooks\n\n* https://www.kaggle.com/code/tarunpaparaju/lyft-competition-understanding-the-data\n* https://www.kaggle.com/gaborfodor/eda-3d-object-detection-challenge\n* https://www.kaggle.com/xhlulu/lyft-eda-animations-generating-csvs\n* https://www.kaggle.com/jesucristo/starter-devkit-lyft3d","metadata":{}},{"cell_type":"markdown","source":"# 1. Introduction: TBD\n\nThe Lyft Level 5 dataset provides richly annotated 3D bounding boxes for a variety of object classes in urban driving scenes. Initial analysis shows significant and interpretable structure in the spatial location and physical properties of these objects, reflecting real-world lane usage, road organization, and expected dimensions. These insights motivate further research in autonomous vehicle perception and benchmarking for robust object detection models.","metadata":{}},{"cell_type":"markdown","source":"## 1.1 Importing Libraries","metadata":{}},{"cell_type":"code","source":"# Installations\n# Uncomment if running on Kaggle or Colab to install required packages\n!pip install lyft-dataset-sdk --quiet","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:08.257679Z","iopub.execute_input":"2025-11-22T15:18:08.259076Z","iopub.status.idle":"2025-11-22T15:18:14.884964Z","shell.execute_reply.started":"2025-11-22T15:18:08.259013Z","shell.execute_reply":"2025-11-22T15:18:14.883973Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# The upgrades below are because of a scipy error. This works so do not touch. Please.\n!pip install --upgrade numpy==1.26.4 --quiet\n!pip install --upgrade scipy==1.14.1 --quiet","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:14.887289Z","iopub.execute_input":"2025-11-22T15:18:14.887541Z","iopub.status.idle":"2025-11-22T15:18:24.458308Z","shell.execute_reply.started":"2025-11-22T15:18:14.887521Z","shell.execute_reply":"2025-11-22T15:18:24.456739Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# System & Utilities\nimport os\nimport glob\nimport gc\nimport numpy as np\nimport pandas as pd\nfrom pandas import json_normalize\nimport sys\nfrom pathlib import Path\n\n# Visualization\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as mpatches\nfrom matplotlib.axes import Axes\nfrom matplotlib import animation, rc\nimport plotly.graph_objs as go\nimport plotly.tools as tls\nfrom plotly.offline import plot, init_notebook_mode\nimport plotly.figure_factory as ff\ninit_notebook_mode(connected=True)\nimport seaborn as sns\nfrom PIL import Image\nimport cv2\nfrom IPython.display import HTML\n\n# Lfyt Dataset SDK\n# The Lyft Level 5 Dataset was released publicly for research on autonomous driving, similar to the nuScenes dataset by Motional. \n# In fact, lyft_dataset_sdk is built upon a modified version of the nuScenes devkit, which means its APIs and data structures are nearly identical.\nimport lyft_dataset_sdk\nfrom lyft_dataset_sdk.utils.map_mask import MapMask\nfrom lyft_dataset_sdk.lyftdataset import LyftDataset\nfrom lyft_dataset_sdk.utils.geometry_utils import view_points, box_in_image, BoxVisibility\nfrom lyft_dataset_sdk.utils.geometry_utils import view_points, transform_matrix\n\n# sklearn\nimport sklearn.metrics\nfrom sklearn.metrics import classification_report, confusion_matrix\n\n# Misc\nfrom collections import Counter\nimport warnings\nwarnings.filterwarnings('ignore')\nimport random\nimport json\nimport pickle\nfrom datetime import datetime\nimport math\nimport time\nfrom typing import Tuple, List, Dict\nfrom pyquaternion import Quaternion\nfrom tqdm import tqdm # popular Python library designed to provide fast, extensible progress bars for loops and iterable tasks. \nimport struct\nfrom abc import ABC, abstractmethod\nfrom functools import reduce\nimport copy","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:24.459836Z","iopub.execute_input":"2025-11-22T15:18:24.460092Z","iopub.status.idle":"2025-11-22T15:18:24.475536Z","shell.execute_reply.started":"2025-11-22T15:18:24.460071Z","shell.execute_reply":"2025-11-22T15:18:24.473837Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1.2 Version Checks","metadata":{}},{"cell_type":"code","source":"?lyft_dataset_sdk","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:24.477139Z","iopub.execute_input":"2025-11-22T15:18:24.477559Z","iopub.status.idle":"2025-11-22T15:18:24.500098Z","shell.execute_reply.started":"2025-11-22T15:18:24.477526Z","shell.execute_reply":"2025-11-22T15:18:24.499252Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"As per the official documentation on Lyft Dataset SDK 0.0.08, this is the latest version which was updated in 2019. However, as it is based on the Nuscenes SDK, and there are many proven notebooks on Kaggle that can help a newbie learn \"Object Detection\", this notebook will continue to use this module for academic and research purpose.","metadata":{}},{"cell_type":"markdown","source":"## 1.3 User Variables","metadata":{}},{"cell_type":"code","source":"os.listdir('/kaggle/input') # The above code goes into a big loop as there are too many files in the subdirs","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:24.502788Z","iopub.execute_input":"2025-11-22T15:18:24.502969Z","iopub.status.idle":"2025-11-22T15:18:24.518705Z","shell.execute_reply.started":"2025-11-22T15:18:24.502955Z","shell.execute_reply":"2025-11-22T15:18:24.517689Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"common_path = \"/kaggle/input/3d-object-detection-for-autonomous-vehicles/\"\ntrain_datasets_path = common_path + 'train.csv'\ntrain_json_path = common_path + 'train_data'\nsample_submission_path = common_path + 'sample_submission.csv'","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:24.519716Z","iopub.execute_input":"2025-11-22T15:18:24.519977Z","iopub.status.idle":"2025-11-22T15:18:24.535611Z","shell.execute_reply.started":"2025-11-22T15:18:24.519957Z","shell.execute_reply":"2025-11-22T15:18:24.534702Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"output_dir = '/kaggle/working/outputs'\nos.makedirs(output_dir, exist_ok=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:24.536579Z","iopub.execute_input":"2025-11-22T15:18:24.536832Z","iopub.status.idle":"2025-11-22T15:18:24.557533Z","shell.execute_reply.started":"2025-11-22T15:18:24.536815Z","shell.execute_reply":"2025-11-22T15:18:24.556376Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 2. EDA\n\nThe scope of this notebook is to first learn the basics. The organization and decoration of the codes will be done in the future :)","metadata":{}},{"cell_type":"markdown","source":"## 2.1 Exploring training data","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv(train_datasets_path)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:24.558575Z","iopub.execute_input":"2025-11-22T15:18:24.558851Z","iopub.status.idle":"2025-11-22T15:18:25.100806Z","shell.execute_reply.started":"2025-11-22T15:18:24.558832Z","shell.execute_reply":"2025-11-22T15:18:25.099107Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(f\"The columns of the training data are: {df_train.columns}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:25.102347Z","iopub.execute_input":"2025-11-22T15:18:25.102609Z","iopub.status.idle":"2025-11-22T15:18:25.107543Z","shell.execute_reply.started":"2025-11-22T15:18:25.102593Z","shell.execute_reply":"2025-11-22T15:18:25.106431Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_train.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:25.108660Z","iopub.execute_input":"2025-11-22T15:18:25.108862Z","iopub.status.idle":"2025-11-22T15:18:25.133108Z","shell.execute_reply.started":"2025-11-22T15:18:25.108846Z","shell.execute_reply":"2025-11-22T15:18:25.132158Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Exploring the columns of the df_train\n\nprint(f\"ID Column sample: {df_train['Id'][0]}\")\nprint(\"*\"*100)\nprint(f\"Prediction Column sample: {df_train['PredictionString'][0][:500]}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:25.134177Z","iopub.execute_input":"2025-11-22T15:18:25.134430Z","iopub.status.idle":"2025-11-22T15:18:25.154966Z","shell.execute_reply.started":"2025-11-22T15:18:25.134414Z","shell.execute_reply":"2025-11-22T15:18:25.153340Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The name of the vehicle repeats after every `8`th string in the `Prediction String` column ","metadata":{}},{"cell_type":"code","source":"print(f\"First 8 split: {df_train['PredictionString'][0].split()[:8]}\")\nprint(f\"Second split: {df_train['PredictionString'][0].split()[8:16]}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:25.156001Z","iopub.execute_input":"2025-11-22T15:18:25.156252Z","iopub.status.idle":"2025-11-22T15:18:25.172186Z","shell.execute_reply.started":"2025-11-22T15:18:25.156232Z","shell.execute_reply":"2025-11-22T15:18:25.171202Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now, we need to retrieve these annotated objects where the order of the objects are as follows:\n\n`center_x` -> `center_y` -> `center_z` -> `width` -> `length` -> `height` -> `yaw` -> `class_name` ","metadata":{}},{"cell_type":"code","source":"def retrieve_annotated_objects(df_train, df_train_index, number_of_annotated_objects, annotation_start):\n if 0 <= df_train_index < len(df_train):\n try:\n prediction_string_list = df_train.loc[df_train_index, \"PredictionString\"].split()\n _start = annotation_start\n _end = annotation_start + number_of_annotated_objects\n\n # Ensure the indices are within bounds\n if _end <= len(prediction_string_list):\n center_x, center_y, center_z, width, length, height, yaw, class_name = prediction_string_list[_start:_end]\n return center_x, center_y, center_z, width, length, height, yaw, class_name\n else:\n return \"Error: Annotation indices exceed PredictionString length.\"\n except Exception as e:\n return f\"Error processing annotation: {e}\"\n else:\n return \"Error: Invalid df_train_index.\"","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:25.173560Z","iopub.execute_input":"2025-11-22T15:18:25.173902Z","iopub.status.idle":"2025-11-22T15:18:25.192799Z","shell.execute_reply.started":"2025-11-22T15:18:25.173881Z","shell.execute_reply":"2025-11-22T15:18:25.191803Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"center_x, center_y, center_z, width, length, height, yaw, class_name = retrieve_annotated_objects(df_train, 0, 8, 0)\n\nprint(f\"center_x: {center_x}\")\nprint(f\"center_y: {center_y}\")\nprint(f\"center_z: {center_z}\")\nprint(f\"width: {width}\")\nprint(f\"length: {length}\")\nprint(f\"height: {height}\")\nprint(f\"yaw: {yaw}\")\nprint(f\"class_name: {class_name}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:25.196713Z","iopub.execute_input":"2025-11-22T15:18:25.197706Z","iopub.status.idle":"2025-11-22T15:18:25.213448Z","shell.execute_reply.started":"2025-11-22T15:18:25.197664Z","shell.execute_reply":"2025-11-22T15:18:25.212719Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_train[\"PredictionString\"].shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:25.214057Z","iopub.execute_input":"2025-11-22T15:18:25.214293Z","iopub.status.idle":"2025-11-22T15:18:25.236687Z","shell.execute_reply.started":"2025-11-22T15:18:25.214277Z","shell.execute_reply":"2025-11-22T15:18:25.235703Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"number_of_annotated_objects = 8 # center_x, center_y, center_z, width, length, height, yaw, class_name\n\ndef count_objects(pred_string):\n if pd.isna(pred_string):\n return 0\n return len(pred_string.strip().split()) // number_of_annotated_objects\n\nnp.sum(df_train['PredictionString'].apply(count_objects))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:25.237593Z","iopub.execute_input":"2025-11-22T15:18:25.237818Z","iopub.status.idle":"2025-11-22T15:18:25.442795Z","shell.execute_reply.started":"2025-11-22T15:18:25.237804Z","shell.execute_reply":"2025-11-22T15:18:25.442017Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"There are `22680` rows worth of `638179` annotated objects in the `PredictionString` column","metadata":{}},{"cell_type":"markdown","source":"Now, we try to convert the `df_train` into a `DataFrame` so that we can retrieve actionable insights easily using `Pandas` (future, if required)","metadata":{}},{"cell_type":"code","source":"def convert_df_actionable_insights(df_train):\n\n import pandas as pd\n import numpy as np\n from tqdm import tqdm\n \n numerical_cols = ['object_id', 'center_x', 'center_y', 'center_z', 'width', 'length', 'height', 'yaw']\n df_train_object_columns = numerical_cols + ['class_name']\n object_columns = [\"sample_id\"] + df_train_object_columns\n \n objects = []\n \n for sample_id, row in tqdm(df_train.values[:]):\n row_params = row.split()\n \n num_objects = len(row_params)\n number_of_annotated_objects = 8 # 'center_x', 'center_y', 'center_z', 'width', 'length', 'height', 'yaw', 'class_name'\n row_range = num_objects // number_of_annotated_objects\n \n for i in range(row_range):\n _start = i*8\n _end = (i+1)*8\n center_x, center_y, center_z, width, length, height, yaw, class_name = tuple(row_params[_start: _end])\n \n objects.append([sample_id, i, center_x, center_y, center_z, width, length, height, yaw, class_name])\n\n df_train_objects = pd.DataFrame(objects, columns = object_columns)\n\n # Convert string type to float32 for numerical columns\n df_train_objects[numerical_cols] = np.float32(df_train_objects[numerical_cols].values)\n\n return df_train_objects","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:25.444007Z","iopub.execute_input":"2025-11-22T15:18:25.444250Z","iopub.status.idle":"2025-11-22T15:18:25.450353Z","shell.execute_reply.started":"2025-11-22T15:18:25.444231Z","shell.execute_reply":"2025-11-22T15:18:25.449505Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_train_objects = convert_df_actionable_insights(df_train)\ndf_train_objects.head(2)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:25.451252Z","iopub.execute_input":"2025-11-22T15:18:25.451465Z","iopub.status.idle":"2025-11-22T15:18:32.009110Z","shell.execute_reply.started":"2025-11-22T15:18:25.451447Z","shell.execute_reply":"2025-11-22T15:18:32.007713Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We now have the training data with the appropriate columns and splits","metadata":{}},{"cell_type":"markdown","source":"### Notes\n\n* The annotation objects in the Lyft dataset's \"PredictionString\" are the 3D bounding boxes that represent various objects detected in the environment around the autonomous vehicle. Each annotated object describes a traffic agent or dynamic entity in the scene defined by the following fields in `train.csv`:\n\n * `center_x`, `center_y` and `center_z` are the world coordinates of the center of the 3D bounding volume.\n * `width`, `length` and `height` are the dimensions of the volume.\n * `yaw` is the angle of the volume around the z axis (where y is forward/back, x is left/right, and z is up/down - making 'yaw' the direction the front of the vehicle / bounding box is pointing at while on the ground).\n * `class_name` is the type of object contained by the bounding volume.\n\n* We have 638179 annotated objects in 22680 train samples.","metadata":{}},{"cell_type":"markdown","source":"## 2.2 Distributions between the annotated objects","metadata":{}},{"cell_type":"markdown","source":"### 2.2.1 Distributions of `center_x` and `center_y`","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(5, 5))\n\n# \"center_x\" distribution plot\nsns.distplot(\n df_train_objects['center_x'], \n color='darkorange', \n ax=ax,\n label=\"center_x\"\n).set_title('center_x and center_y', fontsize=10)\n\n# \"center_y\" distribution plot\nsns.distplot(\n df_train_objects['center_y'], \n color='purple', \n ax=ax,\n label=\"center_y\"\n).set_title('center_x and center_y', fontsize=10)\n\nplt.xlabel('center_x and center_y', fontsize=10)\nplt.legend()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:32.010079Z","iopub.execute_input":"2025-11-22T15:18:32.010358Z","iopub.status.idle":"2025-11-22T15:18:36.017230Z","shell.execute_reply.started":"2025-11-22T15:18:32.010343Z","shell.execute_reply":"2025-11-22T15:18:36.016112Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Notes\n\n* Multiple Peaks:\n * Both the distributions have multiple peaks or multimodal distribution.\n * This implies that objects' centers often cluster at certain locations in the vehicle/environment coordinate space, reflecting structured scenes such as lanes, roads, and common object/vehicle trajectories.\n\n* Overlap and Spread:\n * The plots for ``center_x`` and ``center_y`` have overlapping regions but with distinct peak placements and spreads.\n * This suggests regular patterns in where objects are found, but the density and location clusters differ along the ``x`` and ``y`` spatial axes.\n\n* Density Magnitude:\n * The higher the density at a particular location (y-axis), the more often objects appear at that spatial value in the dataset.\n * Some positions are far more common, likely correlating with traffic lanes or common driveable space.\n\n* Right Skew:\n * The main spread is between roughly ``500`` and ``2500`` for both axes, indicating most annotated objects fall within this spatial corridor.\n * The differences in density shapes can reveal the geometry of the road network, sensor perspective, or dataset collection bias.","metadata":{}},{"cell_type":"markdown","source":"### 2.2.2 KDE Plot: Relationship between `center_x` and `center_y`\n\nA KDE plot (Kernel Density Estimate plot) is a method for visualizing the probability density function of a continuous variable, serving as a smoothed alternative to histograms. Instead of showing raw counts in bins, it estimates the probability density by averaging over 'kernels' (bell-shaped curves) centered at each data point, creating a continuous profile that reveals patterns and modes in your data.","metadata":{}},{"cell_type":"code","source":"df_new_train_objects = df_train_objects.query('class_name == \"car\"')\n\nplot = sns.jointplot(\n x=df_new_train_objects['center_x'][:1000],\n y=df_new_train_objects['center_y'][:1000], \n kind='kde',\n color='blueviolet',\n fill=True,\n height=5\n)\n\nplot.set_axis_labels('center_x', 'center_y', fontsize=10)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:36.018371Z","iopub.execute_input":"2025-11-22T15:18:36.018892Z","iopub.status.idle":"2025-11-22T15:18:37.131380Z","shell.execute_reply.started":"2025-11-22T15:18:36.018875Z","shell.execute_reply":"2025-11-22T15:18:37.130507Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Check the correlation\ndf_new_train_objects['center_x'][:1000].corr(df_new_train_objects['center_y'][:1000])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:37.132427Z","iopub.execute_input":"2025-11-22T15:18:37.132762Z","iopub.status.idle":"2025-11-22T15:18:37.140502Z","shell.execute_reply.started":"2025-11-22T15:18:37.132745Z","shell.execute_reply":"2025-11-22T15:18:37.139528Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Notes\n\n* Object Detection distance limitations\n * Camera cannot detect objects that are both far ahead and far to the side.\n * Objects in these both extremities may not get detected at all.\n * Objects in one of these extremities may get detected.\n * Negative Correlation between `center_x` and `center_y`","metadata":{}},{"cell_type":"markdown","source":"### 2.2.3 Distribution of `center-z`","metadata":{}},{"cell_type":"code","source":"def plot_single_distplot(df, _x, _color):\n fig, ax = plt.subplots(figsize=(5, 5))\n\n sns.distplot(\n df[_x],\n color=_color,\n ax=ax).set_title(_x, fontsize=10)\n \n plt.xlabel(_x, fontsize=10)\n plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:37.141516Z","iopub.execute_input":"2025-11-22T15:18:37.141771Z","iopub.status.idle":"2025-11-22T15:18:37.159412Z","shell.execute_reply.started":"2025-11-22T15:18:37.141750Z","shell.execute_reply":"2025-11-22T15:18:37.158608Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plot_single_distplot(df_train_objects, \"center_z\", \"navy\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:37.160041Z","iopub.execute_input":"2025-11-22T15:18:37.160245Z","iopub.status.idle":"2025-11-22T15:18:39.626070Z","shell.execute_reply.started":"2025-11-22T15:18:37.160232Z","shell.execute_reply":"2025-11-22T15:18:39.624519Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Notes\n\n* Right skew\n * Right positive skew\n * Clustered around `-20` mark -> mean value\n\n* Variation is smaller than that of join plot of `center_x` and `center_y`\n * Most objects are very close to the flat plan of the road\n * Results in no great variation in the height of the objects above/below (z-axis) of the camera.\n\n* Negative z-coordinates\n * Camera is attached on top of the car\n * Camera has to \"look down\" to see objects","metadata":{}},{"cell_type":"markdown","source":"### 2.2.4 Distribution of `yaw`","metadata":{}},{"cell_type":"code","source":"plot_single_distplot(df_train_objects, \"yaw\", \"darkgreen\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:39.627269Z","iopub.execute_input":"2025-11-22T15:18:39.627546Z","iopub.status.idle":"2025-11-22T15:18:41.486070Z","shell.execute_reply.started":"2025-11-22T15:18:39.627526Z","shell.execute_reply":"2025-11-22T15:18:41.485074Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Notes\n\n* Bimodal distribution\n * Two major peaks in the distribution: 0.5 and then 2.5\n * Mean is between 1 and 2\n\n* No clear skew\n * The bimodal peaks reduces the skew in both directions, cancelling each other out\n * Balanced distribution","metadata":{}},{"cell_type":"markdown","source":"### 2.2.5 Distribution of `width`","metadata":{}},{"cell_type":"code","source":"plot_single_distplot(df_train_objects, \"width\", \"magenta\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:41.487054Z","iopub.execute_input":"2025-11-22T15:18:41.487319Z","iopub.status.idle":"2025-11-22T15:18:43.885682Z","shell.execute_reply.started":"2025-11-22T15:18:41.487300Z","shell.execute_reply":"2025-11-22T15:18:43.884657Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Notes\n\n* Unimodal and Normal distributed\n * Mean is around 2\n * Majority of the objects are cars, around the mean\n * Outliers on both sides of the peak\n * Left Side: smaller objects: pedestrians, bicycles\n * Right Side: larger objects: trucks, vans","metadata":{}},{"cell_type":"markdown","source":"### 2.2.6 Distribution of `length`","metadata":{}},{"cell_type":"code","source":"plot_single_distplot(df_train_objects, \"length\", \"crimson\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:43.886744Z","iopub.execute_input":"2025-11-22T15:18:43.886974Z","iopub.status.idle":"2025-11-22T15:18:46.337217Z","shell.execute_reply.started":"2025-11-22T15:18:43.886958Z","shell.execute_reply":"2025-11-22T15:18:46.336361Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Notes\n\n* Unimodal with Right skew\n * Mean is around 5\n * Majority of the objects are cars, around the mean\n\n* Outliers on both sides of the peak\n * Left Side: smaller objects: pedestrians, bicycles\n * Right Side: larger objects: trucks, vans","metadata":{}},{"cell_type":"markdown","source":"### 2.2.7 Distribution of `height`","metadata":{}},{"cell_type":"code","source":"plot_single_distplot(df_train_objects, \"height\", \"indigo\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:46.338239Z","iopub.execute_input":"2025-11-22T15:18:46.338541Z","iopub.status.idle":"2025-11-22T15:18:48.918533Z","shell.execute_reply.started":"2025-11-22T15:18:46.338519Z","shell.execute_reply":"2025-11-22T15:18:48.917313Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Notes\n\n* Unimodal with Right skew\n * Mean is around 2\n * Majority of the objects are cars, around the mean\n\n* Outliers on both sides of the peak\n * Left Side: smaller objects: pedestrians, bicycles\n * Right Side: larger objects: trucks, vans","metadata":{}},{"cell_type":"markdown","source":"### 2.2.8 Frequency of object classes","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(5, 5))\n\n# Filter the dataframe\ndf_train_objects_filtered = df_train_objects.query(\n 'class_name != \"motorcycle\" and class_name != \"emergency_vehicle\" and class_name != \"animal\"'\n)\n\n# Get class counts and sort in descending order\nclass_counts = df_train_objects_filtered['class_name'].value_counts()\nsorted_classes = class_counts.index.tolist()\n\nplot = sns.countplot(\n y=\"class_name\",\n data=df_train_objects_filtered,\n palette=['navy', 'darkblue', 'blue', 'dodgerblue', 'skyblue', 'lightblue'],\n order=sorted_classes).set_title('Object Frequencies', fontsize=10)\n\nplt.yticks(fontsize=10)\nplt.xlabel(\"Count\", fontsize=10)\nplt.ylabel(\"Class Name\", fontsize=10)\nplt.show(plot)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:48.919478Z","iopub.execute_input":"2025-11-22T15:18:48.919748Z","iopub.status.idle":"2025-11-22T15:18:49.746358Z","shell.execute_reply.started":"2025-11-22T15:18:48.919730Z","shell.execute_reply":"2025-11-22T15:18:49.745370Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Notes\n\n* The most common object is `car`\n * Background of the data: Images were taken from the streets of Palo Alto, Silicon Valley, CA, USA\n * Car centric roads","metadata":{}},{"cell_type":"markdown","source":"### 2.2.9 `center_x` vs `class_name`","metadata":{}},{"cell_type":"code","source":"def comparison_plots(df, _x, _y, _query, _palette, _title, type_of_plot=\"violinplot\"):\n\n fig, ax = plt.subplots(figsize=(8, 8))\n\n if type_of_plot == \"violinplot\":\n plot = sns.violinplot(\n x=_x,\n y=_y,\n data=df.query(_query),\n palette=_palette,\n split=True,\n ax=ax).set_title(_title, fontsize=10)\n elif type_of_plot == \"boxplot\":\n plot = sns.boxplot(\n x=_x,\n y=_y,\n data=df.query(_query),\n palette=_palette,\n ax=ax).set_title(_title, fontsize=10)\n else:\n print(\"Choose: violinplot or boxplot!\")\n \n plt.yticks(fontsize=10)\n plt.xticks(fontsize=10)\n plt.xlabel(_x, fontsize=10)\n plt.ylabel(_y, fontsize=10)\n plt.show(plot)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:49.747307Z","iopub.execute_input":"2025-11-22T15:18:49.747590Z","iopub.status.idle":"2025-11-22T15:18:49.754575Z","shell.execute_reply.started":"2025-11-22T15:18:49.747567Z","shell.execute_reply":"2025-11-22T15:18:49.753411Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"comparison_plots(\n df_train_objects,\n _x=\"class_name\",\n _y=\"center_x\", \n _query='class_name != \"motorcycle\" and class_name != \"emergency_vehicle\" and class_name != \"animal\"', \n _palette='YlGnBu',\n _title='center_x (for different objects)', \n type_of_plot=\"violinplot\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:49.755373Z","iopub.execute_input":"2025-11-22T15:18:49.755610Z","iopub.status.idle":"2025-11-22T15:18:51.616254Z","shell.execute_reply.started":"2025-11-22T15:18:49.755591Z","shell.execute_reply":"2025-11-22T15:18:51.614670Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"comparison_plots(\n df_train_objects,\n _x=\"class_name\",\n _y=\"center_x\",\n _query='class_name != \"motorcycle\" and class_name != \"emergency_vehicle\" and class_name != \"animal\"', \n _palette='YlGnBu',\n _title='center_x (for different objects)', \n type_of_plot=\"boxplot\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:51.617497Z","iopub.execute_input":"2025-11-22T15:18:51.617885Z","iopub.status.idle":"2025-11-22T15:18:52.486148Z","shell.execute_reply.started":"2025-11-22T15:18:51.617858Z","shell.execute_reply":"2025-11-22T15:18:52.485279Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Notes\n\n* Variation Across Classes:\n * All object classes show a spread of center_x values, but the central tendency and spread differ.\n * For example, cars, trucks, buses, and other vehicles cover a wide range, while pedestrians and bicycles are concentrated at lower center_x values.\n\n* Class-Specific Locations:\n * Pedestrians and bicycles tend to cluster around lower center_x regions, visible as both denser violin sections and lower boxes.\n * This suggests these classes are often found closer to one side of the spatial axis (perhaps road edges or crosswalks).\n\n* Central Tendency:\n * The median (white dot in violins, dark bar in boxes) shows cars/trucks/buses generally have higher medians than pedestrians/bicycles, highlighting road placement differences.\n\n* Spread and Outliers:\n * Cars, trucks, and buses feature a large spread, visible in both the wide box range and broad violin density.\n * Pedestrians and bicycles exhibit more visible outliers and a tighter interquartile range.\n\n* Skewness and Multimodality:\n * The violin plot reveals multimodal distributions for some classes (especially cars and other vehicles), indicating multiple preferred positions on the axis—implying the presence of lanes or frequent stopping locations.\n\n* Density vs. Frequency:\n * Both plots reflect the frequency and density of objects at each position.\n * Where the violin widens or the box has many dots, more objects are present at that location.","metadata":{}},{"cell_type":"markdown","source":"### 2.2.10 `center_y` vs `class_name`","metadata":{}},{"cell_type":"code","source":"comparison_plots(\n df_train_objects,\n _x=\"class_name\",\n _y=\"center_y\", \n _query='class_name != \"motorcycle\" and class_name != \"emergency_vehicle\" and class_name != \"animal\"', \n _palette='YlOrRd',\n _title='center_y (for different objects)', \n type_of_plot=\"violinplot\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:52.487676Z","iopub.execute_input":"2025-11-22T15:18:52.487965Z","iopub.status.idle":"2025-11-22T15:18:54.465169Z","shell.execute_reply.started":"2025-11-22T15:18:52.487950Z","shell.execute_reply":"2025-11-22T15:18:54.464446Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"comparison_plots(\n df_train_objects,\n _x=\"class_name\",\n _y=\"center_y\", \n _query='class_name != \"motorcycle\" and class_name != \"emergency_vehicle\" and class_name != \"animal\"', \n _palette='YlOrRd',\n _title='center_y (for different objects)', \n type_of_plot=\"boxplot\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:54.465981Z","iopub.execute_input":"2025-11-22T15:18:54.466218Z","iopub.status.idle":"2025-11-22T15:18:55.354273Z","shell.execute_reply.started":"2025-11-22T15:18:54.466196Z","shell.execute_reply":"2025-11-22T15:18:55.353358Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Notes\n\n* Spread and Range by Class:\n * Every object class shows a wide range of center_y positions, though the interquartile range (the colored section in the box, the thickest part of the violin) and the spread differ between classes.\n\n* Class-Specific Densities:\n * As with center_x, heavier vehicles (car, truck, other_vehicle, bus) typically have broader and multi-peaked distributions, indicating that these classes appear across a large part of the y-axis (road width or positional field).\n\n* Central Tendency:\n * The median lines (white dots in the violin, middle line in the boxplot) show that the typical center_y for cars, trucks, and buses lies lower than for pedestrian and bicycle.\n * Pedestrians and bicycles have higher median center_y values, indicating these objects often appear further in one direction (possibly closer to sidewalks or road edges)\n\n* Multiple Modes:\n * The violin plot makes it clear that some classes show multimodal distributions (multiple peaks and valleys), which reflects the existence of commonly occupied regions—such as parallel lanes, crossing areas, or traffic patterns.\n\n* Outliers and Extreme Values:\n * The boxplot shows outliers for all classes, but especially for pedestrian and bicycle, suggesting occasional detections far from the typical region (possibly crossing or sidewalk areas).\n\n* Comparison Across Classes:\n * Cars, trucks, and buses share some overlap in their spatial distributions but remain clearly distinguishable from pedestrians and bicycles.\n * Pedestrians and bicycles are more tightly clustered around certain y values, reflected in smaller boxes and more peaked violins.\n","metadata":{}},{"cell_type":"markdown","source":"### 2.2.11 `center_z` vs `class_name`","metadata":{}},{"cell_type":"code","source":"comparison_plots(\n df_train_objects,\n _x=\"class_name\",\n _y=\"center_z\", \n _query='class_name != \"motorcycle\" and class_name != \"emergency_vehicle\" and class_name != \"animal\"', \n _palette='RdPu',\n _title='center_z (for different objects)', \n type_of_plot=\"violinplot\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:55.355072Z","iopub.execute_input":"2025-11-22T15:18:55.355272Z","iopub.status.idle":"2025-11-22T15:18:57.485061Z","shell.execute_reply.started":"2025-11-22T15:18:55.355259Z","shell.execute_reply":"2025-11-22T15:18:57.484016Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"comparison_plots(\n df_train_objects,\n _x=\"class_name\",\n _y=\"center_z\", \n _query='class_name != \"motorcycle\" and class_name != \"emergency_vehicle\" and class_name != \"animal\"', \n _palette='RdPu',\n _title='center_z (for different objects)', \n type_of_plot=\"boxplot\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:57.486262Z","iopub.execute_input":"2025-11-22T15:18:57.486436Z","iopub.status.idle":"2025-11-22T15:18:58.361670Z","shell.execute_reply.started":"2025-11-22T15:18:57.486422Z","shell.execute_reply":"2025-11-22T15:18:58.360890Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Notes\n\n* Typical Heights Differ by Class:\n * Cars, trucks, buses, and other vehicles have center_z (height) values that generally cluster around -20 to -15 (units depend on dataset, often meters or centimeters but negative means below a reference plane, usually the LiDAR or camera origin).\n * Pedestrians and bicycles are often even lower, reflecting that their bounding box centers are closer to the ground.\n\n* Range and Outliers:\n * Every class contains a spread of values, but almost all vehicle types exhibit a wider interquartile range and more outliers than bicycles and pedestrians.\n * Many outliers exist above the typical \"ground level,\" suggesting rare annotation points, elevated objects, or labeling noise.\n\n* Peaks and Modal Concentration:\n * The violin plot shows that for most classes, there are concentrated peaks (higher density), indicating common box center heights, with some classes showing multi-modal distributions (multiple bands).\n * This is most visible for cars and trucks, which often have both a main peak and secondary densities at different heights.\n\n* Pedestrian and Bicycle Distributions:\n * The range for pedestrians and bicycles is tighter and lower, and also sometimes includes extreme outliers (perhaps due to mounting errors, sidewalk curbs, or annotation artifacts).\n","metadata":{}},{"cell_type":"markdown","source":"### 2.2.12 `width` vs `class_name`","metadata":{}},{"cell_type":"code","source":"comparison_plots(\n df_train_objects,\n _x=\"class_name\",\n _y=\"width\", \n _query='class_name != \"motorcycle\" and class_name != \"emergency_vehicle\" and class_name != \"animal\"', \n _palette='YlGn',\n _title='width (for different objects)', \n type_of_plot=\"violinplot\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:18:58.362361Z","iopub.execute_input":"2025-11-22T15:18:58.362526Z","iopub.status.idle":"2025-11-22T15:19:00.510995Z","shell.execute_reply.started":"2025-11-22T15:18:58.362513Z","shell.execute_reply":"2025-11-22T15:19:00.510106Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"comparison_plots(\n df_train_objects,\n _x=\"class_name\",\n _y=\"width\", \n _query='class_name != \"motorcycle\" and class_name != \"emergency_vehicle\" and class_name != \"animal\"', \n _palette='YlGn',\n _title='width (for different objects)', \n type_of_plot=\"boxplot\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:00.512010Z","iopub.execute_input":"2025-11-22T15:19:00.512274Z","iopub.status.idle":"2025-11-22T15:19:01.416478Z","shell.execute_reply.started":"2025-11-22T15:19:00.512254Z","shell.execute_reply":"2025-11-22T15:19:01.415393Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Notes\n\n* Clear Class Separation:\n * Vehicle classes (truck, other_vehicle, bus) have much larger widths (centered near 3.0) compared to cars and especially compared to pedestrians and bicycles, which have widths less than 1.0.\n\n* Consistent Ordering:\n * The median width increases from pedestrians and bicycles (smallest), to cars (medium), to trucks, other vehicles, and buses (widest).\n * This matches real-world expectations for object width in autonomous driving data.\n\n* Narrow vs. Wide Spread:\n * Pedestrians and bicycles not only have the smallest average width, but also the narrowest interquartile range (both plots), indicating consistency in object size.\n * Cars and especially heavy vehicles show more variation in width.\n\n* Multimodality and Density Peaks:\n * The violin plot reveals multiple peaks for trucks, other vehicles, and buses, suggesting subclasses or annotation clusters (e.g., different vehicle types or annotation standards).\n\n* Outlier Presence:\n * Boxplots show outliers especially for the car and heavy vehicle classes.\n * Some outlier widths for pedestrians and bicycles may indicate annotation noise or unusual scenarios (groups, occlusions, or side-view errors).\n\n* Violin and Boxplot Consistency:\n * Both plots support each other's findings: the dense, fat regions of the violins correspond to the box heights in the boxplot.","metadata":{}},{"cell_type":"markdown","source":"### 2.2.13 `length` vs `class_name`","metadata":{}},{"cell_type":"code","source":"comparison_plots(\n df_train_objects,\n _x=\"class_name\",\n _y=\"length\", \n _query='class_name != \"motorcycle\" and class_name != \"emergency_vehicle\" and class_name != \"animal\"', \n _palette='Purples',\n _title='length (for different objects)', \n type_of_plot=\"violinplot\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:01.417491Z","iopub.execute_input":"2025-11-22T15:19:01.417715Z","iopub.status.idle":"2025-11-22T15:19:03.559311Z","shell.execute_reply.started":"2025-11-22T15:19:01.417699Z","shell.execute_reply":"2025-11-22T15:19:03.558245Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"comparison_plots(\n df_train_objects,\n _x=\"class_name\",\n _y=\"length\", \n _query='class_name != \"motorcycle\" and class_name != \"emergency_vehicle\" and class_name != \"animal\"', \n _palette='Purples',\n _title='length (for different objects)', \n type_of_plot=\"boxplot\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:03.560785Z","iopub.execute_input":"2025-11-22T15:19:03.561121Z","iopub.status.idle":"2025-11-22T15:19:04.484202Z","shell.execute_reply.started":"2025-11-22T15:19:03.561092Z","shell.execute_reply":"2025-11-22T15:19:04.483315Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Notes\n\n* Length Orders Reflect Reality:\n * Buses have the highest median length, followed by trucks and other vehicles.\n * Cars have a lower median and less spread than the larger vehicles, while pedestrians and bicycles exhibit the smallest lengths.\n\n* Well-Separated Classes:\n * There’s a distinct separation between vehicle classes (car, truck, other_vehicle, bus), which have much greater lengths, versus non-vehicle classes (pedestrian, bicycle), which are much shorter on average.\n\n* Spread and Outliers:\n * Vehicles, especially trucks and buses, show a wide range and many outliers (visible as dots above the boxes and as tails on the violins), signaling substantial physical variation and possibly annotation outliers.\n * Pedestrians and bicycles have tighter interquartile ranges and fewer extreme outliers, reflecting greater uniformity in bounding box length.\n\n* Multimodal and Right-Skewed Distributions:\n * In the violin plot, several vehicle categories appear to have long tails or multiple peaks, suggesting diversity in vehicle subtypes, perspectives, or scene conditions.\n * Cars (and to some extent bicycles and pedestrians) display tighter and more unimodal distributions.\n\n* Consistent Patterns in Both Plots:\n * Both plots convey that objects in your data conform to real-world object length rankings, with so-called “heavy” vehicles substantially exceeding both cars and vulnerable road users (pedestrian/bicycle).","metadata":{}},{"cell_type":"markdown","source":"### 2.2.14 `height` vs `class_name`","metadata":{}},{"cell_type":"code","source":"comparison_plots(\n df_train_objects,\n _x=\"class_name\",\n _y=\"height\", \n _query='class_name != \"motorcycle\" and class_name != \"emergency_vehicle\" and class_name != \"animal\"', \n _palette='Reds',\n _title='height (for different objects)', \n type_of_plot=\"violinplot\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:04.485316Z","iopub.execute_input":"2025-11-22T15:19:04.485903Z","iopub.status.idle":"2025-11-22T15:19:06.570235Z","shell.execute_reply.started":"2025-11-22T15:19:04.485886Z","shell.execute_reply":"2025-11-22T15:19:06.569135Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"comparison_plots(df_train_objects, _x=\"class_name\", _y=\"height\", \n _query='class_name != \"motorcycle\" and class_name != \"emergency_vehicle\" and class_name != \"animal\"', \n _palette='Reds',\n _title='height (for different objects)', \n type_of_plot=\"boxplot\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:06.575525Z","iopub.execute_input":"2025-11-22T15:19:06.575798Z","iopub.status.idle":"2025-11-22T15:19:07.474827Z","shell.execute_reply.started":"2025-11-22T15:19:06.575782Z","shell.execute_reply":"2025-11-22T15:19:07.473952Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Notes so far\n\n* Height Ranks by Class:\n * Trucks, buses, and other heavy vehicles have the highest typical heights, with medians around 3 to 4 units. Cars rank lower, while pedestrians and bicycles are the shortest object classes.\n\n* Distinct Separation:\n * There's clear visual separation in median height between object classes—heavy vehicles are consistently taller than cars, which are taller than pedestrians and bicycles.\n\n* Tight Spread for Non-Vehicles:\n * Pedestrians and bicycles show smaller interquartile ranges and less spread. Their heights are concentrated and less variable, as expected given the physical constraints of people and bicycles.\n\n* Outliers and Maximums:\n * Every class has some outliers, but especially trucks, other vehicles, and buses, reflecting a combination of rare very tall vehicles, potential annotation errors, or stacked/double-decker types.\n * A few extreme outliers (as seen in the boxplot) may signal annotation anomalies.\n\n* Density Peaks and Modalities:\n * Violin plots reveal where most measurements cluster—the fattest parts are the most common heights for each class.\n * Cars and heavy vehicles usually have a primary mode, while a more complex shape for buses, trucks, or other vehicles suggests multiple subtypes or sample characteristics.","metadata":{}},{"cell_type":"markdown","source":"# 3. LIDAR Code","metadata":{}},{"cell_type":"markdown","source":"## 3.1 Helper Functions","metadata":{}},{"cell_type":"code","source":"# Lyft Dataset SDK dev-kit.\n# Code written by Oscar Beijbom, 2018.\n# Licensed under the Creative Commons [see licence.txt]\n# Modified by Vladimir Iglovikov 2019.\n\nclass PointCloud(ABC):\n \"\"\"\n Abstract class for manipulating and viewing point clouds.\n Every point cloud (lidar and radar) consists of points where:\n - Dimensions 0, 1, 2 represent x, y, z coordinates.\n These are modified when the point cloud is rotated or translated.\n - All other dimensions are optional. Hence these have to be manually modified if the reference frame changes.\n \"\"\"\n\n def __init__(self, points: np.ndarray):\n \"\"\"\n Initialize a point cloud and check it has the correct dimensions.\n :param points: . d-dimensional input point cloud matrix.\n \"\"\"\n assert points.shape[0] == self.nbr_dims(), (\n \"Error: Pointcloud points must have format: %d x n\" % self.nbr_dims()\n )\n self.points = points\n\n @staticmethod\n @abstractmethod\n def nbr_dims() -> int:\n \"\"\"Returns the number of dimensions.\n Returns: Number of dimensions.\n \"\"\"\n pass\n\n @classmethod\n @abstractmethod\n def from_file(cls, file_name: str) -> \"PointCloud\":\n \"\"\"Loads point cloud from disk.\n Args:\n file_name: Path of the pointcloud file on disk.\n Returns: PointCloud instance.\n \"\"\"\n pass\n\n @classmethod\n def from_file_multisweep(\n cls, lyftd, sample_rec: Dict, chan: str, ref_chan: str, num_sweeps: int = 26, min_distance: float = 1.0\n ) -> Tuple[\"PointCloud\", np.ndarray]:\n \"\"\"Return a point cloud that aggregates multiple sweeps.\n As every sweep is in a different coordinate frame, we need to map the coordinates to a single reference frame.\n As every sweep has a different timestamp, we need to account for that in the transformations and timestamps.\n Args:\n lyftd: A LyftDataset instance.\n sample_rec: The current sample.\n chan: The radar channel from which we track back n sweeps to aggregate the point cloud.\n ref_chan: The reference channel of the current sample_rec that the point clouds are mapped to.\n num_sweeps: Number of sweeps to aggregated.\n min_distance: Distance below which points are discarded.\n Returns: (all_pc, all_times). The aggregated point cloud and timestamps.\n \"\"\"\n\n # Init\n points = np.zeros((cls.nbr_dims(), 0))\n all_pc = cls(points)\n all_times = np.zeros((1, 0))\n\n # Get reference pose and timestamp\n ref_sd_token = sample_rec[\"data\"][ref_chan]\n ref_sd_rec = lyftd.get(\"sample_data\", ref_sd_token)\n ref_pose_rec = lyftd.get(\"ego_pose\", ref_sd_rec[\"ego_pose_token\"])\n ref_cs_rec = lyftd.get(\"calibrated_sensor\", ref_sd_rec[\"calibrated_sensor_token\"])\n ref_time = 1e-6 * ref_sd_rec[\"timestamp\"]\n\n # Homogeneous transform from ego car frame to reference frame\n ref_from_car = transform_matrix(ref_cs_rec[\"translation\"], Quaternion(ref_cs_rec[\"rotation\"]), inverse=True)\n\n # Homogeneous transformation matrix from global to _current_ ego car frame\n car_from_global = transform_matrix(\n ref_pose_rec[\"translation\"], Quaternion(ref_pose_rec[\"rotation\"]), inverse=True\n )\n\n # Aggregate current and previous sweeps.\n sample_data_token = sample_rec[\"data\"][chan]\n current_sd_rec = lyftd.get(\"sample_data\", sample_data_token)\n for _ in range(num_sweeps):\n # Load up the pointcloud.\n current_pc = cls.from_file(lyftd.data_path / ('train_' + current_sd_rec[\"filename\"]))\n\n # Get past pose.\n current_pose_rec = lyftd.get(\"ego_pose\", current_sd_rec[\"ego_pose_token\"])\n global_from_car = transform_matrix(\n current_pose_rec[\"translation\"], Quaternion(current_pose_rec[\"rotation\"]), inverse=False\n )\n\n # Homogeneous transformation matrix from sensor coordinate frame to ego car frame.\n current_cs_rec = lyftd.get(\"calibrated_sensor\", current_sd_rec[\"calibrated_sensor_token\"])\n car_from_current = transform_matrix(\n current_cs_rec[\"translation\"], Quaternion(current_cs_rec[\"rotation\"]), inverse=False\n )\n\n # Fuse four transformation matrices into one and perform transform.\n trans_matrix = reduce(np.dot, [ref_from_car, car_from_global, global_from_car, car_from_current])\n current_pc.transform(trans_matrix)\n\n # Remove close points and add timevector.\n current_pc.remove_close(min_distance)\n time_lag = ref_time - 1e-6 * current_sd_rec[\"timestamp\"] # positive difference\n times = time_lag * np.ones((1, current_pc.nbr_points()))\n all_times = np.hstack((all_times, times))\n\n # Merge with key pc.\n all_pc.points = np.hstack((all_pc.points, current_pc.points))\n\n # Abort if there are no previous sweeps.\n if current_sd_rec[\"prev\"] == \"\":\n break\n else:\n current_sd_rec = lyftd.get(\"sample_data\", current_sd_rec[\"prev\"])\n\n return all_pc, all_times\n\n def nbr_points(self) -> int:\n \"\"\"Returns the number of points.\"\"\"\n return self.points.shape[1]\n\n def subsample(self, ratio: float) -> None:\n \"\"\"Sub-samples the pointcloud.\n Args:\n ratio: Fraction to keep.\n \"\"\"\n selected_ind = np.random.choice(np.arange(0, self.nbr_points()), size=int(self.nbr_points() * ratio))\n self.points = self.points[:, selected_ind]\n\n def remove_close(self, radius: float) -> None:\n \"\"\"Removes point too close within a certain radius from origin.\n Args:\n radius: Radius below which points are removed.\n Returns:\n \"\"\"\n x_filt = np.abs(self.points[0, :]) < radius\n y_filt = np.abs(self.points[1, :]) < radius\n not_close = np.logical_not(np.logical_and(x_filt, y_filt))\n self.points = self.points[:, not_close]\n\n def translate(self, x: np.ndarray) -> None:\n \"\"\"Applies a translation to the point cloud.\n Args:\n x: . Translation in x, y, z.\n \"\"\"\n for i in range(3):\n self.points[i, :] = self.points[i, :] + x[i]\n\n def rotate(self, rot_matrix: np.ndarray) -> None:\n \"\"\"Applies a rotation.\n Args:\n rot_matrix: . Rotation matrix.\n Returns:\n \"\"\"\n self.points[:3, :] = np.dot(rot_matrix, self.points[:3, :])\n\n def transform(self, transf_matrix: np.ndarray) -> None:\n \"\"\"Applies a homogeneous transform.\n Args:\n transf_matrix: transf_matrix: . Homogenous transformation matrix.\n \"\"\"\n self.points[:3, :] = transf_matrix.dot(np.vstack((self.points[:3, :], np.ones(self.nbr_points()))))[:3, :]\n\n def render_height(\n self,\n ax: Axes,\n view: np.ndarray = np.eye(4),\n x_lim: Tuple = (-20, 20),\n y_lim: Tuple = (-20, 20),\n marker_size: float = 1,\n ) -> None:\n \"\"\"Simple method that applies a transformation and then scatter plots the points colored by height (z-value).\n Args:\n ax: Axes on which to render the points.\n view: . Defines an arbitrary projection (n <= 4).\n x_lim: (min , max ). x range for plotting.\n y_lim: (min , max ). y range for plotting.\n marker_size: Marker size.\n \"\"\"\n self._render_helper(2, ax, view, x_lim, y_lim, marker_size)\n\n def render_intensity(\n self,\n ax: Axes,\n view: np.ndarray = np.eye(4),\n x_lim: Tuple = (-20, 20),\n y_lim: Tuple = (-20, 20),\n marker_size: float = 1,\n ) -> None:\n \"\"\"Very simple method that applies a transformation and then scatter plots the points colored by intensity.\n Args:\n ax: Axes on which to render the points.\n view: . Defines an arbitrary projection (n <= 4).\n x_lim: (min , max ).\n y_lim: (min , max ).\n marker_size: Marker size.\n Returns:\n \"\"\"\n self._render_helper(3, ax, view, x_lim, y_lim, marker_size)\n\n def _render_helper(\n self, color_channel: int, ax: Axes, view: np.ndarray, x_lim: Tuple, y_lim: Tuple, marker_size: float\n ) -> None:\n \"\"\"Helper function for rendering.\n Args:\n color_channel: Point channel to use as color.\n ax: Axes on which to render the points.\n view: . Defines an arbitrary projection (n <= 4).\n x_lim: (min , max ).\n y_lim: (min , max ).\n marker_size: Marker size.\n \"\"\"\n points = view_points(self.points[:3, :], view, normalize=False)\n ax.scatter(points[0, :], points[1, :], c=self.points[color_channel, :], s=marker_size)\n ax.set_xlim(x_lim[0], x_lim[1])\n ax.set_ylim(y_lim[0], y_lim[1])\n\n\nclass LidarPointCloud(PointCloud):\n @staticmethod\n def nbr_dims() -> int:\n \"\"\"Returns the number of dimensions.\n Returns: Number of dimensions.\n \"\"\"\n return 4\n\n @classmethod\n def from_file(cls, file_name: Path) -> \"LidarPointCloud\":\n \"\"\"Loads LIDAR data from binary numpy format. Data is stored as (x, y, z, intensity, ring index).\n Args:\n file_name: Path of the pointcloud file on disk.\n Returns: LidarPointCloud instance (x, y, z, intensity).\n \"\"\"\n\n assert file_name.suffix == \".bin\", \"Unsupported filetype {}\".format(file_name)\n\n scan = np.fromfile(str(file_name), dtype=np.float32)\n points = scan.reshape((-1, 5))[:, : cls.nbr_dims()]\n return cls(points.T)\n\n\nclass RadarPointCloud(PointCloud):\n\n # Class-level settings for radar pointclouds, see from_file().\n invalid_states = [0] # type: List[int]\n dynprop_states = range(7) # type: List[int] # Use [0, 2, 6] for moving objects only.\n ambig_states = [3] # type: List[int]\n\n @staticmethod\n def nbr_dims() -> int:\n \"\"\"Returns the number of dimensions.\n Returns: Number of dimensions.\n \"\"\"\n return 18\n\n @classmethod\n def from_file(\n cls,\n file_name: Path,\n invalid_states: List[int] = None,\n dynprop_states: List[int] = None,\n ambig_states: List[int] = None,\n ) -> \"RadarPointCloud\":\n \"\"\"Loads RADAR data from a Point Cloud Data file. See details below.\n Args:\n file_name: The path of the pointcloud file.\n invalid_states: Radar states to be kept. See details below.\n dynprop_states: Radar states to be kept. Use [0, 2, 6] for moving objects only. See details below.\n ambig_states: Radar states to be kept. See details below. To keep all radar returns,\n set each state filter to range(18).\n Returns: . Point cloud matrix with d dimensions and n points.\n Example of the header fields:\n # .PCD v0.7 - Point Cloud Data file format\n VERSION 0.7\n FIELDS x y z dyn_prop id rcs vx vy vx_comp vy_comp is_quality_valid ambig_\n state x_rms y_rms invalid_state pdh0 vx_rms vy_rms\n SIZE 4 4 4 1 2 4 4 4 4 4 1 1 1 1 1 1 1 1\n TYPE F F F I I F F F F F I I I I I I I I\n COUNT 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n WIDTH 125\n HEIGHT 1\n VIEWPOINT 0 0 0 1 0 0 0\n POINTS 125\n DATA binary\n Below some of the fields are explained in more detail:\n x is front, y is left\n vx, vy are the velocities in m/s.\n vx_comp, vy_comp are the velocities in m/s compensated by the ego motion.\n We recommend using the compensated velocities.\n invalid_state: state of Cluster validity state.\n (Invalid states)\n 0x01\tinvalid due to low RCS\n 0x02\tinvalid due to near-field artefact\n 0x03\tinvalid far range cluster because not confirmed in near range\n 0x05\treserved\n 0x06\tinvalid cluster due to high mirror probability\n 0x07\tInvalid cluster because outside sensor field of view\n 0x0d\treserved\n 0x0e\tinvalid cluster because it is a harmonics\n (Valid states)\n 0x00\tvalid\n 0x04\tvalid cluster with low RCS\n 0x08\tvalid cluster with azimuth correction due to elevation\n 0x09\tvalid cluster with high child probability\n 0x0a\tvalid cluster with high probability of being a 50 deg artefact\n 0x0b\tvalid cluster but no local maximum\n 0x0c\tvalid cluster with high artefact probability\n 0x0f\tvalid cluster with above 95m in near range\n 0x10\tvalid cluster with high multi-target probability\n 0x11\tvalid cluster with suspicious angle\n dynProp: Dynamic property of cluster to indicate if is moving or not.\n 0: moving\n 1: stationary\n 2: oncoming\n 3: stationary candidate\n 4: unknown\n 5: crossing stationary\n 6: crossing moving\n 7: stopped\n ambig_state: State of Doppler (radial velocity) ambiguity solution.\n 0: invalid\n 1: ambiguous\n 2: staggered ramp\n 3: unambiguous\n 4: stationary candidates\n pdh0: False alarm probability of cluster (i.e. probability of being an artefact caused\n by multipath or similar).\n 0: invalid\n 1: <25%\n 2: 50%\n 3: 75%\n 4: 90%\n 5: 99%\n 6: 99.9%\n 7: <=100%\n \"\"\"\n\n assert file_name.suffix == \".pcd\", \"Unsupported filetype {}\".format(file_name)\n\n meta = []\n with open(str(file_name), \"rb\") as f:\n for line in f:\n line = line.strip().decode(\"utf-8\")\n meta.append(line)\n if line.startswith(\"DATA\"):\n break\n\n data_binary = f.read()\n\n # Get the header rows and check if they appear as expected.\n assert meta[0].startswith(\"#\"), \"First line must be comment\"\n assert meta[1].startswith(\"VERSION\"), \"Second line must be VERSION\"\n sizes = meta[3].split(\" \")[1:]\n types = meta[4].split(\" \")[1:]\n counts = meta[5].split(\" \")[1:]\n width = int(meta[6].split(\" \")[1])\n height = int(meta[7].split(\" \")[1])\n data = meta[10].split(\" \")[1]\n feature_count = len(types)\n assert width > 0\n assert len([c for c in counts if c != c]) == 0, \"Error: COUNT not supported!\"\n assert height == 1, \"Error: height != 0 not supported!\"\n assert data == \"binary\"\n\n # Lookup table for how to decode the binaries.\n unpacking_lut = {\n \"F\": {2: \"e\", 4: \"f\", 8: \"d\"},\n \"I\": {1: \"b\", 2: \"h\", 4: \"i\", 8: \"q\"},\n \"U\": {1: \"B\", 2: \"H\", 4: \"I\", 8: \"Q\"},\n }\n types_str = \"\".join([unpacking_lut[t][int(s)] for t, s in zip(types, sizes)])\n\n # Decode each point.\n offset = 0\n point_count = width\n points = []\n for i in range(point_count):\n point = []\n for p in range(feature_count):\n start_p = offset\n end_p = start_p + int(sizes[p])\n assert end_p < len(data_binary)\n point_p = struct.unpack(types_str[p], data_binary[start_p:end_p])[0]\n point.append(point_p)\n offset = end_p\n points.append(point)\n\n # A NaN in the first point indicates an empty pointcloud.\n point = np.array(points[0])\n if np.any(np.isnan(point)):\n return cls(np.zeros((feature_count, 0)))\n\n # Convert to numpy matrix.\n points = np.array(points).transpose()\n\n # If no parameters are provided, use default settings.\n invalid_states = cls.invalid_states if invalid_states is None else invalid_states\n dynprop_states = cls.dynprop_states if dynprop_states is None else dynprop_states\n ambig_states = cls.ambig_states if ambig_states is None else ambig_states\n\n # Filter points with an invalid state.\n valid = [p in invalid_states for p in points[-4, :]]\n points = points[:, valid]\n\n # Filter by dynProp.\n valid = [p in dynprop_states for p in points[3, :]]\n points = points[:, valid]\n\n # Filter by ambig_state.\n valid = [p in ambig_states for p in points[11, :]]\n points = points[:, valid]\n\n return cls(points)\n\n\nclass Box:\n \"\"\" Simple data class representing a 3d box including, label, score and velocity. \"\"\"\n\n def __init__(\n self,\n center: List[float],\n size: List[float],\n orientation: Quaternion,\n label: int = np.nan,\n score: float = np.nan,\n velocity: Tuple = (np.nan, np.nan, np.nan),\n name: str = None,\n token: str = None,\n ):\n \"\"\"\n Args:\n center: Center of box given as x, y, z.\n size: Size of box in width, length, height.\n orientation: Box orientation.\n label: Integer label, optional.\n score: Classification score, optional.\n velocity: Box velocity in x, y, z direction.\n name: Box name, optional. Can be used e.g. for denote category name.\n token: Unique string identifier from DB.\n \"\"\"\n assert not np.any(np.isnan(center))\n assert not np.any(np.isnan(size))\n assert len(center) == 3\n assert len(size) == 3\n assert type(orientation) == Quaternion\n\n self.center = np.array(center)\n self.wlh = np.array(size)\n self.orientation = orientation\n self.label = int(label) if not np.isnan(label) else label\n self.score = float(score) if not np.isnan(score) else score\n self.velocity = np.array(velocity)\n self.name = name\n self.token = token\n\n def __eq__(self, other):\n center = np.allclose(self.center, other.center)\n wlh = np.allclose(self.wlh, other.wlh)\n orientation = np.allclose(self.orientation.elements, other.orientation.elements)\n label = (self.label == other.label) or (np.isnan(self.label) and np.isnan(other.label))\n score = (self.score == other.score) or (np.isnan(self.score) and np.isnan(other.score))\n vel = np.allclose(self.velocity, other.velocity) or (\n np.all(np.isnan(self.velocity)) and np.all(np.isnan(other.velocity))\n )\n\n return center and wlh and orientation and label and score and vel\n\n def __repr__(self):\n repr_str = (\n \"label: {}, score: {:.2f}, xyz: [{:.2f}, {:.2f}, {:.2f}], wlh: [{:.2f}, {:.2f}, {:.2f}], \"\n \"rot axis: [{:.2f}, {:.2f}, {:.2f}], ang(degrees): {:.2f}, ang(rad): {:.2f}, \"\n \"vel: {:.2f}, {:.2f}, {:.2f}, name: {}, token: {}\"\n )\n\n return repr_str.format(\n self.label,\n self.score,\n self.center[0],\n self.center[1],\n self.center[2],\n self.wlh[0],\n self.wlh[1],\n self.wlh[2],\n self.orientation.axis[0],\n self.orientation.axis[1],\n self.orientation.axis[2],\n self.orientation.degrees,\n self.orientation.radians,\n self.velocity[0],\n self.velocity[1],\n self.velocity[2],\n self.name,\n self.token,\n )\n\n @property\n def rotation_matrix(self) -> np.ndarray:\n \"\"\"Return a rotation matrix.\n Returns: . The box's rotation matrix.\n \"\"\"\n return self.orientation.rotation_matrix\n\n def translate(self, x: np.ndarray) -> None:\n \"\"\"Applies a translation.\n Args:\n x: . Translation in x, y, z direction.\n \"\"\"\n self.center += x\n\n def rotate(self, quaternion: Quaternion) -> None:\n \"\"\"Rotates box.\n Args:\n quaternion: Rotation to apply.\n \"\"\"\n self.center = np.dot(quaternion.rotation_matrix, self.center)\n self.orientation = quaternion * self.orientation\n self.velocity = np.dot(quaternion.rotation_matrix, self.velocity)\n\n def corners(self, wlh_factor: float = 1.0) -> np.ndarray:\n \"\"\"Returns the bounding box corners.\n Args:\n wlh_factor: Multiply width, length, height by a factor to scale the box.\n Returns: First four corners are the ones facing forward.\n The last four are the ones facing backwards.\n \"\"\"\n\n width, length, height = self.wlh * wlh_factor\n\n # 3D bounding box corners. (Convention: x points forward, y to the left, z up.)\n x_corners = length / 2 * np.array([1, 1, 1, 1, -1, -1, -1, -1])\n y_corners = width / 2 * np.array([1, -1, -1, 1, 1, -1, -1, 1])\n z_corners = height / 2 * np.array([1, 1, -1, -1, 1, 1, -1, -1])\n corners = np.vstack((x_corners, y_corners, z_corners))\n\n # Rotate\n corners = np.dot(self.orientation.rotation_matrix, corners)\n\n # Translate\n x, y, z = self.center\n corners[0, :] = corners[0, :] + x\n corners[1, :] = corners[1, :] + y\n corners[2, :] = corners[2, :] + z\n\n return corners\n\n def bottom_corners(self) -> np.ndarray:\n \"\"\"Returns the four bottom corners.\n Returns: . Bottom corners. First two face forward, last two face backwards.\n \"\"\"\n return self.corners()[:, [2, 3, 7, 6]]\n\n def render(\n self,\n axis: Axes,\n view: np.ndarray = np.eye(3),\n normalize: bool = False,\n colors: Tuple = (\"b\", \"r\", \"k\"),\n linewidth: float = 2,\n ):\n \"\"\"Renders the box in the provided Matplotlib axis.\n Args:\n axis: Axis onto which the box should be drawn.\n view: . Define a projection in needed (e.g. for drawing projection in an image).\n normalize: Whether to normalize the remaining coordinate.\n colors: (: 3). Valid Matplotlib colors ( or normalized RGB tuple) for front,\n back and sides.\n linewidth: Width in pixel of the box sides.\n \"\"\"\n corners = view_points(self.corners(), view, normalize=normalize)[:2, :]\n\n def draw_rect(selected_corners, color):\n prev = selected_corners[-1]\n for corner in selected_corners:\n axis.plot([prev[0], corner[0]], [prev[1], corner[1]], color=color, linewidth=linewidth)\n prev = corner\n\n # Draw the sides\n for i in range(4):\n axis.plot(\n [corners.T[i][0], corners.T[i + 4][0]],\n [corners.T[i][1], corners.T[i + 4][1]],\n color=colors[2],\n linewidth=linewidth,\n )\n\n # Draw front (first 4 corners) and rear (last 4 corners) rectangles(3d)/lines(2d)\n draw_rect(corners.T[:4], colors[0])\n draw_rect(corners.T[4:], colors[1])\n\n # Draw line indicating the front\n center_bottom_forward = np.mean(corners.T[2:4], axis=0)\n center_bottom = np.mean(corners.T[[2, 3, 7, 6]], axis=0)\n axis.plot(\n [center_bottom[0], center_bottom_forward[0]],\n [center_bottom[1], center_bottom_forward[1]],\n color=colors[0],\n linewidth=linewidth,\n )\n\n def render_cv2(\n self,\n image: np.ndarray,\n view: np.ndarray = np.eye(3),\n normalize: bool = False,\n colors: Tuple = ((0, 0, 255), (255, 0, 0), (155, 155, 155)),\n linewidth: int = 2,\n ) -> None:\n \"\"\"Renders box using OpenCV2.\n Args:\n image: . Image array. Channels are in BGR order.\n view: . Define a projection if needed (e.g. for drawing projection in an image).\n normalize: Whether to normalize the remaining coordinate.\n colors: ((R, G, B), (R, G, B), (R, G, B)). Colors for front, side & rear.\n linewidth: Linewidth for plot.\n Returns:\n \"\"\"\n corners = view_points(self.corners(), view, normalize=normalize)[:2, :]\n\n def draw_rect(selected_corners, color):\n prev = selected_corners[-1]\n for corner in selected_corners:\n cv2.line(image, (int(prev[0]), int(prev[1])), (int(corner[0]), int(corner[1])), color, linewidth)\n prev = corner\n\n # Draw the sides\n for i in range(4):\n cv2.line(\n image,\n (int(corners.T[i][0]), int(corners.T[i][1])),\n (int(corners.T[i + 4][0]), int(corners.T[i + 4][1])),\n colors[2][::-1],\n linewidth,\n )\n\n # Draw front (first 4 corners) and rear (last 4 corners) rectangles(3d)/lines(2d)\n draw_rect(corners.T[:4], colors[0][::-1])\n draw_rect(corners.T[4:], colors[1][::-1])\n\n # Draw line indicating the front\n center_bottom_forward = np.mean(corners.T[2:4], axis=0)\n center_bottom = np.mean(corners.T[[2, 3, 7, 6]], axis=0)\n cv2.line(\n image,\n (int(center_bottom[0]), int(center_bottom[1])),\n (int(center_bottom_forward[0]), int(center_bottom_forward[1])),\n colors[0][::-1],\n linewidth,\n )\n\n def copy(self) -> \"Box\":\n \"\"\" Create a copy of self.\n Returns: A copy.\n \"\"\"\n return copy.deepcopy(self)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:07.475861Z","iopub.execute_input":"2025-11-22T15:19:07.476111Z","iopub.status.idle":"2025-11-22T15:19:07.522592Z","shell.execute_reply.started":"2025-11-22T15:19:07.476093Z","shell.execute_reply":"2025-11-22T15:19:07.521572Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Lyft Dataset SDK dev-kit.\n# Code written by Oscar Beijbom, 2018.\n# Licensed under the Creative Commons [see licence.txt]\n# Modified by Vladimir Iglovikov 2019.\n\nPYTHON_VERSION = sys.version_info[0]\n\nif not PYTHON_VERSION == 3:\n raise ValueError(\"LyftDataset sdk only supports Python version 3.\")\n\n\nclass LyftDataset:\n \"\"\"Database class for Lyft Dataset to help query and retrieve information from the database.\"\"\"\n\n def __init__(self, data_path: str, json_path: str, verbose: bool = True, map_resolution: float = 0.1):\n \"\"\"Loads database and creates reverse indexes and shortcuts.\n Args:\n data_path: Path to the tables and data.\n json_path: Path to the folder with json files\n verbose: Whether to print status messages during load.\n map_resolution: Resolution of maps (meters).\n \"\"\"\n\n self.data_path = Path(data_path).expanduser().absolute()\n self.json_path = Path(json_path)\n\n self.table_names = [\n \"category\",\n \"attribute\",\n \"visibility\",\n \"instance\",\n \"sensor\",\n \"calibrated_sensor\",\n \"ego_pose\",\n \"log\",\n \"scene\",\n \"sample\",\n \"sample_data\",\n \"sample_annotation\",\n \"map\",\n ]\n\n start_time = time.time()\n\n # Explicitly assign tables to help the IDE determine valid class members.\n self.category = self.__load_table__(\"category\")\n self.attribute = self.__load_table__(\"attribute\")\n self.visibility = self.__load_table__(\"visibility\")\n self.instance = self.__load_table__(\"instance\")\n self.sensor = self.__load_table__(\"sensor\")\n self.calibrated_sensor = self.__load_table__(\"calibrated_sensor\")\n self.ego_pose = self.__load_table__(\"ego_pose\")\n self.log = self.__load_table__(\"log\")\n self.scene = self.__load_table__(\"scene\")\n self.sample = self.__load_table__(\"sample\")\n self.sample_data = self.__load_table__(\"sample_data\")\n self.sample_annotation = self.__load_table__(\"sample_annotation\")\n self.map = self.__load_table__(\"map\")\n\n # Initialize map mask for each map record.\n for map_record in self.map:\n map_record[\"mask\"] = MapMask(self.data_path / 'train_maps/map_raster_palo_alto.png', resolution=map_resolution)\n\n if verbose:\n for table in self.table_names:\n print(\"{} {},\".format(len(getattr(self, table)), table))\n print(\"Done loading in {:.1f} seconds.\\n======\".format(time.time() - start_time))\n\n # Make reverse indexes for common lookups.\n self.__make_reverse_index__(verbose)\n\n # Initialize LyftDatasetExplorer class\n self.explorer = LyftDatasetExplorer(self)\n\n def __load_table__(self, table_name) -> dict:\n \"\"\"Loads a table.\"\"\"\n with open(str(self.json_path.joinpath(\"{}.json\".format(table_name)))) as f:\n table = json.load(f)\n return table\n\n def __make_reverse_index__(self, verbose: bool) -> None:\n \"\"\"De-normalizes database to create reverse indices for common cases.\n Args:\n verbose: Whether to print outputs.\n \"\"\"\n\n start_time = time.time()\n if verbose:\n print(\"Reverse indexing ...\")\n\n # Store the mapping from token to table index for each table.\n self._token2ind = dict()\n for table in self.table_names:\n self._token2ind[table] = dict()\n\n for ind, member in enumerate(getattr(self, table)):\n self._token2ind[table][member[\"token\"]] = ind\n\n # Decorate (adds short-cut) sample_annotation table with for category name.\n for record in self.sample_annotation:\n inst = self.get(\"instance\", record[\"instance_token\"])\n record[\"category_name\"] = self.get(\"category\", inst[\"category_token\"])[\"name\"]\n\n # Decorate (adds short-cut) sample_data with sensor information.\n for record in self.sample_data:\n cs_record = self.get(\"calibrated_sensor\", record[\"calibrated_sensor_token\"])\n sensor_record = self.get(\"sensor\", cs_record[\"sensor_token\"])\n record[\"sensor_modality\"] = sensor_record[\"modality\"]\n record[\"channel\"] = sensor_record[\"channel\"]\n\n # Reverse-index samples with sample_data and annotations.\n for record in self.sample:\n record[\"data\"] = {}\n record[\"anns\"] = []\n\n for record in self.sample_data:\n if record[\"is_key_frame\"]:\n sample_record = self.get(\"sample\", record[\"sample_token\"])\n sample_record[\"data\"][record[\"channel\"]] = record[\"token\"]\n\n for ann_record in self.sample_annotation:\n sample_record = self.get(\"sample\", ann_record[\"sample_token\"])\n sample_record[\"anns\"].append(ann_record[\"token\"])\n\n # Add reverse indices from log records to map records.\n if \"log_tokens\" not in self.map[0].keys():\n raise Exception(\"Error: log_tokens not in map table. This code is not compatible with the teaser dataset.\")\n log_to_map = dict()\n for map_record in self.map:\n for log_token in map_record[\"log_tokens\"]:\n log_to_map[log_token] = map_record[\"token\"]\n for log_record in self.log:\n log_record[\"map_token\"] = log_to_map[log_record[\"token\"]]\n\n if verbose:\n print(\"Done reverse indexing in {:.1f} seconds.\\n======\".format(time.time() - start_time))\n\n def get(self, table_name: str, token: str) -> dict:\n \"\"\"Returns a record from table in constant runtime.\n Args:\n table_name: Table name.\n token: Token of the record.\n Returns: Table record.\n \"\"\"\n\n assert table_name in self.table_names, \"Table {} not found\".format(table_name)\n\n return getattr(self, table_name)[self.getind(table_name, token)]\n\n def getind(self, table_name: str, token: str) -> int:\n \"\"\"Returns the index of the record in a table in constant runtime.\n Args:\n table_name: Table name.\n token: The index of the record in table, table is an array.\n Returns:\n \"\"\"\n return self._token2ind[table_name][token]\n\n def field2token(self, table_name: str, field: str, query) -> List[str]:\n \"\"\"Query all records for a certain field value, and returns the tokens for the matching records.\n Runs in linear time.\n Args:\n table_name: Table name.\n field: Field name.\n query: Query to match against. Needs to type match the content of the query field.\n Returns: List of tokens for the matching records.\n \"\"\"\n matches = []\n for member in getattr(self, table_name):\n if member[field] == query:\n matches.append(member[\"token\"])\n return matches\n\n def get_sample_data_path(self, sample_data_token: str) -> Path:\n \"\"\"Returns the path to a sample_data.\n Args:\n sample_data_token:\n Returns:\n \"\"\"\n\n sd_record = self.get(\"sample_data\", sample_data_token)\n return self.data_path / sd_record[\"filename\"]\n\n def get_sample_data(\n self,\n sample_data_token: str,\n box_vis_level: BoxVisibility = BoxVisibility.ANY,\n selected_anntokens: List[str] = None,\n flat_vehicle_coordinates: bool = False,\n ) -> Tuple[Path, List[Box], np.array]:\n \"\"\"Returns the data path as well as all annotations related to that sample_data.\n The boxes are transformed into the current sensor's coordinate frame.\n Args:\n sample_data_token: Sample_data token.\n box_vis_level: If sample_data is an image, this sets required visibility for boxes.\n selected_anntokens: If provided only return the selected annotation.\n flat_vehicle_coordinates: Instead of current sensor's coordinate frame, use vehicle frame which is\n aligned to z-plane in world\n Returns: (data_path, boxes, camera_intrinsic )\n \"\"\"\n\n # Retrieve sensor & pose records\n sd_record = self.get(\"sample_data\", sample_data_token)\n cs_record = self.get(\"calibrated_sensor\", sd_record[\"calibrated_sensor_token\"])\n sensor_record = self.get(\"sensor\", cs_record[\"sensor_token\"])\n pose_record = self.get(\"ego_pose\", sd_record[\"ego_pose_token\"])\n\n data_path = self.get_sample_data_path(sample_data_token)\n\n if sensor_record[\"modality\"] == \"camera\":\n cam_intrinsic = np.array(cs_record[\"camera_intrinsic\"])\n imsize = (sd_record[\"width\"], sd_record[\"height\"])\n else:\n cam_intrinsic = None\n imsize = None\n\n # Retrieve all sample annotations and map to sensor coordinate system.\n if selected_anntokens is not None:\n boxes = list(map(self.get_box, selected_anntokens))\n else:\n boxes = self.get_boxes(sample_data_token)\n\n # Make list of Box objects including coord system transforms.\n box_list = []\n for box in boxes:\n if flat_vehicle_coordinates:\n # Move box to ego vehicle coord system parallel to world z plane\n ypr = Quaternion(pose_record[\"rotation\"]).yaw_pitch_roll\n yaw = ypr[0]\n\n box.translate(-np.array(pose_record[\"translation\"]))\n box.rotate(Quaternion(scalar=np.cos(yaw / 2), vector=[0, 0, np.sin(yaw / 2)]).inverse)\n\n else:\n # Move box to ego vehicle coord system\n box.translate(-np.array(pose_record[\"translation\"]))\n box.rotate(Quaternion(pose_record[\"rotation\"]).inverse)\n\n # Move box to sensor coord system\n box.translate(-np.array(cs_record[\"translation\"]))\n box.rotate(Quaternion(cs_record[\"rotation\"]).inverse)\n\n if sensor_record[\"modality\"] == \"camera\" and not box_in_image(\n box, cam_intrinsic, imsize, vis_level=box_vis_level\n ):\n continue\n\n box_list.append(box)\n\n return data_path, box_list, cam_intrinsic\n\n def get_box(self, sample_annotation_token: str) -> Box:\n \"\"\"Instantiates a Box class from a sample annotation record.\n Args:\n sample_annotation_token: Unique sample_annotation identifier.\n Returns:\n \"\"\"\n record = self.get(\"sample_annotation\", sample_annotation_token)\n return Box(\n record[\"translation\"],\n record[\"size\"],\n Quaternion(record[\"rotation\"]),\n name=record[\"category_name\"],\n token=record[\"token\"],\n )\n\n def get_boxes(self, sample_data_token: str) -> List[Box]:\n \"\"\"Instantiates Boxes for all annotation for a particular sample_data record. If the sample_data is a\n keyframe, this returns the annotations for that sample. But if the sample_data is an intermediate\n sample_data, a linear interpolation is applied to estimate the location of the boxes at the time the\n sample_data was captured.\n Args:\n sample_data_token: Unique sample_data identifier.\n Returns:\n \"\"\"\n\n # Retrieve sensor & pose records\n sd_record = self.get(\"sample_data\", sample_data_token)\n curr_sample_record = self.get(\"sample\", sd_record[\"sample_token\"])\n\n if curr_sample_record[\"prev\"] == \"\" or sd_record[\"is_key_frame\"]:\n # If no previous annotations available, or if sample_data is keyframe just return the current ones.\n boxes = list(map(self.get_box, curr_sample_record[\"anns\"]))\n\n else:\n prev_sample_record = self.get(\"sample\", curr_sample_record[\"prev\"])\n\n curr_ann_recs = [self.get(\"sample_annotation\", token) for token in curr_sample_record[\"anns\"]]\n prev_ann_recs = [self.get(\"sample_annotation\", token) for token in prev_sample_record[\"anns\"]]\n\n # Maps instance tokens to prev_ann records\n prev_inst_map = {entry[\"instance_token\"]: entry for entry in prev_ann_recs}\n\n t0 = prev_sample_record[\"timestamp\"]\n t1 = curr_sample_record[\"timestamp\"]\n t = sd_record[\"timestamp\"]\n\n # There are rare situations where the timestamps in the DB are off so ensure that t0 < t < t1.\n t = max(t0, min(t1, t))\n\n boxes = []\n for curr_ann_rec in curr_ann_recs:\n\n if curr_ann_rec[\"instance_token\"] in prev_inst_map:\n # If the annotated instance existed in the previous frame, interpolate center & orientation.\n prev_ann_rec = prev_inst_map[curr_ann_rec[\"instance_token\"]]\n\n # Interpolate center.\n center = [\n np.interp(t, [t0, t1], [c0, c1])\n for c0, c1 in zip(prev_ann_rec[\"translation\"], curr_ann_rec[\"translation\"])\n ]\n\n # Interpolate orientation.\n rotation = Quaternion.slerp(\n q0=Quaternion(prev_ann_rec[\"rotation\"]),\n q1=Quaternion(curr_ann_rec[\"rotation\"]),\n amount=(t - t0) / (t1 - t0),\n )\n\n box = Box(\n center,\n curr_ann_rec[\"size\"],\n rotation,\n name=curr_ann_rec[\"category_name\"],\n token=curr_ann_rec[\"token\"],\n )\n else:\n # If not, simply grab the current annotation.\n box = self.get_box(curr_ann_rec[\"token\"])\n\n boxes.append(box)\n return boxes\n\n def box_velocity(self, sample_annotation_token: str, max_time_diff: float = 1.5) -> np.ndarray:\n \"\"\"Estimate the velocity for an annotation.\n If possible, we compute the centered difference between the previous and next frame.\n Otherwise we use the difference between the current and previous/next frame.\n If the velocity cannot be estimated, values are set to np.nan.\n Args:\n sample_annotation_token: Unique sample_annotation identifier.\n max_time_diff: Max allowed time diff between consecutive samples that are used to estimate velocities.\n Returns: . Velocity in x/y/z direction in m/s.\n \"\"\"\n\n current = self.get(\"sample_annotation\", sample_annotation_token)\n has_prev = current[\"prev\"] != \"\"\n has_next = current[\"next\"] != \"\"\n\n # Cannot estimate velocity for a single annotation.\n if not has_prev and not has_next:\n return np.array([np.nan, np.nan, np.nan])\n\n if has_prev:\n first = self.get(\"sample_annotation\", current[\"prev\"])\n else:\n first = current\n\n if has_next:\n last = self.get(\"sample_annotation\", current[\"next\"])\n else:\n last = current\n\n pos_last = np.array(last[\"translation\"])\n pos_first = np.array(first[\"translation\"])\n pos_diff = pos_last - pos_first\n\n time_last = 1e-6 * self.get(\"sample\", last[\"sample_token\"])[\"timestamp\"]\n time_first = 1e-6 * self.get(\"sample\", first[\"sample_token\"])[\"timestamp\"]\n time_diff = time_last - time_first\n\n if has_next and has_prev:\n # If doing centered difference, allow for up to double the max_time_diff.\n max_time_diff *= 2\n\n if time_diff > max_time_diff:\n # If time_diff is too big, don't return an estimate.\n return np.array([np.nan, np.nan, np.nan])\n else:\n return pos_diff / time_diff\n\n def list_categories(self) -> None:\n self.explorer.list_categories()\n\n def list_attributes(self) -> None:\n self.explorer.list_attributes()\n\n def list_scenes(self) -> None:\n self.explorer.list_scenes()\n\n def list_sample(self, sample_token: str) -> None:\n self.explorer.list_sample(sample_token)\n\n def render_pointcloud_in_image(\n self,\n sample_token: str,\n dot_size: int = 5,\n pointsensor_channel: str = \"LIDAR_TOP\",\n camera_channel: str = \"CAM_FRONT\",\n out_path: str = None,\n ) -> None:\n self.explorer.render_pointcloud_in_image(\n sample_token,\n dot_size,\n pointsensor_channel=pointsensor_channel,\n camera_channel=camera_channel,\n out_path=out_path,\n )\n\n def render_sample(\n self,\n sample_token: str,\n box_vis_level: BoxVisibility = BoxVisibility.ANY,\n nsweeps: int = 1,\n out_path: str = None,\n ) -> None:\n self.explorer.render_sample(sample_token, box_vis_level, nsweeps=nsweeps, out_path=out_path)\n\n def render_sample_data(\n self,\n sample_data_token: str,\n with_anns: bool = True,\n box_vis_level: BoxVisibility = BoxVisibility.ANY,\n axes_limit: float = 40,\n ax: Axes = None,\n nsweeps: int = 1,\n out_path: str = None,\n underlay_map: bool = False,\n ) -> None:\n return self.explorer.render_sample_data(\n sample_data_token,\n with_anns,\n box_vis_level,\n axes_limit,\n ax,\n num_sweeps=nsweeps,\n out_path=out_path,\n underlay_map=underlay_map,\n )\n\n def render_annotation(\n self,\n sample_annotation_token: str,\n margin: float = 10,\n view: np.ndarray = np.eye(4),\n box_vis_level: BoxVisibility = BoxVisibility.ANY,\n out_path: str = None,\n ) -> None:\n self.explorer.render_annotation(sample_annotation_token, margin, view, box_vis_level, out_path)\n\n def render_instance(self, instance_token: str, out_path: str = None) -> None:\n self.explorer.render_instance(instance_token, out_path=out_path)\n\n def render_scene(self, scene_token: str, freq: float = 10, imwidth: int = 640, out_path: str = None) -> None:\n self.explorer.render_scene(scene_token, freq, image_width=imwidth, out_path=out_path)\n\n def render_scene_channel(\n self,\n scene_token: str,\n channel: str = \"CAM_FRONT\",\n freq: float = 10,\n imsize: Tuple[float, float] = (640, 360),\n out_path: str = None,\n ) -> None:\n self.explorer.render_scene_channel(\n scene_token=scene_token, channel=channel, freq=freq, image_size=imsize, out_path=out_path\n )\n\n def render_egoposes_on_map(self, log_location: str, scene_tokens: List = None, out_path: str = None) -> None:\n self.explorer.render_egoposes_on_map(log_location, scene_tokens, out_path=out_path)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:07.523587Z","iopub.execute_input":"2025-11-22T15:19:07.523824Z","iopub.status.idle":"2025-11-22T15:19:07.567166Z","shell.execute_reply.started":"2025-11-22T15:19:07.523805Z","shell.execute_reply":"2025-11-22T15:19:07.566179Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class LyftDatasetExplorer:\n \"\"\"Helper class to list and visualize Lyft Dataset data. These are meant to serve as tutorials and templates for\n working with the data.\"\"\"\n\n def __init__(self, lyftd: LyftDataset):\n self.lyftd = lyftd\n\n @staticmethod\n def get_color(category_name: str) -> Tuple[int, int, int]:\n \"\"\"Provides the default colors based on the category names.\n This method works for the general Lyft Dataset categories, as well as the Lyft Dataset detection categories.\n Args:\n category_name:\n Returns:\n \"\"\"\n if \"bicycle\" in category_name or \"motorcycle\" in category_name:\n return 255, 61, 99 # Red\n elif \"vehicle\" in category_name or category_name in [\"bus\", \"car\", \"construction_vehicle\", \"trailer\", \"truck\"]:\n return 255, 158, 0 # Orange\n elif \"pedestrian\" in category_name:\n return 0, 0, 230 # Blue\n elif \"cone\" in category_name or \"barrier\" in category_name:\n return 0, 0, 0 # Black\n else:\n return 255, 0, 255 # Magenta\n\n def list_categories(self) -> None:\n \"\"\"Print categories, counts and stats.\"\"\"\n\n print(\"Category stats\")\n\n # Add all annotations\n categories = dict()\n for record in self.lyftd.sample_annotation:\n if record[\"category_name\"] not in categories:\n categories[record[\"category_name\"]] = []\n categories[record[\"category_name\"]].append(record[\"size\"] + [record[\"size\"][1] / record[\"size\"][0]])\n\n # Print stats\n for name, stats in sorted(categories.items()):\n stats = np.array(stats)\n print(\n \"{:27} n={:5}, width={:5.2f}\\u00B1{:.2f}, len={:5.2f}\\u00B1{:.2f}, height={:5.2f}\\u00B1{:.2f}, \"\n \"lw_aspect={:5.2f}\\u00B1{:.2f}\".format(\n name[:27],\n stats.shape[0],\n np.mean(stats[:, 0]),\n np.std(stats[:, 0]),\n np.mean(stats[:, 1]),\n np.std(stats[:, 1]),\n np.mean(stats[:, 2]),\n np.std(stats[:, 2]),\n np.mean(stats[:, 3]),\n np.std(stats[:, 3]),\n )\n )\n\n def list_attributes(self) -> None:\n \"\"\"Prints attributes and counts.\"\"\"\n attribute_counts = dict()\n for record in self.lyftd.sample_annotation:\n for attribute_token in record[\"attribute_tokens\"]:\n att_name = self.lyftd.get(\"attribute\", attribute_token)[\"name\"]\n if att_name not in attribute_counts:\n attribute_counts[att_name] = 0\n attribute_counts[att_name] += 1\n\n for name, count in sorted(attribute_counts.items()):\n print(\"{}: {}\".format(name, count))\n\n def list_scenes(self) -> None:\n \"\"\" Lists all scenes with some meta data. \"\"\"\n\n def ann_count(record):\n count = 0\n sample = self.lyftd.get(\"sample\", record[\"first_sample_token\"])\n while not sample[\"next\"] == \"\":\n count += len(sample[\"anns\"])\n sample = self.lyftd.get(\"sample\", sample[\"next\"])\n return count\n\n recs = [\n (self.lyftd.get(\"sample\", record[\"first_sample_token\"])[\"timestamp\"], record)\n for record in self.lyftd.scene\n ]\n\n for start_time, record in sorted(recs):\n start_time = self.lyftd.get(\"sample\", record[\"first_sample_token\"])[\"timestamp\"] / 1000000\n length_time = self.lyftd.get(\"sample\", record[\"last_sample_token\"])[\"timestamp\"] / 1000000 - start_time\n location = self.lyftd.get(\"log\", record[\"log_token\"])[\"location\"]\n desc = record[\"name\"] + \", \" + record[\"description\"]\n if len(desc) > 55:\n desc = desc[:51] + \"...\"\n if len(location) > 18:\n location = location[:18]\n\n print(\n \"{:16} [{}] {:4.0f}s, {}, #anns:{}\".format(\n desc,\n datetime.utcfromtimestamp(start_time).strftime(\"%y-%m-%d %H:%M:%S\"),\n length_time,\n location,\n ann_count(record),\n )\n )\n\n def list_sample(self, sample_token: str) -> None:\n \"\"\"Prints sample_data tokens and sample_annotation tokens related to the sample_token.\"\"\"\n\n sample_record = self.lyftd.get(\"sample\", sample_token)\n print(\"Sample: {}\\n\".format(sample_record[\"token\"]))\n for sd_token in sample_record[\"data\"].values():\n sd_record = self.lyftd.get(\"sample_data\", sd_token)\n print(\n \"sample_data_token: {}, mod: {}, channel: {}\".format(\n sd_token, sd_record[\"sensor_modality\"], sd_record[\"channel\"]\n )\n )\n print(\"\")\n for ann_token in sample_record[\"anns\"]:\n ann_record = self.lyftd.get(\"sample_annotation\", ann_token)\n print(\"sample_annotation_token: {}, category: {}\".format(ann_record[\"token\"], ann_record[\"category_name\"]))\n\n def map_pointcloud_to_image(self, pointsensor_token: str, camera_token: str) -> Tuple:\n \"\"\"Given a point sensor (lidar/radar) token and camera sample_data token, load point-cloud and map it to\n the image plane.\n Args:\n pointsensor_token: Lidar/radar sample_data token.\n camera_token: Camera sample_data token.\n Returns: (pointcloud , coloring , image ).\n \"\"\"\n\n cam = self.lyftd.get(\"sample_data\", camera_token)\n pointsensor = self.lyftd.get(\"sample_data\", pointsensor_token)\n pcl_path = self.lyftd.data_path / ('train_' + pointsensor[\"filename\"])\n if pointsensor[\"sensor_modality\"] == \"lidar\":\n pc = LidarPointCloud.from_file(pcl_path)\n else:\n pc = RadarPointCloud.from_file(pcl_path)\n im = Image.open(str(self.lyftd.data_path / ('train_' + cam[\"filename\"])))\n\n # Points live in the point sensor frame. So they need to be transformed via global to the image plane.\n # First step: transform the point-cloud to the ego vehicle frame for the timestamp of the sweep.\n cs_record = self.lyftd.get(\"calibrated_sensor\", pointsensor[\"calibrated_sensor_token\"])\n pc.rotate(Quaternion(cs_record[\"rotation\"]).rotation_matrix)\n pc.translate(np.array(cs_record[\"translation\"]))\n\n # Second step: transform to the global frame.\n poserecord = self.lyftd.get(\"ego_pose\", pointsensor[\"ego_pose_token\"])\n pc.rotate(Quaternion(poserecord[\"rotation\"]).rotation_matrix)\n pc.translate(np.array(poserecord[\"translation\"]))\n\n # Third step: transform into the ego vehicle frame for the timestamp of the image.\n poserecord = self.lyftd.get(\"ego_pose\", cam[\"ego_pose_token\"])\n pc.translate(-np.array(poserecord[\"translation\"]))\n pc.rotate(Quaternion(poserecord[\"rotation\"]).rotation_matrix.T)\n\n # Fourth step: transform into the camera.\n cs_record = self.lyftd.get(\"calibrated_sensor\", cam[\"calibrated_sensor_token\"])\n pc.translate(-np.array(cs_record[\"translation\"]))\n pc.rotate(Quaternion(cs_record[\"rotation\"]).rotation_matrix.T)\n\n # Fifth step: actually take a \"picture\" of the point cloud.\n # Grab the depths (camera frame z axis points away from the camera).\n depths = pc.points[2, :]\n\n # Retrieve the color from the depth.\n coloring = depths\n\n # Take the actual picture (matrix multiplication with camera-matrix + renormalization).\n points = view_points(pc.points[:3, :], np.array(cs_record[\"camera_intrinsic\"]), normalize=True)\n\n # Remove points that are either outside or behind the camera. Leave a margin of 1 pixel for aesthetic reasons.\n mask = np.ones(depths.shape[0], dtype=bool)\n mask = np.logical_and(mask, depths > 0)\n mask = np.logical_and(mask, points[0, :] > 1)\n mask = np.logical_and(mask, points[0, :] < im.size[0] - 1)\n mask = np.logical_and(mask, points[1, :] > 1)\n mask = np.logical_and(mask, points[1, :] < im.size[1] - 1)\n points = points[:, mask]\n coloring = coloring[mask]\n\n return points, coloring, im\n\n def render_pointcloud_in_image(\n self,\n sample_token: str,\n dot_size: int = 2,\n pointsensor_channel: str = \"LIDAR_TOP\",\n camera_channel: str = \"CAM_FRONT\",\n out_path: str = None,\n ) -> None:\n \"\"\"Scatter-plots a point-cloud on top of image.\n Args:\n sample_token: Sample token.\n dot_size: Scatter plot dot size.\n pointsensor_channel: RADAR or LIDAR channel name, e.g. 'LIDAR_TOP'.\n camera_channel: Camera channel name, e.g. 'CAM_FRONT'.\n out_path: Optional path to save the rendered figure to disk.\n Returns:\n \"\"\"\n sample_record = self.lyftd.get(\"sample\", sample_token)\n\n # Here we just grab the front camera and the point sensor.\n pointsensor_token = sample_record[\"data\"][pointsensor_channel]\n camera_token = sample_record[\"data\"][camera_channel]\n\n points, coloring, im = self.map_pointcloud_to_image(pointsensor_token, camera_token)\n plt.figure(figsize=(9, 16))\n plt.imshow(im)\n plt.scatter(points[0, :], points[1, :], c=coloring, s=dot_size)\n plt.axis(\"off\")\n\n if out_path is not None:\n plt.savefig(out_path)\n\n def render_sample(\n self, token: str, box_vis_level: BoxVisibility = BoxVisibility.ANY, nsweeps: int = 1, out_path: str = None\n ) -> None:\n \"\"\"Render all LIDAR and camera sample_data in sample along with annotations.\n Args:\n token: Sample token.\n box_vis_level: If sample_data is an image, this sets required visibility for boxes.\n nsweeps: Number of sweeps for lidar and radar.\n out_path: Optional path to save the rendered figure to disk.\n Returns:\n \"\"\"\n record = self.lyftd.get(\"sample\", token)\n\n # Separate RADAR from LIDAR and vision.\n radar_data = {}\n nonradar_data = {}\n for channel, token in record[\"data\"].items():\n sd_record = self.lyftd.get(\"sample_data\", token)\n sensor_modality = sd_record[\"sensor_modality\"]\n if sensor_modality in [\"lidar\", \"camera\"]:\n nonradar_data[channel] = token\n else:\n radar_data[channel] = token\n\n num_radar_plots = 1 if len(radar_data) > 0 else 0\n\n # Create plots.\n n = num_radar_plots + len(nonradar_data)\n cols = 2\n fig, axes = plt.subplots(int(np.ceil(n / cols)), cols, figsize=(16, 24))\n\n if len(radar_data) > 0:\n # Plot radar into a single subplot.\n ax = axes[0, 0]\n for i, (_, sd_token) in enumerate(radar_data.items()):\n self.render_sample_data(\n sd_token, with_anns=i == 0, box_vis_level=box_vis_level, ax=ax, num_sweeps=nsweeps\n )\n ax.set_title(\"Fused RADARs\")\n\n # Plot camera and lidar in separate subplots.\n for (_, sd_token), ax in zip(nonradar_data.items(), axes.flatten()[num_radar_plots:]):\n self.render_sample_data(sd_token, box_vis_level=box_vis_level, ax=ax, num_sweeps=nsweeps)\n\n axes.flatten()[-1].axis(\"off\")\n plt.tight_layout()\n fig.subplots_adjust(wspace=0, hspace=0)\n\n if out_path is not None:\n plt.savefig(out_path)\n\n def render_ego_centric_map(self, sample_data_token: str, axes_limit: float = 40, ax: Axes = None) -> None:\n \"\"\"Render map centered around the associated ego pose.\n Args:\n sample_data_token: Sample_data token.\n axes_limit: Axes limit measured in meters.\n ax: Axes onto which to render.\n \"\"\"\n\n def crop_image(image: np.array, x_px: int, y_px: int, axes_limit_px: int) -> np.array:\n x_min = int(x_px - axes_limit_px)\n x_max = int(x_px + axes_limit_px)\n y_min = int(y_px - axes_limit_px)\n y_max = int(y_px + axes_limit_px)\n\n cropped_image = image[y_min:y_max, x_min:x_max]\n\n return cropped_image\n\n sd_record = self.lyftd.get(\"sample_data\", sample_data_token)\n\n # Init axes.\n if ax is None:\n _, ax = plt.subplots(1, 1, figsize=(9, 9))\n\n sample = self.lyftd.get(\"sample\", sd_record[\"sample_token\"])\n scene = self.lyftd.get(\"scene\", sample[\"scene_token\"])\n log = self.lyftd.get(\"log\", scene[\"log_token\"])\n map = self.lyftd.get(\"map\", log[\"map_token\"])\n map_mask = map[\"mask\"]\n\n pose = self.lyftd.get(\"ego_pose\", sd_record[\"ego_pose_token\"])\n pixel_coords = map_mask.to_pixel_coords(pose[\"translation\"][0], pose[\"translation\"][1])\n\n scaled_limit_px = int(axes_limit * (1.0 / map_mask.resolution))\n mask_raster = map_mask.mask()\n\n cropped = crop_image(mask_raster, pixel_coords[0], pixel_coords[1], int(scaled_limit_px * math.sqrt(2)))\n\n ypr_rad = Quaternion(pose[\"rotation\"]).yaw_pitch_roll\n yaw_deg = -math.degrees(ypr_rad[0])\n\n rotated_cropped = np.array(Image.fromarray(cropped).rotate(yaw_deg))\n ego_centric_map = crop_image(\n rotated_cropped, rotated_cropped.shape[1] / 2, rotated_cropped.shape[0] / 2, scaled_limit_px\n )\n ax.imshow(\n ego_centric_map, extent=[-axes_limit, axes_limit, -axes_limit, axes_limit], cmap=\"gray\", vmin=0, vmax=150\n )\n\n def render_sample_data(\n self,\n sample_data_token: str,\n with_anns: bool = True,\n box_vis_level: BoxVisibility = BoxVisibility.ANY,\n axes_limit: float = 40,\n ax: Axes = None,\n num_sweeps: int = 1,\n out_path: str = None,\n underlay_map: bool = False,\n ):\n \"\"\"Render sample data onto axis.\n Args:\n sample_data_token: Sample_data token.\n with_anns: Whether to draw annotations.\n box_vis_level: If sample_data is an image, this sets required visibility for boxes.\n axes_limit: Axes limit for lidar and radar (measured in meters).\n ax: Axes onto which to render.\n num_sweeps: Number of sweeps for lidar and radar.\n out_path: Optional path to save the rendered figure to disk.\n underlay_map: When set to true, LIDAR data is plotted onto the map. This can be slow.\n \"\"\"\n\n # Get sensor modality.\n sd_record = self.lyftd.get(\"sample_data\", sample_data_token)\n sensor_modality = sd_record[\"sensor_modality\"]\n\n if sensor_modality == \"lidar\":\n # Get boxes in lidar frame.\n _, boxes, _ = self.lyftd.get_sample_data(\n sample_data_token, box_vis_level=box_vis_level, flat_vehicle_coordinates=True\n )\n\n # Get aggregated point cloud in lidar frame.\n sample_rec = self.lyftd.get(\"sample\", sd_record[\"sample_token\"])\n chan = sd_record[\"channel\"]\n ref_chan = \"LIDAR_TOP\"\n pc, times = LidarPointCloud.from_file_multisweep(\n self.lyftd, sample_rec, chan, ref_chan, num_sweeps=num_sweeps\n )\n\n # Compute transformation matrices for lidar point cloud\n cs_record = self.lyftd.get(\"calibrated_sensor\", sd_record[\"calibrated_sensor_token\"])\n pose_record = self.lyftd.get(\"ego_pose\", sd_record[\"ego_pose_token\"])\n vehicle_from_sensor = np.eye(4)\n vehicle_from_sensor[:3, :3] = Quaternion(cs_record[\"rotation\"]).rotation_matrix\n vehicle_from_sensor[:3, 3] = cs_record[\"translation\"]\n\n ego_yaw = Quaternion(pose_record[\"rotation\"]).yaw_pitch_roll[0]\n rot_vehicle_flat_from_vehicle = np.dot(\n Quaternion(scalar=np.cos(ego_yaw / 2), vector=[0, 0, np.sin(ego_yaw / 2)]).rotation_matrix,\n Quaternion(pose_record[\"rotation\"]).inverse.rotation_matrix,\n )\n\n vehicle_flat_from_vehicle = np.eye(4)\n vehicle_flat_from_vehicle[:3, :3] = rot_vehicle_flat_from_vehicle\n\n # Init axes.\n if ax is None:\n _, ax = plt.subplots(1, 1, figsize=(9, 9))\n\n if underlay_map:\n self.render_ego_centric_map(sample_data_token=sample_data_token, axes_limit=axes_limit, ax=ax)\n\n # Show point cloud.\n points = view_points(\n pc.points[:3, :], np.dot(vehicle_flat_from_vehicle, vehicle_from_sensor), normalize=False\n )\n dists = np.sqrt(np.sum(pc.points[:2, :] ** 2, axis=0))\n colors = np.minimum(1, dists / axes_limit / np.sqrt(2))\n ax.scatter(points[0, :], points[1, :], c=colors, s=0.2)\n\n # Show ego vehicle.\n ax.plot(0, 0, \"x\", color=\"red\")\n\n # Show boxes.\n if with_anns:\n for box in boxes:\n c = np.array(self.get_color(box.name)) / 255.0\n box.render(ax, view=np.eye(4), colors=(c, c, c))\n\n # Limit visible range.\n ax.set_xlim(-axes_limit, axes_limit)\n ax.set_ylim(-axes_limit, axes_limit)\n\n elif sensor_modality == \"radar\":\n # Get boxes in lidar frame.\n sample_rec = self.lyftd.get(\"sample\", sd_record[\"sample_token\"])\n lidar_token = sample_rec[\"data\"][\"LIDAR_TOP\"]\n _, boxes, _ = self.lyftd.get_sample_data(lidar_token, box_vis_level=box_vis_level)\n\n # Get aggregated point cloud in lidar frame.\n # The point cloud is transformed to the lidar frame for visualization purposes.\n chan = sd_record[\"channel\"]\n ref_chan = \"LIDAR_TOP\"\n pc, times = RadarPointCloud.from_file_multisweep(\n self.lyftd, sample_rec, chan, ref_chan, num_sweeps=num_sweeps\n )\n\n # Transform radar velocities (x is front, y is left), as these are not transformed when loading the point\n # cloud.\n radar_cs_record = self.lyftd.get(\"calibrated_sensor\", sd_record[\"calibrated_sensor_token\"])\n lidar_sd_record = self.lyftd.get(\"sample_data\", lidar_token)\n lidar_cs_record = self.lyftd.get(\"calibrated_sensor\", lidar_sd_record[\"calibrated_sensor_token\"])\n velocities = pc.points[8:10, :] # Compensated velocity\n velocities = np.vstack((velocities, np.zeros(pc.points.shape[1])))\n velocities = np.dot(Quaternion(radar_cs_record[\"rotation\"]).rotation_matrix, velocities)\n velocities = np.dot(Quaternion(lidar_cs_record[\"rotation\"]).rotation_matrix.T, velocities)\n velocities[2, :] = np.zeros(pc.points.shape[1])\n\n # Init axes.\n if ax is None:\n _, ax = plt.subplots(1, 1, figsize=(9, 9))\n\n # Show point cloud.\n points = view_points(pc.points[:3, :], np.eye(4), normalize=False)\n dists = np.sqrt(np.sum(pc.points[:2, :] ** 2, axis=0))\n colors = np.minimum(1, dists / axes_limit / np.sqrt(2))\n sc = ax.scatter(points[0, :], points[1, :], c=colors, s=3)\n\n # Show velocities.\n points_vel = view_points(pc.points[:3, :] + velocities, np.eye(4), normalize=False)\n max_delta = 10\n deltas_vel = points_vel - points\n deltas_vel = 3 * deltas_vel # Arbitrary scaling\n deltas_vel = np.clip(deltas_vel, -max_delta, max_delta) # Arbitrary clipping\n colors_rgba = sc.to_rgba(colors)\n for i in range(points.shape[1]):\n ax.arrow(points[0, i], points[1, i], deltas_vel[0, i], deltas_vel[1, i], color=colors_rgba[i])\n\n # Show ego vehicle.\n ax.plot(0, 0, \"x\", color=\"black\")\n\n # Show boxes.\n if with_anns:\n for box in boxes:\n c = np.array(self.get_color(box.name)) / 255.0\n box.render(ax, view=np.eye(4), colors=(c, c, c))\n\n # Limit visible range.\n ax.set_xlim(-axes_limit, axes_limit)\n ax.set_ylim(-axes_limit, axes_limit)\n\n elif sensor_modality == \"camera\":\n # Load boxes and image.\n data_path, boxes, camera_intrinsic = self.lyftd.get_sample_data(\n sample_data_token, box_vis_level=box_vis_level\n )\n\n data = Image.open(str(data_path)[:len(str(data_path)) - 46] + 'train_images/' +\\\n str(data_path)[len(str(data_path)) - 39 : len(str(data_path))])\n\n # Init axes.\n if ax is None:\n _, ax = plt.subplots(1, 1, figsize=(9, 16))\n\n # Show image.\n ax.imshow(data)\n\n # Show boxes.\n if with_anns:\n for box in boxes:\n c = np.array(self.get_color(box.name)) / 255.0\n box.render(ax, view=camera_intrinsic, normalize=True, colors=(c, c, c))\n\n # Limit visible range.\n ax.set_xlim(0, data.size[0])\n ax.set_ylim(data.size[1], 0)\n\n else:\n raise ValueError(\"Error: Unknown sensor modality!\")\n\n ax.axis(\"off\")\n ax.set_title(sd_record[\"channel\"])\n ax.set_aspect(\"equal\")\n\n if out_path is not None:\n num = len([name for name in os.listdir(out_path)])\n out_path = out_path + str(num).zfill(5) + \"_\" + sample_data_token + \".png\"\n plt.savefig(out_path)\n plt.close(\"all\")\n return out_path\n\n def render_annotation(\n self,\n ann_token: str,\n margin: float = 10,\n view: np.ndarray = np.eye(4),\n box_vis_level: BoxVisibility = BoxVisibility.ANY,\n out_path: str = None,\n ) -> None:\n \"\"\"Render selected annotation.\n Args:\n ann_token: Sample_annotation token.\n margin: How many meters in each direction to include in LIDAR view.\n view: LIDAR view point.\n box_vis_level: If sample_data is an image, this sets required visibility for boxes.\n out_path: Optional path to save the rendered figure to disk.\n \"\"\"\n\n ann_record = self.lyftd.get(\"sample_annotation\", ann_token)\n sample_record = self.lyftd.get(\"sample\", ann_record[\"sample_token\"])\n assert \"LIDAR_TOP\" in sample_record[\"data\"].keys(), \"No LIDAR_TOP in data, cant render\"\n\n fig, axes = plt.subplots(1, 2, figsize=(18, 9))\n\n # Figure out which camera the object is fully visible in (this may return nothing)\n boxes, cam = [], []\n cams = [key for key in sample_record[\"data\"].keys() if \"CAM\" in key]\n for cam in cams:\n _, boxes, _ = self.lyftd.get_sample_data(\n sample_record[\"data\"][cam], box_vis_level=box_vis_level, selected_anntokens=[ann_token]\n )\n if len(boxes) > 0:\n break # We found an image that matches. Let's abort.\n assert len(boxes) > 0, \"Could not find image where annotation is visible. Try using e.g. BoxVisibility.ANY.\"\n assert len(boxes) < 2, \"Found multiple annotations. Something is wrong!\"\n\n cam = sample_record[\"data\"][cam]\n\n # Plot LIDAR view\n lidar = sample_record[\"data\"][\"LIDAR_TOP\"]\n data_path, boxes, camera_intrinsic = self.lyftd.get_sample_data(lidar, selected_anntokens=[ann_token])\n LidarPointCloud.from_file(Path(str(data_path)[:len(str(data_path)) - 46] + 'train_lidar/' +\\\n str(data_path)[len(str(data_path)) - 40 : len(str(data_path))])).render_height(axes[0], view=view)\n for box in boxes:\n c = np.array(self.get_color(box.name)) / 255.0\n box.render(axes[0], view=view, colors=(c, c, c))\n corners = view_points(boxes[0].corners(), view, False)[:2, :]\n axes[0].set_xlim([np.min(corners[0, :]) - margin, np.max(corners[0, :]) + margin])\n axes[0].set_ylim([np.min(corners[1, :]) - margin, np.max(corners[1, :]) + margin])\n axes[0].axis(\"off\")\n axes[0].set_aspect(\"equal\")\n\n # Plot CAMERA view\n data_path, boxes, camera_intrinsic = self.lyftd.get_sample_data(cam, selected_anntokens=[ann_token])\n im = Image.open(Path(str(data_path)[:len(str(data_path)) - 46] + 'train_images/' +\\\n str(data_path)[len(str(data_path)) - 39 : len(str(data_path))]))\n axes[1].imshow(im)\n axes[1].set_title(self.lyftd.get(\"sample_data\", cam)[\"channel\"])\n axes[1].axis(\"off\")\n axes[1].set_aspect(\"equal\")\n for box in boxes:\n c = np.array(self.get_color(box.name)) / 255.0\n box.render(axes[1], view=camera_intrinsic, normalize=True, colors=(c, c, c))\n\n if out_path is not None:\n plt.savefig(out_path)\n\n def render_instance(self, instance_token: str, out_path: str = None) -> None:\n \"\"\"Finds the annotation of the given instance that is closest to the vehicle, and then renders it.\n Args:\n instance_token: The instance token.\n out_path: Optional path to save the rendered figure to disk.\n Returns:\n \"\"\"\n\n ann_tokens = self.lyftd.field2token(\"sample_annotation\", \"instance_token\", instance_token)\n closest = [np.inf, None]\n for ann_token in ann_tokens:\n ann_record = self.lyftd.get(\"sample_annotation\", ann_token)\n sample_record = self.lyftd.get(\"sample\", ann_record[\"sample_token\"])\n sample_data_record = self.lyftd.get(\"sample_data\", sample_record[\"data\"][\"LIDAR_TOP\"])\n pose_record = self.lyftd.get(\"ego_pose\", sample_data_record[\"ego_pose_token\"])\n dist = np.linalg.norm(np.array(pose_record[\"translation\"]) - np.array(ann_record[\"translation\"]))\n if dist < closest[0]:\n closest[0] = dist\n closest[1] = ann_token\n self.render_annotation(closest[1], out_path=out_path)\n\n def render_scene(self, scene_token: str, freq: float = 10, image_width: int = 640, out_path: Path = None) -> None:\n \"\"\"Renders a full scene with all surround view camera channels.\n Args:\n scene_token: Unique identifier of scene to render.\n freq: Display frequency (Hz).\n image_width: Width of image to render. Height is determined automatically to preserve aspect ratio.\n out_path: Optional path to write a video file of the rendered frames.\n \"\"\"\n\n if out_path is not None:\n assert out_path.suffix == \".avi\"\n\n # Get records from DB.\n scene_rec = self.lyftd.get(\"scene\", scene_token)\n first_sample_rec = self.lyftd.get(\"sample\", scene_rec[\"first_sample_token\"])\n last_sample_rec = self.lyftd.get(\"sample\", scene_rec[\"last_sample_token\"])\n\n channels = [\"CAM_FRONT_LEFT\", \"CAM_FRONT\", \"CAM_FRONT_RIGHT\", \"CAM_BACK_LEFT\", \"CAM_BACK\", \"CAM_BACK_RIGHT\"]\n\n horizontal_flip = [\"CAM_BACK_LEFT\", \"CAM_BACK\", \"CAM_BACK_RIGHT\"] # Flip these for aesthetic reasons.\n\n time_step = 1 / freq * 1e6 # Time-stamps are measured in micro-seconds.\n\n window_name = \"{}\".format(scene_rec[\"name\"])\n cv2.namedWindow(window_name)\n cv2.moveWindow(window_name, 0, 0)\n\n # Load first sample_data record for each channel\n current_recs = {} # Holds the current record to be displayed by channel.\n prev_recs = {} # Hold the previous displayed record by channel.\n for channel in channels:\n current_recs[channel] = self.lyftd.get(\"sample_data\", first_sample_rec[\"data\"][channel])\n prev_recs[channel] = None\n\n # We assume that the resolution is the same for all surround view cameras.\n image_height = int(image_width * current_recs[channels[0]][\"height\"] / current_recs[channels[0]][\"width\"])\n image_size = (image_width, image_height)\n\n # Set some display parameters\n layout = {\n \"CAM_FRONT_LEFT\": (0, 0),\n \"CAM_FRONT\": (image_size[0], 0),\n \"CAM_FRONT_RIGHT\": (2 * image_size[0], 0),\n \"CAM_BACK_LEFT\": (0, image_size[1]),\n \"CAM_BACK\": (image_size[0], image_size[1]),\n \"CAM_BACK_RIGHT\": (2 * image_size[0], image_size[1]),\n }\n\n canvas = np.ones((2 * image_size[1], 3 * image_size[0], 3), np.uint8)\n if out_path is not None:\n fourcc = cv2.VideoWriter_fourcc(*\"MJPG\")\n out = cv2.VideoWriter(out_path, fourcc, freq, canvas.shape[1::-1])\n else:\n out = None\n\n current_time = first_sample_rec[\"timestamp\"]\n\n while current_time < last_sample_rec[\"timestamp\"]:\n\n current_time += time_step\n\n # For each channel, find first sample that has time > current_time.\n for channel, sd_rec in current_recs.items():\n while sd_rec[\"timestamp\"] < current_time and sd_rec[\"next\"] != \"\":\n sd_rec = self.lyftd.get(\"sample_data\", sd_rec[\"next\"])\n current_recs[channel] = sd_rec\n\n # Now add to canvas\n for channel, sd_rec in current_recs.items():\n\n # Only update canvas if we have not already rendered this one.\n if not sd_rec == prev_recs[channel]:\n\n # Get annotations and params from DB.\n image_path, boxes, camera_intrinsic = self.lyftd.get_sample_data(\n sd_rec[\"token\"], box_vis_level=BoxVisibility.ANY\n )\n\n # Load and render\n if not image_path.exists():\n raise Exception(\"Error: Missing image %s\" % image_path)\n im = cv2.imread(str(image_path))\n for box in boxes:\n c = self.get_color(box.name)\n box.render_cv2(im, view=camera_intrinsic, normalize=True, colors=(c, c, c))\n\n im = cv2.resize(im, image_size)\n if channel in horizontal_flip:\n im = im[:, ::-1, :]\n\n canvas[\n layout[channel][1] : layout[channel][1] + image_size[1],\n layout[channel][0] : layout[channel][0] + image_size[0],\n :,\n ] = im\n\n prev_recs[channel] = sd_rec # Store here so we don't render the same image twice.\n\n # Show updated canvas.\n cv2.imshow(window_name, canvas)\n if out_path is not None:\n out.write(canvas)\n\n key = cv2.waitKey(1) # Wait a very short time (1 ms).\n\n if key == 32: # if space is pressed, pause.\n key = cv2.waitKey()\n\n if key == 27: # if ESC is pressed, exit.\n cv2.destroyAllWindows()\n break\n\n cv2.destroyAllWindows()\n if out_path is not None:\n out.release()\n\n def render_scene_channel(\n self,\n scene_token: str,\n channel: str = \"CAM_FRONT\",\n freq: float = 10,\n image_size: Tuple[float, float] = (640, 360),\n out_path: Path = None,\n ) -> None:\n \"\"\"Renders a full scene for a particular camera channel.\n Args:\n scene_token: Unique identifier of scene to render.\n channel: Channel to render.\n freq: Display frequency (Hz).\n image_size: Size of image to render. The larger the slower this will run.\n out_path: Optional path to write a video file of the rendered frames.\n \"\"\"\n\n valid_channels = [\n \"CAM_FRONT_LEFT\",\n \"CAM_FRONT\",\n \"CAM_FRONT_RIGHT\",\n \"CAM_BACK_LEFT\",\n \"CAM_BACK\",\n \"CAM_BACK_RIGHT\",\n ]\n\n assert image_size[0] / image_size[1] == 16 / 9, \"Aspect ratio should be 16/9.\"\n assert channel in valid_channels, \"Input channel {} not valid.\".format(channel)\n\n if out_path is not None:\n assert out_path.suffix == \".avi\"\n\n # Get records from DB\n scene_rec = self.lyftd.get(\"scene\", scene_token)\n sample_rec = self.lyftd.get(\"sample\", scene_rec[\"first_sample_token\"])\n sd_rec = self.lyftd.get(\"sample_data\", sample_rec[\"data\"][channel])\n\n # Open CV init\n name = \"{}: {} (Space to pause, ESC to exit)\".format(scene_rec[\"name\"], channel)\n cv2.namedWindow(name)\n cv2.moveWindow(name, 0, 0)\n\n if out_path is not None:\n fourcc = cv2.VideoWriter_fourcc(*\"MJPG\")\n out = cv2.VideoWriter(out_path, fourcc, freq, image_size)\n else:\n out = None\n\n has_more_frames = True\n while has_more_frames:\n\n # Get data from DB\n image_path, boxes, camera_intrinsic = self.lyftd.get_sample_data(\n sd_rec[\"token\"], box_vis_level=BoxVisibility.ANY\n )\n\n # Load and render\n if not image_path.exists():\n raise Exception(\"Error: Missing image %s\" % image_path)\n image = cv2.imread(str(image_path))\n for box in boxes:\n c = self.get_color(box.name)\n box.render_cv2(image, view=camera_intrinsic, normalize=True, colors=(c, c, c))\n\n # Render\n image = cv2.resize(image, image_size)\n cv2.imshow(name, image)\n if out_path is not None:\n out.write(image)\n\n key = cv2.waitKey(10) # Images stored at approx 10 Hz, so wait 10 ms.\n if key == 32: # If space is pressed, pause.\n key = cv2.waitKey()\n\n if key == 27: # if ESC is pressed, exit\n cv2.destroyAllWindows()\n break\n\n if not sd_rec[\"next\"] == \"\":\n sd_rec = self.lyftd.get(\"sample_data\", sd_rec[\"next\"])\n else:\n has_more_frames = False\n\n cv2.destroyAllWindows()\n if out_path is not None:\n out.release()\n\n def render_egoposes_on_map(\n self,\n log_location: str,\n scene_tokens: List = None,\n close_dist: float = 100,\n color_fg: Tuple[int, int, int] = (167, 174, 186),\n color_bg: Tuple[int, int, int] = (255, 255, 255),\n out_path: Path = None,\n ) -> None:\n \"\"\"Renders ego poses a the map. These can be filtered by location or scene.\n Args:\n log_location: Name of the location, e.g. \"singapore-onenorth\", \"singapore-hollandvillage\",\n \"singapore-queenstown' and \"boston-seaport\".\n scene_tokens: Optional list of scene tokens.\n close_dist: Distance in meters for an ego pose to be considered within range of another ego pose.\n color_fg: Color of the semantic prior in RGB format (ignored if map is RGB).\n color_bg: Color of the non-semantic prior in RGB format (ignored if map is RGB).\n out_path: Optional path to save the rendered figure to disk.\n Returns:\n \"\"\"\n\n # Get logs by location\n log_tokens = [l[\"token\"] for l in self.lyftd.log if l[\"location\"] == log_location]\n assert len(log_tokens) > 0, \"Error: This split has 0 scenes for location %s!\" % log_location\n\n # Filter scenes\n scene_tokens_location = [e[\"token\"] for e in self.lyftd.scene if e[\"log_token\"] in log_tokens]\n if scene_tokens is not None:\n scene_tokens_location = [t for t in scene_tokens_location if t in scene_tokens]\n if len(scene_tokens_location) == 0:\n print(\"Warning: Found 0 valid scenes for location %s!\" % log_location)\n\n map_poses = []\n map_mask = None\n\n print(\"Adding ego poses to map...\")\n for scene_token in tqdm(scene_tokens_location):\n\n # Get records from the database.\n scene_record = self.lyftd.get(\"scene\", scene_token)\n log_record = self.lyftd.get(\"log\", scene_record[\"log_token\"])\n map_record = self.lyftd.get(\"map\", log_record[\"map_token\"])\n map_mask = map_record[\"mask\"]\n\n # For each sample in the scene, store the ego pose.\n sample_tokens = self.lyftd.field2token(\"sample\", \"scene_token\", scene_token)\n for sample_token in sample_tokens:\n sample_record = self.lyftd.get(\"sample\", sample_token)\n\n # Poses are associated with the sample_data. Here we use the lidar sample_data.\n sample_data_record = self.lyftd.get(\"sample_data\", sample_record[\"data\"][\"LIDAR_TOP\"])\n pose_record = self.lyftd.get(\"ego_pose\", sample_data_record[\"ego_pose_token\"])\n\n # Calculate the pose on the map and append\n map_poses.append(\n np.concatenate(\n map_mask.to_pixel_coords(pose_record[\"translation\"][0], pose_record[\"translation\"][1])\n )\n )\n\n # Compute number of close ego poses.\n print(\"Creating plot...\")\n map_poses = np.vstack(map_poses)\n dists = sklearn.metrics.pairwise.euclidean_distances(map_poses * map_mask.resolution)\n close_poses = np.sum(dists < close_dist, axis=0)\n\n if len(np.array(map_mask.mask()).shape) == 3 and np.array(map_mask.mask()).shape[2] == 3:\n # RGB Colour maps\n mask = map_mask.mask()\n else:\n # Monochrome maps\n # Set the colors for the mask.\n mask = Image.fromarray(map_mask.mask())\n mask = np.array(mask)\n\n maskr = color_fg[0] * np.ones(np.shape(mask), dtype=np.uint8)\n maskr[mask == 0] = color_bg[0]\n maskg = color_fg[1] * np.ones(np.shape(mask), dtype=np.uint8)\n maskg[mask == 0] = color_bg[1]\n maskb = color_fg[2] * np.ones(np.shape(mask), dtype=np.uint8)\n maskb[mask == 0] = color_bg[2]\n mask = np.concatenate(\n (np.expand_dims(maskr, axis=2), np.expand_dims(maskg, axis=2), np.expand_dims(maskb, axis=2)), axis=2\n )\n\n # Plot.\n _, ax = plt.subplots(1, 1, figsize=(10, 10))\n ax.imshow(mask)\n title = \"Number of ego poses within {}m in {}\".format(close_dist, log_location)\n ax.set_title(title, color=\"k\")\n sc = ax.scatter(map_poses[:, 0], map_poses[:, 1], s=10, c=close_poses)\n color_bar = plt.colorbar(sc, fraction=0.025, pad=0.04)\n plt.rcParams[\"figure.facecolor\"] = \"black\"\n color_bar_ticklabels = plt.getp(color_bar.ax.axes, \"yticklabels\")\n plt.setp(color_bar_ticklabels, color=\"k\")\n plt.rcParams[\"figure.facecolor\"] = \"white\" # Reset for future plots\n\n if out_path is not None:\n plt.savefig(out_path)\n plt.close(\"all\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:07.568337Z","iopub.execute_input":"2025-11-22T15:19:07.568556Z","iopub.status.idle":"2025-11-22T15:19:07.652912Z","shell.execute_reply.started":"2025-11-22T15:19:07.568541Z","shell.execute_reply":"2025-11-22T15:19:07.651916Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def render_scene(index):\n my_scene = lyft_dataset.scene[index]\n my_sample_token = my_scene[\"first_sample_token\"]\n lyft_dataset.render_sample(my_sample_token)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:07.653841Z","iopub.execute_input":"2025-11-22T15:19:07.654085Z","iopub.status.idle":"2025-11-22T15:19:07.674903Z","shell.execute_reply.started":"2025-11-22T15:19:07.654060Z","shell.execute_reply":"2025-11-22T15:19:07.673866Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def generate_next_token(scene):\n scene = lyft_dataset.scene[scene]\n sample_token = scene['first_sample_token']\n sample_record = lyft_dataset.get(\"sample\", sample_token)\n \n while sample_record['next']:\n sample_token = sample_record['next']\n sample_record = lyft_dataset.get(\"sample\", sample_token)\n \n yield sample_token","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:07.676262Z","iopub.execute_input":"2025-11-22T15:19:07.676498Z","iopub.status.idle":"2025-11-22T15:19:07.698470Z","shell.execute_reply.started":"2025-11-22T15:19:07.676483Z","shell.execute_reply":"2025-11-22T15:19:07.697417Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def animate_images(scene, frames, pointsensor_channel='LIDAR_TOP', interval=1):\n cams = [\n 'CAM_FRONT',\n 'CAM_FRONT_RIGHT',\n 'CAM_BACK_RIGHT',\n 'CAM_BACK',\n 'CAM_BACK_LEFT',\n 'CAM_FRONT_LEFT',\n ]\n\n generator = generate_next_token(scene)\n\n fig, axs = plt.subplots(\n 2, len(cams), figsize=(3*len(cams), 6), \n sharex=True, sharey=True, gridspec_kw = {'wspace': 0, 'hspace': 0.1}\n )\n \n plt.close(fig)\n\n def animate_fn(i):\n for _ in range(interval):\n sample_token = next(generator)\n \n for c, camera_channel in enumerate(cams): \n sample_record = lyft_dataset.get(\"sample\", sample_token)\n\n pointsensor_token = sample_record[\"data\"][pointsensor_channel]\n camera_token = sample_record[\"data\"][camera_channel]\n \n axs[0, c].clear()\n axs[1, c].clear()\n \n lyft_dataset.render_sample_data(camera_token, with_anns=False, ax=axs[0, c])\n lyft_dataset.render_sample_data(camera_token, with_anns=True, ax=axs[1, c])\n \n axs[0, c].set_title(\"\")\n axs[1, c].set_title(\"\")\n\n anim = animation.FuncAnimation(fig, animate_fn, frames=frames, interval=interval)\n \n return anim","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:07.699594Z","iopub.execute_input":"2025-11-22T15:19:07.699887Z","iopub.status.idle":"2025-11-22T15:19:07.719842Z","shell.execute_reply.started":"2025-11-22T15:19:07.699870Z","shell.execute_reply":"2025-11-22T15:19:07.718508Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def animate_lidar(scene, frames, pointsensor_channel='LIDAR_TOP', with_anns=True, interval=1):\n generator = generate_next_token(scene)\n\n fig, axs = plt.subplots(1, 1, figsize=(8, 8))\n plt.close(fig)\n\n def animate_fn(i):\n for _ in range(interval):\n sample_token = next(generator)\n \n axs.clear()\n sample_record = lyft_dataset.get(\"sample\", sample_token)\n pointsensor_token = sample_record[\"data\"][pointsensor_channel]\n lyft_dataset.render_sample_data(pointsensor_token, with_anns=with_anns, ax=axs)\n\n anim = animation.FuncAnimation(fig, animate_fn, frames=frames, interval=interval)\n \n return anim","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:07.720768Z","iopub.execute_input":"2025-11-22T15:19:07.720979Z","iopub.status.idle":"2025-11-22T15:19:07.745180Z","shell.execute_reply.started":"2025-11-22T15:19:07.720960Z","shell.execute_reply":"2025-11-22T15:19:07.743875Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3.2 LyftDataset Object","metadata":{}},{"cell_type":"code","source":"lyft_dataset = LyftDataset(data_path=common_path, json_path=train_json_path)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:07.746524Z","iopub.execute_input":"2025-11-22T15:19:07.746875Z","iopub.status.idle":"2025-11-22T15:19:24.318468Z","shell.execute_reply.started":"2025-11-22T15:19:07.746855Z","shell.execute_reply":"2025-11-22T15:19:24.317690Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The dataset consists of several scences, which are 25-45 second clips of image of LiDAR data from a self-driving car. We can extract and look at the first scence as follows:","metadata":{}},{"cell_type":"code","source":"my_scene = lyft_dataset.scene[0]\nmy_scene","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:24.319426Z","iopub.execute_input":"2025-11-22T15:19:24.319707Z","iopub.status.idle":"2025-11-22T15:19:24.327384Z","shell.execute_reply.started":"2025-11-22T15:19:24.319683Z","shell.execute_reply":"2025-11-22T15:19:24.326023Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"As it can be seen above, each scence consists of a dictionary of information. There are a few token IDs and a name for each scene. The ``name`` matches with the name of the LiDAR data file associated with the given scene.\n\nHere, the LiDAR file's name is: ``host-a101-lidar0-1241893239199111666-1241893264098084346``.","metadata":{}},{"cell_type":"code","source":"render_scene(0)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:24.328954Z","iopub.execute_input":"2025-11-22T15:19:24.329201Z","iopub.status.idle":"2025-11-22T15:19:32.505072Z","shell.execute_reply.started":"2025-11-22T15:19:24.329184Z","shell.execute_reply":"2025-11-22T15:19:32.502851Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"render_scene(1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:32.506063Z","iopub.execute_input":"2025-11-22T15:19:32.506384Z","iopub.status.idle":"2025-11-22T15:19:36.928462Z","shell.execute_reply.started":"2025-11-22T15:19:32.506358Z","shell.execute_reply":"2025-11-22T15:19:36.927252Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"my_sample_token = my_scene[\"first_sample_token\"]\nmy_sample = lyft_dataset.get('sample', my_sample_token)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:36.932532Z","iopub.execute_input":"2025-11-22T15:19:36.932927Z","iopub.status.idle":"2025-11-22T15:19:36.937448Z","shell.execute_reply.started":"2025-11-22T15:19:36.932905Z","shell.execute_reply":"2025-11-22T15:19:36.936631Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"lyft_dataset.render_pointcloud_in_image(sample_token = my_sample[\"token\"],\n dot_size = 1,\n camera_channel = 'CAM_FRONT')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:36.938242Z","iopub.execute_input":"2025-11-22T15:19:36.938415Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"my_sample['data']","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sensor_channel = 'CAM_FRONT'\nmy_sample_data = lyft_dataset.get('sample_data', my_sample['data'][sensor_channel])","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"lyft_dataset.render_sample_data(my_sample_data['token'])","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sensor_channel = 'CAM_BACK'\nmy_sample_data = lyft_dataset.get('sample_data', my_sample['data'][sensor_channel])\nlyft_dataset.render_sample_data(my_sample_data['token'])","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sensor_channel = 'CAM_FRONT_LEFT'\nmy_sample_data = lyft_dataset.get('sample_data', my_sample['data'][sensor_channel])\nlyft_dataset.render_sample_data(my_sample_data['token'])","metadata":{"trusted":true,"execution":{"iopub.status.idle":"2025-11-22T15:19:39.433198Z","shell.execute_reply.started":"2025-11-22T15:19:38.855890Z","shell.execute_reply":"2025-11-22T15:19:39.431342Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sensor_channel = 'CAM_FRONT_RIGHT'\nmy_sample_data = lyft_dataset.get('sample_data', my_sample['data'][sensor_channel])\nlyft_dataset.render_sample_data(my_sample_data['token'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:39.434335Z","iopub.execute_input":"2025-11-22T15:19:39.434676Z","iopub.status.idle":"2025-11-22T15:19:40.032789Z","shell.execute_reply.started":"2025-11-22T15:19:39.434627Z","shell.execute_reply":"2025-11-22T15:19:40.031489Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sensor_channel = 'CAM_BACK_LEFT'\nmy_sample_data = lyft_dataset.get('sample_data', my_sample['data'][sensor_channel])\nlyft_dataset.render_sample_data(my_sample_data['token'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:40.034109Z","iopub.execute_input":"2025-11-22T15:19:40.034400Z","iopub.status.idle":"2025-11-22T15:19:40.532293Z","shell.execute_reply.started":"2025-11-22T15:19:40.034380Z","shell.execute_reply":"2025-11-22T15:19:40.530526Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sensor_channel = 'CAM_BACK_RIGHT'\nmy_sample_data = lyft_dataset.get('sample_data', my_sample['data'][sensor_channel])\nlyft_dataset.render_sample_data(my_sample_data['token'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:40.533359Z","iopub.execute_input":"2025-11-22T15:19:40.533658Z","iopub.status.idle":"2025-11-22T15:19:41.028539Z","shell.execute_reply.started":"2025-11-22T15:19:40.533620Z","shell.execute_reply":"2025-11-22T15:19:41.027697Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"my_annotation_token = my_sample['anns'][10]\nmy_annotation = my_sample_data.get('sample_annotation', my_annotation_token)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:41.029771Z","iopub.execute_input":"2025-11-22T15:19:41.029962Z","iopub.status.idle":"2025-11-22T15:19:41.033899Z","shell.execute_reply.started":"2025-11-22T15:19:41.029947Z","shell.execute_reply":"2025-11-22T15:19:41.032834Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"lyft_dataset.render_annotation(my_annotation_token)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:41.035089Z","iopub.execute_input":"2025-11-22T15:19:41.035355Z","iopub.status.idle":"2025-11-22T15:19:42.791923Z","shell.execute_reply.started":"2025-11-22T15:19:41.035333Z","shell.execute_reply":"2025-11-22T15:19:42.790369Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"my_instance = lyft_dataset.instance[100]\nmy_instance","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:42.793237Z","iopub.execute_input":"2025-11-22T15:19:42.793559Z","iopub.status.idle":"2025-11-22T15:19:42.798382Z","shell.execute_reply.started":"2025-11-22T15:19:42.793543Z","shell.execute_reply":"2025-11-22T15:19:42.797734Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"instance_token = my_instance['token']\nlyft_dataset.render_instance(instance_token)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:42.798949Z","iopub.execute_input":"2025-11-22T15:19:42.799159Z","iopub.status.idle":"2025-11-22T15:19:44.084329Z","shell.execute_reply.started":"2025-11-22T15:19:42.799140Z","shell.execute_reply":"2025-11-22T15:19:44.082542Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"lyft_dataset.render_annotation(my_instance['last_annotation_token'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:44.085339Z","iopub.execute_input":"2025-11-22T15:19:44.085542Z","iopub.status.idle":"2025-11-22T15:19:45.212417Z","shell.execute_reply.started":"2025-11-22T15:19:44.085526Z","shell.execute_reply":"2025-11-22T15:19:45.211216Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"my_scene = lyft_dataset.scene[0]\nmy_sample_token = my_scene[\"first_sample_token\"]\nmy_sample = lyft_dataset.get('sample', my_sample_token)\nlyft_dataset.render_sample_data(my_sample['data']['LIDAR_TOP'], nsweeps=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:45.213880Z","iopub.execute_input":"2025-11-22T15:19:45.214138Z","iopub.status.idle":"2025-11-22T15:19:47.277622Z","shell.execute_reply.started":"2025-11-22T15:19:45.214118Z","shell.execute_reply":"2025-11-22T15:19:47.276425Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"my_scene = lyft_dataset.scene[0]\nmy_sample_token = my_scene[\"first_sample_token\"]\nmy_sample = lyft_dataset.get('sample', my_sample_token)\nlyft_dataset.render_sample_data(my_sample['data']['LIDAR_FRONT_LEFT'], nsweeps=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:47.278761Z","iopub.execute_input":"2025-11-22T15:19:47.279013Z","iopub.status.idle":"2025-11-22T15:19:48.399465Z","shell.execute_reply.started":"2025-11-22T15:19:47.278994Z","shell.execute_reply":"2025-11-22T15:19:48.397901Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"my_scene = lyft_dataset.scene[0]\nmy_sample_token = my_scene[\"first_sample_token\"]\nmy_sample = lyft_dataset.get('sample', my_sample_token)\nlyft_dataset.render_sample_data(my_sample['data']['LIDAR_FRONT_RIGHT'], nsweeps=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:48.401017Z","iopub.execute_input":"2025-11-22T15:19:48.401417Z","iopub.status.idle":"2025-11-22T15:19:49.501298Z","shell.execute_reply.started":"2025-11-22T15:19:48.401396Z","shell.execute_reply":"2025-11-22T15:19:49.500302Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"anim = animate_images(scene=3, frames=100, interval=1)\n# HTML(anim.to_jshtml(fps=8))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:49.502134Z","iopub.execute_input":"2025-11-22T15:19:49.502327Z","iopub.status.idle":"2025-11-22T15:19:49.644772Z","shell.execute_reply.started":"2025-11-22T15:19:49.502313Z","shell.execute_reply":"2025-11-22T15:19:49.644004Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"anim = animate_images(scene=7, frames=100, interval=1)\n# HTML(anim.to_jshtml(fps=8))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:49.645494Z","iopub.execute_input":"2025-11-22T15:19:49.645726Z","iopub.status.idle":"2025-11-22T15:19:49.783199Z","shell.execute_reply.started":"2025-11-22T15:19:49.645710Z","shell.execute_reply":"2025-11-22T15:19:49.781819Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"anim = animate_images(scene=4, frames=100, interval=1)\n# HTML(anim.to_jshtml(fps=8))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:49.784772Z","iopub.execute_input":"2025-11-22T15:19:49.784942Z","iopub.status.idle":"2025-11-22T15:19:49.937274Z","shell.execute_reply.started":"2025-11-22T15:19:49.784928Z","shell.execute_reply":"2025-11-22T15:19:49.935661Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"anim = animate_lidar(scene=5, frames=100, interval=1)\n# HTML(anim.to_jshtml(fps=8))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:49.938820Z","iopub.execute_input":"2025-11-22T15:19:49.939164Z","iopub.status.idle":"2025-11-22T15:19:49.961224Z","shell.execute_reply.started":"2025-11-22T15:19:49.939142Z","shell.execute_reply":"2025-11-22T15:19:49.960308Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"anim = animate_lidar(scene=25, frames=100, interval=1)\n# HTML(anim.to_jshtml(fps=8))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:49.962516Z","iopub.execute_input":"2025-11-22T15:19:49.962811Z","iopub.status.idle":"2025-11-22T15:19:49.992757Z","shell.execute_reply.started":"2025-11-22T15:19:49.962786Z","shell.execute_reply":"2025-11-22T15:19:49.991789Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"anim = animate_lidar(scene=10, frames=100, interval=1)\n# HTML(anim.to_jshtml(fps=8))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-22T15:19:49.993608Z","iopub.execute_input":"2025-11-22T15:19:49.993805Z","iopub.status.idle":"2025-11-22T15:19:50.022477Z","shell.execute_reply.started":"2025-11-22T15:19:49.993791Z","shell.execute_reply":"2025-11-22T15:19:50.021874Z"}},"outputs":[],"execution_count":null}]}