{"cells":[{"metadata":{},"cell_type":"markdown","source":"## <u>Table of Contents</u>\n* [Introduction](#Introduction)\n* [Setup](#Setup)\n* [Knowing About the Visualization Configuration File](#Knowing-About-the-Visualization-Configuration-File)\n* [Loading and Working with the Sample Data](#Loading-and-Working-with-the-Sample-Data)\n* [Dataset Visualization for AV and Agents](#Dataset-Visualization-for-AV-and-Agents)\n    * [Visualize the Autonomous Vehicle](#Visualize-the-Autonomous-Vehicle)\n    * [Visualizing an Agent](#Visualizing-an-Agent)\n    * [Looking at en Entire Scene](#Looking-at-an-Entire-Scene)\n* [Some Resource Lists and References](#Some-Resource-Lists-and-References)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## <u>Introduction</u>\n* We will try to understand and visualize the data for the [Lyft Motion Prediction for Autonomous Vehicles](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/overview) compeition. \n* This competition is quite unique, both in terms of what our objective is and how we handle and interpret the data. So, if you find any mistake or discrepancies anywhere in the notebook, feel free to use the comment section. I will try my best to address them.\n\n### Core Packages\nThere are two core packages for visualization.\n* `rasterization`: This package contains the classes for getting the visual data as multi-channel tensors. After that, this package will help us to turn those multi-channel tensors into RGB images.\n    * `rasterize` method: To get the tensor.\n    * `to_rgb` method: to convert those tensors into RGB image.\n* `visualization`: This package will help us to draw the visual information like trajectories on the converted RGB images.\n\nI hope that everything is understandable upto this point. Still to get to a better idea, let's see how an RGB image after visualization should look like.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Take a look at the following GIF ([Source](https://self-driving.lyft.com/level5/prediction/))**.\n![](https://self-driving.lyft.com/wp-content/uploads/2020/06/motion_dataset_lrg_redux.gif)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"So, what can we infer from the above GIF.\n* One thing is sure, the red box is the autonomous vehicle.\n* The yellow boxes are the agents. We can see vehicular ageints as yellow boxes on the road. But what about other agents like pedestrians? Well, if we look closely, we can see a few yellow dots in the black area. Those maybe pedestrians.\n* And the semantic colored lines are the paths along which we have to predict the motions of the exernal agents in this compeition.\n\nLet's try to confirm our theory by looking at the following image.\n\n![](https://self-driving.lyft.com/wp-content/uploads/2020/06/motion_dataset_2-1.png)\n\nEverything seems fine, except that we did not get confirmation on our pedestrian theory. Hopefully, we can skip that part for now.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## <u>Setup</u>\nWe can either add the L5kit as a utility script or install using pip. Both ways work fine.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install -U l5kit","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's do a preliminary setup of all the things that we will need. This includes all the imports and loading the configuration file for visualization.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"%%writefile visualisation_config.yaml\n# Config format schema number\nformat_version: 4\n\n###################\n## Model options\nmodel_params:\n  model_architecture: \"resnet50\"\n\n  history_num_frames: 0\n  history_step_size: 1\n  history_delta_time: 0.1\n\n  future_num_frames: 50\n  future_step_size: 1\n  future_delta_time: 0.1\n\n###################\n## Input raster parameters\nraster_params:\n  # raster image size [pixels]\n  raster_size:\n    - 224\n    - 224\n  # raster's spatial resolution [meters per pixel]: the size in the real world one pixel corresponds to.\n  pixel_size:\n    - 0.5\n    - 0.5\n  # From 0 to 1 per axis, [0.5,0.5] would show the ego centered in the image.\n  ego_center:\n    - 0.25\n    - 0.5\n  map_type: \"py_semantic\"\n\n  # the keys are relative to the dataset environment variable\n  satellite_map_key: \"aerial_map/aerial_map.png\"\n  semantic_map_key: \"semantic_map/semantic_map.pb\"\n  dataset_meta_key: \"meta.json\"\n\n  # e.g. 0.0 include every obstacle, 0.5 show those obstacles with >0.5 probability of being\n  # one of the classes we care about (cars, bikes, peds, etc.), >=1.0 filter all other agents.\n  filter_agents_threshold: 0.5\n\n###################\n## Data loader options\nval_data_loader:\n  key: \"scenes/sample.zarr\"\n  batch_size: 12\n  shuffle: False\n  num_workers: 16","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport zarr\nimport numpy as np\nimport l5kit\nimport matplotlib.pyplot as plt\nimport matplotlib\n\nfrom l5kit.data import PERCEPTION_LABELS\nfrom l5kit.configs import load_config_data\nfrom l5kit.data import ChunkedDataset, LocalDataManager\nfrom l5kit.dataset import EgoDataset, AgentDataset\nfrom l5kit.visualization import draw_trajectory, TARGET_POINTS_COLOR\nfrom l5kit.geometry import transform_points\nfrom l5kit.rasterization import build_rasterizer\nfrom tqdm import tqdm\nfrom prettytable import PrettyTable","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# matplotlib.style.use('ggplot')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# load the visualizationconfiguration file\ncfg = load_config_data('visualisation_config.yaml')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# set the env variable for the data\nos.environ['L5KIT_DATA_FOLDER'] = '../input/lyft-motion-prediction-autonomous-vehicles'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## <u>Knowing About the Visualization Configuration File</u>\n* Let get to know what information the configuration file can provide us.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Raster Param\")\nfor k, v in cfg['raster_params'].items():\n    print(f\"{k}: {v}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, what do all the above information mean? We will tackle a few important ones.\n* `raster_size`: This is size of the image. That is 224x224 pixels.\n* `pixel_size`: One pixel corresponds to these many meters. So, along the width and height of the image, one pixel will correspond to 0.5 metres in the real world.\n* `ego_enter`: As per the [source](https://github.com/lyft/l5kit/blob/master/examples/visualisation/visualise_data.ipynb) =>  Our raster is centered around an agent, we can move the agent in the image plane with this param. Okay, but still confused what the number mean. If you find an explanation, please tell in the comment section.\n* `map_type`: The rasterizer to be used for visualization.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## <u>Loading and Working with the Sample Data</u>","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dm = LocalDataManager()\ndataset_path = dm.require(cfg['val_data_loader']['key'])\nprint(dataset_path)\nzarr_dataset = ChunkedDataset(dataset_path)\nzarr_dataset.open()\nprint(zarr_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above output gives some useful imformation about the sample scenes.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now, the following block of code will iterate over all the frames in the sample dataset and plot the trajectory of the autonomous vehicle.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"frames = zarr_dataset.frames\ncoords = np.zeros((len(frames), 2))\nfor idx_coord, idx_data in enumerate(tqdm(range(len(frames)), desc='getting centroid to plot trajectory')):\n    frame = zarr_dataset.frames[idx_data]\n    coords[idx_coord] = frame['ego_translation'][:2]\n\n\nplt.scatter(coords[:, 0], coords[:, 1], marker='.')\naxes = plt.gca()\naxes.set_xlim([-2500, 1600])\naxes.set_ylim([-2000, 1600])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also take a look at the different types of agents and thier labels present in all the frames.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"agents = zarr_dataset.agents\nprobabilities = agents['label_probabilities']\nlabels_indexes = np.argmax(probabilities, axis=1)\ncounts = []\nfor idx_label, label in enumerate(PERCEPTION_LABELS):\n    counts.append(np.sum(labels_indexes == idx_label))\n    \ntable = PrettyTable(field_names=['label', 'counts'])\nfor count, label in zip(counts, PERCEPTION_LABELS):\n    table.add_row([label, count])\nprint(table)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If I am not very wrong, then the above table shows the differnet labels present in all the frames of the sample data along the `label` column. And the `counts` column shows the total number of times those labeled agents are present in those frames.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## <u>Dataset Visualization for AV and Agents</u>\nIn this section, we will visulize the autonomous vehicles, thier paths, and the agents on the RGB images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"rast = build_rasterizer(cfg, dm)\ndataset = EgoDataset(cfg, zarr_dataset, rast)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`EgoDataset` will allow us to iterate over the autnomous vehicle annotations.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Visualize the Autonomous Vehicle","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data = dataset[50]\n\nim = data['image'].transpose(1, 2, 0)\nim = dataset.rasterizer.to_rgb(im)\ntarget_positions_pixels = transform_points(data['target_positions'] + data['centroid'][:2], data['world_to_image'])\ndraw_trajectory(im, target_positions_pixels, data['target_yaws'], TARGET_POINTS_COLOR)\n\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above image is 224x224 pixels as originally intended. Let's make it a bit bigger.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 12))\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's lot better now. Let's decode the above image.\n* The green box: It is the autnomous vehicle.\n* The pink line: The trajectory of the automous vehicle.\n* The blue box: An agent.\n* The yellow lines: The semantic maps.\n* But what are the orange outlined boxes: Most prbably traffic posts, but not veru sure.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Okay, now we can take look at the same image but in satellite view.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg['raster_params']['map_type'] = 'py_satellite'\nrast = build_rasterizer(cfg, dm)\ndataset = EgoDataset(cfg, zarr_dataset, rast)\ndata = dataset[50]\n\nim = data['image'].transpose(1, 2, 0)\nim = dataset.rasterizer.to_rgb(im)\ntarget_positions_pixels = transform_points(data['target_positions'] + data['centroid'][:2], data['world_to_image'])\ndraw_trajectory(im, target_positions_pixels, data['target_yaws'], TARGET_POINTS_COLOR)\n\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Again, increasing the figure size for better visualizations.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 12))\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Looks like our guess about the orange boxes being traffic posts was right**.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Visualizing an Agent\n**We can do that with the `AgentDataset` class.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = AgentDataset(cfg, zarr_dataset, rast)\ndata = dataset[0]\n\nim = data['image'].transpose(1, 2, 0)\nim = dataset.rasterizer.to_rgb(im)\ntarget_positions_pixels = transform_points(data['target_positions'] + data['centroid'][:2], data['world_to_image'])\ndraw_trajectory(im, target_positions_pixels, data['target_yaws'], TARGET_POINTS_COLOR)\n\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 12))\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This time the green box with the trajectory is an agent. **According to the docs it is a pace car and we will see this a lot in this dataset**. I am not very sure what this ***pace car is and what is it purpose***. Do let me know if you have any information.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Looking at an Entire Scene\nFinally we can take a look at the entire seen to get a better idea of everything that is happening around in the dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import display, clear_output\nimport PIL\n\ncfg['raster_params']['map_type'] = 'py_semantic'\nrast = build_rasterizer(cfg, dm)\ndataset = EgoDataset(cfg, zarr_dataset, rast)\nscene_idx = 2\nindexes = dataset.get_scene_indices(scene_idx)\nimages = []\n\nfor idx in indexes:\n    data = dataset[idx]\n    im = data['image'].transpose(1, 2, 0)\n    im = dataset.rasterizer.to_rgb(im)\n    target_positions_pixes = transform_points(data['target_positions'] + data['centroid'][:2], data['world_to_image'])\n    center_in_pixels = np.asarray(cfg['raster_params']['ego_center']) * cfg['raster_params']['raster_size']\n    draw_trajectory(im, target_positions_pixels, data['target_yaws'], TARGET_POINTS_COLOR)\n    clear_output(wait=True)\n    display(PIL.Image.fromarray(im[::-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**This is it for this notbook. I will be keep updating it if I find something new. Again, this kind of data is very new to me and most probably to many others as well. If you find that there is any wrong information anywhere in this notebook, then please use the comment section to point it out. It will help others as well.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## <u>Some Resource Lists and References</u>\n* [Dataset Formats.](https://github.com/lyft/l5kit/blob/master/data_format.md)\n* [GitHub L5kit.](https://github.com/lyft/l5kit).\n* [Agent Motion Prediction Config File.](https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_config.yaml)\n* [Visualization.](https://github.com/lyft/l5kit/tree/master/examples/visualisation)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}