{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Lyft basics: fetch the data, visualize it\n\nSince I need to get acquainted with the data myself, I thought sharing this notebook for other people who are starting now the competition would be good.\n\nAs I always do for this kind of notebooks, I am handling the imports cell by cell, so if you need just a part of what I am describing here it will be easier for you to figure out which imports you need (at least, this is my hope). I make an exception for the most common imports:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot  as plt # data visualization","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First of all, let's pip install l5kit (Level 5 Kit). For this notebook I am relying heavily on their [documentation](https://github.com/lyft/l5kit).","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!pip install --upgrade pip\n!pip install pymap3d==2.1.0\n!pip install -U l5kit","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"At this point we need a yaml file to configure the visualization. The one from [l5kit github](https://github.com/lyft/l5kit) will do just fine. If you aren't just copy this notebook, there is already a [dataset](https://www.kaggle.com/jpbremer/lyft-config-files) you can add to your notebook to have the files (or you can download your own copy and add it, but kaggle will suggest you use the existing dataset to avoid having too many copies of the same thing).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nfrom l5kit.configs import load_config_data\n\n# set env variable for data\nos.environ[\"L5KIT_DATA_FOLDER\"] = \"../input/lyft-motion-prediction-autonomous-vehicles\"\n# get config\ncfg = load_config_data(\"../input/lyft-config-files/visualisation_config.yaml\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\\- when loaded in python, the `yaml` file is converted into a python `dict`. \n\n`raster_params` contains all the information related to the transformation of the 3D world onto an image plane. Let's look into it:\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'current raster_param:\\n')\nfor k,v in cfg[\"raster_params\"].items():\n    print(f\"{k}:{v}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"  - `raster_size`: the image plane size\n  - `pixel_size`: how many meters correspond to a pixel\n  - `ego_center`: the raster is centered around an agent, this parameter moves the agent in the image\n  - `map_type`: the rasterizer to be employed. The ones currently supported are a satellite-based and a semantic-based one\n\nTo read the data, we simply do:","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from l5kit.data import ChunkedDataset, LocalDataManager\n\ndm = LocalDataManager()\ndataset_path = dm.require(cfg[\"val_data_loader\"][\"key\"])\nzarr_dataset = ChunkedDataset(dataset_path)\nzarr_dataset.open()\nprint(zarr_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This example shows how `.zarr` files support most of the traditional numpy array operations. \n\nAs an example, here we iterate over the frames to get a scatter plot of the AV locations:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from tqdm import tqdm\n\nframes = zarr_dataset.frames\ncoords = np.zeros((len(frames), 2))\nfor idx_coord, idx_data in enumerate(tqdm(range(len(frames)), desc=\"getting centroid to plot trajectory\")):\n    frame = zarr_dataset.frames[idx_data]\n    coords[idx_coord] = frame[\"ego_translation\"][:2]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.scatter(coords[:, 0], coords[:, 1], marker='.')\naxes = plt.gca()\naxes.set_xlim([-1500, 1600])\naxes.set_ylim([-2500, 1600])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's now move to ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from l5kit.dataset import (EgoDataset,      # iterates over the AV annotations\n                           AgentDataset)    #iterates over other agents annotations\n\nfrom l5kit.rasterization import build_rasterizer\n\nrast = build_rasterizer(cfg, dm)\ndataset = EgoDataset(cfg, zarr_dataset, rast)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualising the Autonomous Vehicle (AV)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from l5kit.geometry import transform_points\n\nfrom l5kit.visualization import (draw_trajectory,       # draws 2D trajectories from coordinates and yaws offset on an image\n                                 TARGET_POINTS_COLOR)\n\ndata = dataset[50]\n\nim = data[\"image\"].transpose(1, 2, 0)\nim = dataset.rasterizer.to_rgb(im)\ntarget_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\ndraw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualising an agent","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = AgentDataset(cfg, zarr_dataset, rast)\ndata = dataset[50]\n\nim = data[\"image\"].transpose(1, 2, 0)\nim = dataset.rasterizer.to_rgb(im)\ntarget_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\ndraw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualizing a whole scene","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import display, clear_output\nimport PIL\n \ncfg[\"raster_params\"][\"map_type\"] = \"py_satellite\"   # This can be changed to \"py_semantic\" to have a view similar to the ones before\nrast = build_rasterizer(cfg, dm)\ndataset = EgoDataset(cfg, zarr_dataset, rast)\nscene_idx = 50\nindexes = dataset.get_scene_indices(scene_idx)\nimages = []\n\nfor idx in indexes:\n    \n    data = dataset[idx]\n    im = data[\"image\"].transpose(1, 2, 0)\n    im = dataset.rasterizer.to_rgb(im)\n    target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n    center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n    draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n    clear_output(wait=True)\n    display(PIL.Image.fromarray(im[::-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Seeing the scene in movement\n\nFor completion sake, I will now take some inspiration/code from this other [notebook](https://www.kaggle.com/nxrprime/lyft-understanding-the-data-and-eda) to animate what we just saw","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from matplotlib import animation, rc\n\ndef animate_solution(images):\n\n    def animate(i):\n        im.set_data(images[i])\n \n    fig, ax = plt.subplots()\n    im = ax.imshow(images[0])\n    \n    return animation.FuncAnimation(fig, animate, frames=len(images), interval=60)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import HTML\n\nscene_idx = 50\nindexes = dataset.get_scene_indices(scene_idx)\n\nfor idx in indexes:\n    \n    data = dataset[idx]\n    im = data[\"image\"].transpose(1, 2, 0)\n    im = dataset.rasterizer.to_rgb(im)\n    target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n    center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n    draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n    clear_output(wait=True)\n    images.append(PIL.Image.fromarray(im[::-1]))\n    \nanim = animate_solution(images)\nHTML(anim.to_jshtml())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Work in progress - Will continue","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}