{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Lyft: Comprehensive guide to start competition\n\n![](http://www.l5kit.org/_images/av.jpg)\nThe image from L5Kit official document: http://www.l5kit.org/README.html\n\nIn this kernel, I will explain how to setup develop environment & the data for this competition.
\n\nThe dataset structure is a bit complicated since the Lyft Level 5 dataset contains various kinds of information.\nLyft provided a useful library to deal with this data, we need to learn how to fully-utilize these provided tools at first!"},{"metadata":{},"cell_type":"markdown","source":"# Competition description\n\n - Official page: [https://self-driving.lyft.com/level5/prediction/](https://self-driving.lyft.com/level5/prediction/)\n\n
\n The dataset consists of 170,000 scenes capturing the environment around the autonomous vehicle. Each scene encodes the state of the vehicle’s surroundings at a given point in time.\n
\n\n
\n\n\n
\n\n
\n

The goal of this competition is to predict other car/cyclist/pedestrian (called \"agent\")'s motion.

\n\n\n\n \n

Lyft Self-Driving Employee Rides Testing from Lyft Level 5 on Vimeo.

"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from IPython.display import IFrame\nIFrame('https://player.vimeo.com/video/389096888', width=640, height=360, frameborder=\"0\", allow=\"autoplay; fullscreen\", allowfullscreen=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Environment setup\n\nFollowing instruction will work only on kaggle kernel.
\nPlease try following this step by creating new kernel (or forking this kernel) on kaggle.\n\n## Install the additional library: l5kit\nWe want to install `l5kit` library, which is provided by Lyft to handle this competition's dataset (Level5 Prediction Dataset).
\nHere I use the \"Utility script\" functionality of kaggle kernel, instead of installing the library by `pip install` command.\n\n\nClick \"File\" botton on top-left, and choose \"Add utility script\". For the pop-up search window, you need to remove \"Your Work\" filter, and search \"[philculliton/kaggle-l5kit](https://www.kaggle.com/mathurinache/kaggle-l5kit)\" on top-right of the search window.\nThen you can add the **kaggle-l5kit** utility script.\n\nIf successful, you can see \"usr/lib/kaggle-l5kit\" is added to the \"Data\" section of this kernel page on right side of the kernel.\n\nReference for utility script\n - [Feature Launch: Import scripts into notebook kernels](https://www.kaggle.com/product-feedback/91185)\n - [Import functions from Kaggle script](https://www.kaggle.com/rtatman/import-functions-from-kaggle-script)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Running this pip install code takes time, we can skip it when we attach utility script correctly!\n# !pip install -U l5kit","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import l5kit\n\nl5kit.__version__","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, we don't need to hassle with the time to run `pip install` which takes time to install the library. Just attach the utility script and the library is already setup!"},{"metadata":{},"cell_type":"markdown","source":"## Attach additional dataset source: for config files\n\nWe are using some config files when we want to load/visualize Lyft level5 dataset.
\n@jpbremer already uploaded config files as Kaggle Dataset platform: [lyft-config-files](https://www.kaggle.com/jpbremer/lyft-config-files).
\nThis is originally from official github [lyft/l5kit example page](https://github.com/lyft/l5kit/tree/master/examples).\n\nClick \"Add data\" button and press \"Search by URL\". Typing \"https://www.kaggle.com/jpbremer/lyft-config-files\" shows the dataset.
\nOnce the dataset is successfully added you can see \"lyft-config-files\" as dataset on left side bar."},{"metadata":{},"cell_type":"markdown","source":"## import"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import gc\nimport os\nfrom pathlib import Path\nimport random\nimport sys\n\nfrom tqdm.notebook import tqdm\nimport numpy as np\nimport pandas as pd\nimport scipy as sp\n\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom IPython.core.display import display, HTML\n\n# --- plotly ---\nfrom plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.express as px\nimport plotly.figure_factory as ff\nimport plotly.io as pio\npio.templates.default = \"plotly_dark\"\n\n# --- models ---\nfrom sklearn import preprocessing\nfrom sklearn.model_selection import KFold\nimport lightgbm as lgb\nimport xgboost as xgb\nimport catboost as cb\n\n# --- setup ---\npd.set_option('max_columns', 50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from l5kit.data import ChunkedDataset, LocalDataManager\nfrom l5kit.dataset import EgoDataset, AgentDataset\n\nfrom l5kit.rasterization import build_rasterizer\nfrom l5kit.configs import load_config_data\nfrom l5kit.visualization import draw_trajectory, TARGET_POINTS_COLOR\nfrom l5kit.geometry import transform_points\nfrom tqdm import tqdm\nfrom collections import Counter\nfrom l5kit.data import PERCEPTION_LABELS\nfrom prettytable import PrettyTable\n\nimport os\n\nfrom matplotlib import animation, rc\nfrom IPython.display import HTML\n\nrc('animation', html='jshtml')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# Input data files are available in the \"../input/\" directory.\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n filenames.sort()\n for filename in filenames:\n print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Basic tutorial\n\nBefore going to look the data, we still need to learn some advanced functionality to handle data.\n\nThis section basically follows [Data format](https://github.com/lyft/l5kit/blob/master/data_format.md) description page of l5kit.\n\n## numpy structured array\n\nLyft dataset uses numpy's [structured array](https://docs.scipy.org/doc/numpy/user/basics.rec.html) functionality to store various kinds of features together.
\nLet's see example to how it works"},{"metadata":{"trusted":true},"cell_type":"code","source":"my_arr = np.zeros(3, dtype=[(\"color\", (np.uint8, 3)), (\"label\", np.bool)])\n\nmy_arr[0][\"color\"] = [0, 218, 130]\nmy_arr[0][\"label\"] = True\nmy_arr[1][\"color\"] = [245, 59, 255]\nmy_arr[1][\"label\"] = True\n\nmy_arr","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"At top line we defined length=3 array. Usually each element of array consists of only integer or float, but we can define custom structured format by specifying `dtype` as list of \"fields\" which consists of name and structure.\n\nAbove example contains 2 fields. 1. 8byte uint with length 3 array, 2. single element boolean array.\n\nAs you can see, `my_arr[i][\"name\"]` will access i-th element's \"name\" field.\n\nUsually when we train neural network, we would like to access all the field of random i-th element.\nAccording to [Lyft Data Format page](https://github.com/lyft/l5kit/blob/master/data_format.md), \"Structured arrays are a great fit to group this data together in memory and on disk.\", rather than preparing \"color\" array and \"label\" array separately and access each array's i-th element, especially when the number of field glow. "},{"metadata":{},"cell_type":"markdown","source":"## zarr"},{"metadata":{},"cell_type":"markdown","source":"Zarr data format is used to store and read these numpy structured arrays from disk.
\nZarr allows us to write very large (structured) arrays to disk in n-dimensional compressed chunks.\n\nHere is a short tutorial:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import zarr\n\nz = zarr.open(\"./dataset.zarr\", mode=\"w\", shape=(500,), dtype=np.float32, chunks=(100,))\n\n# We can write to it by assigning to it. This gets persisted on disk.\nz[0:150] = np.arange(150)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we specified chunks to be of size 100, we just wrote to two separate chunks. On your filesystem in the dataset.zarr folder you will now find these two chunks. As we didn't completely fill the second chunk, those missing values will be set to the fill value (defaults to 0). The chunks are actually compressed on disk too! \n\nWe can print some info: by not doing much work at all we saved almost 75% in disk space!"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(z.info)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When we check filesystem, `dataset.zarr` directory is created and there are 2 files \"0\" and \"1\" which are chunks currently created by just assigning value to zarr array."},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls -l ./*","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Reading from a zarr array is as easy as slicing from it like you would any numpy array. The return value is an ordinary numpy array. Zarr takes care of determining which chunks to read from."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(z[::20]) # Read every 20th value","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Zarr supports StructuredArrays, the data format we use for our datasets are a set of structured arrays stored in zarr format.\n\nSome other zarr benefits are:\n\n - Safe to use in a multithreading or multiprocessing setup. Reading is entirely safe, for writing there are lock mechanisms built-in.\n - If you have a dataset that is too large to fit in memory, loading a single sample becomes my_sample = z[sample_index] and you get compression out of the box.\n - The blosc compressor is so fast that it is faster to read the compressed data and uncompress it than reading the uncompressed data from disk.\n - Zarr supports multiple backend stores, your data could also live in a zip file, or even a remote server or S3 bucket.\n - Other libraries such as xarray, Dask and TensorStore have good interoperability with Zarr.\n - The metadata (e.g. dtype, chunk size, compression type) is stored inside the zarr dataset too. If one day you decide to change your chunk size, you can still read the older datasets without changing any code."},{"metadata":{},"cell_type":"markdown","source":"See the zarr [docs](https://zarr.readthedocs.io/en/stable/) for more details."},{"metadata":{},"cell_type":"markdown","source":"# Lyft's dataset structure\n\nNow the basics learning are done! We can start looking Lyft level 5 dataset using `l5kit` library.\n\nI referenced [l5kit visualize_data.ipynb](https://github.com/lyft/l5kit/blob/master/examples/visualisation/visualise_data.ipynb) for this section.\n\n\n## Word definition\n - **\"Ego\"** is the host car which is recording/measuring the dataset.\n - **\"Agent\"** is the surronding car except \"Ego\" car.\n - **\"Frame\"** is the 1 image snapshot, where **\"Scene\"** is made of multiple frames of contious-time (video).\n\n## Class diagram\n\n\n
\n\n## Initial setup"},{"metadata":{"trusted":true},"cell_type":"code","source":"# set env variable for data\nos.environ[\"L5KIT_DATA_FOLDER\"] = \"/kaggle/input/lyft-motion-prediction-autonomous-vehicles\"\n# get config\ncfg = load_config_data(\"/kaggle/input/lyft-config-files/visualisation_config.yaml\")\nprint(cfg)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loading data\n\nHere we will only use the first dataset from the sample set.\n\nWe're building a `LocalDataManager` object. This will resolve relative paths from the config using the `L5KIT_DATA_FOLDER` env variable we have just set.\n\nHere sample.zarr data is used for visualization, please use train.zarr / validate.zarr / test.zarr for actual model training/validation/prediction.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"dm = LocalDataManager()\ndataset_path = dm.require('scenes/sample.zarr')\nzarr_dataset = ChunkedDataset(dataset_path)\nzarr_dataset.open()\nprint(zarr_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_path","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Working with the raw data\n\n`zarr_dataset` contains **scenes, frames, agents, tl_faces** attributes, which are the raw structured array data.\n\nEach data structure definition can be checked at [here](https://github.com/lyft/l5kit/blob/master/data_format.md#2020-lyft-competition-dataset-format).\n\n### scenes\n\n```\nSCENE_DTYPE = [\n (\"frame_index_interval\", np.int64, (2,)),\n (\"host\", \"\nInstead, we can remove for loop by using slice accessing. `frames[\"ego_translation\"]` is same as `frames[:][\"ego_translation\"]`, which accesses all the element's \"ego_translation\" field. By writing this, number of decompression call is reduced and code runs faster dramatically."},{"metadata":{},"cell_type":"markdown","source":"## pytorch Dataset class\n\nInstead of working with raw data, L5Kit provides PyTorch ready datasets.\nIt's much easier to use this wrapped dataset class to access data.\n\n2 dataset class is implemented.\n\n - **EgoDataset**: this dataset iterates over the AV (Autonomous Vehicle) annotations\n - **AgentDataset**: this dataset iterates over other agents annotations"},{"metadata":{"trusted":true},"cell_type":"code","source":"# 'map_type': 'py_semantic' for cfg.\nsemantic_rasterizer = build_rasterizer(cfg, dm)\nsemantic_dataset = EgoDataset(cfg, zarr_dataset, semantic_rasterizer)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Rasterizer is in charge of visualizing the data, as we will see next."},{"metadata":{},"cell_type":"markdown","source":"## Visualization example\n\nLyft l5kit also provides visualization functionalities.
\nWe will visualize data to understand what kind of information is stored in this dataset.
"},{"metadata":{},"cell_type":"markdown","source":"We can sample a data from dataset, and convert to RGB image using `rasterizer`.\n\n - image: (channel, height, width) image of a frame. This is Birds-eye-view (BEV) representation.\n - target_positions: (n_frames, 2) displacements in meters in world coordinates\n - target_yaws: (n_frames, 1)\n - centroid: (2) center position x&y.\n - world_to_image: (3, 3) 3x3 matrix, used for transform matrix.\n\nData is represented as 2.5D, positions and yaws are separated.
\ntarget_positions are represented in world coordinates and it is converted to pixel coordinates to visualize below:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_trajectory(dataset, index, title=\"target_positions movement with draw_trajectory\"):\n data = dataset[index]\n im = data[\"image\"].transpose(1, 2, 0)\n im = dataset.rasterizer.to_rgb(im)\n target_positions_pixels = transform_points(data[\"target_positions\"], data[\"raster_from_agent\"]) #change it from original notebook\n draw_trajectory(im, target_positions_pixels, rgb_color=TARGET_POINTS_COLOR, yaws=data[\"target_yaws\"]) # change it from original notebook\n plt.title(title)\n plt.imshow(im[::-1])\n plt.show()\n\nvisualize_trajectory(semantic_dataset, index=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can switch rasterizer to visualize satellite image easily!"},{"metadata":{"trusted":true},"cell_type":"code","source":"# map_type was changed from 'py_semantic' to 'py_satellite'.\ncfg[\"raster_params\"][\"map_type\"] = \"py_satellite\"\nsatellite_rasterizer = build_rasterizer(cfg, dm)\nsatellite_dataset = EgoDataset(cfg, zarr_dataset, satellite_rasterizer)\n\nvisualize_trajectory(satellite_dataset, index=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"type(satellite_rasterizer), type(semantic_rasterizer)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we visualized **EgoDataset**.\n\n**AgentDataset** can be used to visualize an agent. This dataset iterates over agents and not the AV anymore, and the first one happens to be the pace car (you will see this one around a lot in the dataset)."},{"metadata":{"trusted":true},"cell_type":"code","source":"agent_dataset = AgentDataset(cfg, zarr_dataset, satellite_rasterizer)\nvisualize_trajectory(agent_dataset, index=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can compare above 2 images, only the target car changed from host to agent."},{"metadata":{},"cell_type":"markdown","source":"
System Origin and Orientation\n\nAt this point you may have noticed that we flip the image on the Y-axis before plotting it.\n\nWhen moving from 3D to 2D we stick to a right-hand system, where the origin is in the bottom-left corner with positive x-values going right and positive y-values going up the image plane. The camera is facing down the negative z axis.\n\nHowever, both opencv and pyplot place the origin in the top-left corner with positive x going right and positive y going down in the image plane. The camera is facing down the positive z-axis.\n\nThe flip done on the resulting image is for visualisation purposes to accommodate the difference in the two coordinate frames.\n\nFurther, all our rotations are counter-clockwise for positive value of the angle.\n
\n"},{"metadata":{},"cell_type":"markdown","source":"## Scene handling\n\nBoth EgoDataset and AgentDataset provide 2 methods for getting interesting indices:\n\n - **get_frame_indices** returns the indices for a given frame. For the `EgoDataset` this matches a single observation, while more than one index could be available for the `AgentDataset`, as that given frame may contain more than one valid agent\n - **get_scene_indices** returns indices for a given scene. For both datasets, these might return more than one index"},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import display, clear_output\nimport PIL\n \ndataset = semantic_dataset\nscene_idx = 34\nindexes = dataset.get_scene_indices(scene_idx)\nimages = []\n\nfor idx in indexes:\n data = dataset[idx]\n im = data[\"image\"].transpose(1, 2, 0)\n im = dataset.rasterizer.to_rgb(im)\n target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n clear_output(wait=True)\n images.append(PIL.Image.fromarray(im[::-1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%capture\n# From https://www.kaggle.com/jpbremer/lyft-scene-visualisations by @jpbremer\ndef animate_solution(images):\n\n def animate(i):\n im.set_data(images[i])\n return (im,)\n \n fig, ax = plt.subplots()\n im = ax.imshow(images[0])\n def init():\n im.set_data(images[0])\n return (im,)\n \n return animation.FuncAnimation(fig, animate, init_func=init, frames=len(images), interval=60, blit=True)\n\nanim = animate_solution(images)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"HTML(anim.to_jshtml())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Next to go\n\nThat's all for first introduction to familialize with l5kit dataset.
\nWe have still need to know a lot to write data processing code to build a better prediction model pipeline.
\nI wrote next kernel [Lyft: Deep into the l5kit library](https://www.kaggle.com/corochann/lyft-deep-into-the-l5kit-library) for this purpose.\n\nIf you don't want quickly try training prediction model baseline, you can go to [agent motion prediction ipynb](https://github.com/lyft/l5kit/tree/master/examples/agent_motion_prediction) to start prediction.
\nThe same content is already uploaded: [a port of l5kit example](https://www.kaggle.com/hirune924/just-a-port-of-l5kit-example/data) by @hirune924.\n\n**[Update] I also wrote training tutorial kernel:**\n - [Lyft: Training with multi-mode confidence](https://www.kaggle.com/corochann/lyft-training-with-multi-mode-confidence)\n - [Lyft: Prediction with multi-mode confidence](https://www.kaggle.com/corochann/lyft-prediction-with-multi-mode-confidence)"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Further reference\n\n - Paper of this Lyft Level 5 prediction dataset: [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://arxiv.org/abs/2006.14480)\n - [jpbremer/lyft-scene-visualisations](https://www.kaggle.com/jpbremer/lyft-scene-visualisations)"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"

If this kernel helps you, please upvote to keep me motivated :)
Thanks!

"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}