{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n!cp -r ../input/lyft-toolkit/lyft-toolkit/* ./\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import os\n\n## =====================================================================================\n## This is a temporarly fix for the freezing and the cuda issues. You can add this\n## utility script instead of kaggle_l5kit until Kaggle resolve these issues.\n## \n## You will be able to train and submit your results, but not all the functionality of\n## l5kit will work properly.\n\n## More details here:\n## https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/177125\n\n## this script transports l5kit and dependencies\nos.system('pip install --target=/kaggle/working pymap3d==2.1.0')\nos.system('pip install --target=/kaggle/working protobuf==3.12.2')\nos.system('pip install --target=/kaggle/working transforms3d')\nos.system('pip install --target=/kaggle/working zarr')\nos.system('pip install --target=/kaggle/working ptable')\n\nos.system('pip install --no-dependencies --target=/kaggle/working l5kit')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"my_arr = np.zeros(3, dtype=[(\"color\", (np.uint8, 3)), (\"label\", np.bool)])\n\nprint(my_arr[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"my_arr[0][\"color\"] = [0, 218, 130]\nmy_arr[0][\"label\"] = True\nmy_arr[1][\"color\"] = [245, 59, 255]\nmy_arr[1][\"label\"] = True\nmy_arr[1][\"color\"] = [7, 6, 97]\nmy_arr[1][\"label\"] = True\n\nprint(my_arr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import zarr\ntrain = zarr.open(\"../input/lyft-motion-prediction-autonomous-vehicles/scenes/train.zarr\")\nvalidation = zarr.open(\"../input/lyft-motion-prediction-autonomous-vehicles/scenes/validate.zarr\")\ntest = zarr.open(\"../input/lyft-motion-prediction-autonomous-vehicles/scenes/test.zarr/\")\ntrain.info","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'We have {len(train.agents)} agents, {len(train.scenes)} scenes, {len(train.frames)} frames and {len(train.traffic_light_faces)} traffic light faces in train.zarr.')\nprint(f'We have {len(validation.agents)} agents, {len(validation.scenes)} scenes, {len(validation.frames)} frames and {len(validation.traffic_light_faces)} traffic light faces in validation.zarr.')\nprint(f'We have {len(test.agents)} agents, {len(test.scenes)} scenes, {len(test.frames)} frames and {len(test.traffic_light_faces)} traffic light faces in test.zarr.')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from l5kit.data import ChunkedDataset, LocalDataManager\nfrom l5kit.dataset import EgoDataset, AgentDataset\n\nfrom l5kit.rasterization import build_rasterizer\nfrom l5kit.configs import load_config_data\nfrom l5kit.visualization import draw_trajectory, TARGET_POINTS_COLOR\nfrom l5kit.geometry import transform_points\nfrom tqdm import tqdm\nfrom collections import Counter\nfrom l5kit.data import PERCEPTION_LABELS\nfrom prettytable import PrettyTable\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# set env variable for data\nos.environ[\"L5KIT_DATA_FOLDER\"] = \"/kaggle/input/lyft-motion-prediction-autonomous-vehicles\"\n# get config\ncfg = load_config_data(\"/kaggle/input/lyft-config-files/visualisation_config.yaml\")\nprint(cfg)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'current raster_param:\\n')          #Raster_size is the image size\nfor k,v in cfg[\"raster_params\"].items():\n    print(f\"{k}:{v}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nrasterize method: to create (ch, height, width) format image. \nBasically this can be used for the input of prediciton model. \nIt can have any number of channels.\n\n\nWe see that\n* StubRasterizer is just for debugging, creates all black image with specified (height, width).\n* BoxRasterizer creates Ego (host car) as green box, and Agent as blue box.\n* SatelliteRasterizer draws satellite map.\n* SemanticRasterizer draws semantic map which contains lane & crosswalk information\n* SatBoxRasterizer = SatelliteRasterizer + BoxRasterizer\n* SemBoxRasterizer = SemanticRasterizer + BoxRasterizer"},{"metadata":{"trusted":true},"cell_type":"code","source":"dm = LocalDataManager()\ndataset_path = dm.require('scenes/sample.zarr')\nzarr_dataset = ChunkedDataset(dataset_path)\nzarr_dataset.open()\nprint(zarr_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_rgb_image(dataset, index, title=\"\", ax=None):\n    \"\"\"Visualizes Rasterizer's RGB image\"\"\"\n    data = dataset[index]\n    im = data[\"image\"].transpose(1, 2, 0)\n    im = dataset.rasterizer.to_rgb(im)\n\n    if ax is None:\n        fig, ax = plt.subplots()\n    if title:\n        ax.set_title(title)\n    ax.imshow(im[::-1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Prepare all rasterizer and EgoDataset for each rasterizer\nrasterizer_dict = {}\ndataset_dict = {}\n\nrasterizer_type_list = [\"py_satellite\", \"satellite_debug\", \"py_semantic\", \"semantic_debug\", \"box_debug\", \"stub_debug\"]\n\nfor i, key in enumerate(rasterizer_type_list):\n    # print(\"key\", key)\n    cfg[\"raster_params\"][\"map_type\"] = key\n    rasterizer_dict[key] = build_rasterizer(cfg, dm)\n    dataset_dict[key] = EgoDataset(cfg, zarr_dataset, rasterizer_dict[key])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 3, figsize=(15, 10))\naxes = axes.flatten()\nfor i, key in enumerate([\"stub_debug\", \"satellite_debug\", \"semantic_debug\", \"box_debug\", \"py_satellite\", \"py_semantic\"]):\n    visualize_rgb_image(dataset_dict[key], index=0, title=f\"{key}: {type(rasterizer_dict[key]).__name__}\", ax=axes[i])\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking the code, I see that\n\n* default lane color is \"light yellow\" (255, 217, 82). code\n* green, yellow, red color on lane is to show trafic light condition. code\n* orange box represents crosswalk. code\nPlease refer below animation to verify it."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Import libraries for animation\n\"\"\"\ndef create_animate_for_indexes(dataset, indexes):\n    images = []\n    timestamps = []\n\n    for idx in indexes:\n        data = dataset[idx]\n        im = data[\"image\"].transpose(1, 2, 0)\n        im = dataset.rasterizer.to_rgb(im)\n        target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n        center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n        draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n        clear_output(wait=True)\n        images.append(PIL.Image.fromarray(im[::-1]))\n        timestamps.append(data[\"timestamp\"])\n\n    anim = animate_solution(images, timestamps)\n    return anim\n\ndef create_animate_for_scene(dataset, scene_idx):\n    indexes = dataset.get_scene_indices(scene_idx)\n    return create_animate_for_indexes(dataset, indexes)\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\ndataset = dataset_dict[\"py_semantic\"]\nscene_idx = 34\nanim = create_animate_for_scene(dataset, scene_idx)\nprint(\"scene_idx\", scene_idx)\nHTML(anim.to_jshtml())\n\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2 dataset class is implemented.\n\n* EgoDataset: this dataset iterates over the AV (Autonomous Vehicle) annotations\n* AgentDataset: this dataset iterates over other agents annotations\n\nVisualization to be added later on"},{"metadata":{},"cell_type":"markdown","source":"As written in Class diagram, both classes are instantiated by:\n\n* cfg: configuration file\n* ChunkedDataset: Internal data class which holds 4 raw data scenes, frames, agents and tl_faces (described later).\n* rasterizer: Rasterizer converts raw data into image."},{"metadata":{},"cell_type":"markdown","source":"EgoDataset consists of multiple scenes. Each scene is usually 25 sec consecutive events, consists of multiple frames.\nA frame reprsents specific time's snapshot. Snapshot is taken in 0.1 sec interval, so usually 1 scene is made of about 250 frames.\n\nBlue box represents each scene, orange box represents each frame as well as each data index.\n\nWhen we access dataset by index i, i-th frame is returned. Frames are concatenated by multiple scenes, we different i-th index points different scene. The point where the scene will change is represented by cumulative_sizes."},{"metadata":{"trusted":true},"cell_type":"code","source":"semantic_rasterizer = rasterizer_dict[\"py_semantic\"]\ndataset = dataset_dict[\"py_semantic\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# It shows the split point of each scene.\nprint(\"cumulative_sizes\", dataset.cumulative_sizes)\n\n# How's the length of each scene?\nprint(\"Each scene's length\", dataset.cumulative_sizes[1:] - dataset.cumulative_sizes[:-1])","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}