{"cells":[{"metadata":{},"cell_type":"markdown","source":"![](https://www.kaggle.com/satnam007/imagelyft?select=pipeline.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## important thigs of competition","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"This new Lyft competition is tasking us, the participants, to predict the motion of external cars, cyclists, pedestrians etc. to assist self-driving cars. This is a step ahead from last year's competition, where we were tasked with detecting three-dimensional objects, like stop signs, to teach AVs how to recognize these. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import Image\n!ls ../input/imagelyft","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### you can view images directly and can download them for better view","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"Image(\"../input/imagelyft/pipeline.png\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Image(\"../input/imagelyft/av.jpg\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!pip install --upgrade pip -q\n!pip install pymap3d==2.1.0 -q\n!pip install -U l5kit -q","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We may now import the lyft level 5 kit, and all that comes with it.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import l5kit, os\nfrom l5kit.rasterization import build_rasterizer\nfrom l5kit.configs import load_config_data\nfrom l5kit.visualization import draw_trajectory, TARGET_POINTS_COLOR\nfrom l5kit.geometry import transform_points\nfrom tqdm import tqdm\nfrom collections import Counter\nfrom l5kit.data import PERCEPTION_LABELS\nfrom prettytable import PrettyTable\n# set env variable for data\nos.environ[\"L5KIT_DATA_FOLDER\"] = \"../input/lyft-motion-prediction-autonomous-vehicles\"\n# get config\ncfg = load_config_data(\"../input/lyft-config-files/visualisation_config.yaml\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nimport numpy as np\n\nfrom l5kit.data import ChunkedDataset, LocalDataManager\nfrom l5kit.dataset import EgoDataset, AgentDataset\n\nfrom matplotlib import animation, rc\nfrom IPython.display import HTML\n\nrc('animation', html='jshtml')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, let's get a sense of the configuration data. This will include metadata pertaining to the agents, the total time, the frames-per-scene, the scene time and the frame frequency.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from l5kit.data import ChunkedDataset, LocalDataManager\nfrom l5kit.dataset import EgoDataset, AgentDataset\ndm = LocalDataManager()\ndataset_path = dm.require(cfg[\"val_data_loader\"][\"key\"])\nzarr_dataset = ChunkedDataset(dataset_path)\nzarr_dataset.open()\nprint(zarr_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can look into our current configuration for interesting fields\n- when loaded in python, the yamlfile is converted into a python dict.\n\nraster_params contains all the information related to the transformation of the 3D world onto an image plane:\n\nraster_size: the image plane size\npixel_size: how many meters correspond to a pixel\nego_center: our raster is centered around an agent, we can move the agent in the image plane with this param\nmap_type: the rasterizer to be employed. We currently support a satellite-based and a semantic-based one. We will look at the differences further down in this script","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'current raster_param:\\n')\nfor k,v in cfg[\"raster_params\"].items():\n    print(f\"{k}:{v}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# getting ccentroid to plot trajectory","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nframes = zarr_dataset.frames\ncoords = np.zeros((len(frames), 2))\nfor idx_coord, idx_data in enumerate(tqdm(range(len(frames)), desc=\"getting centroid to plot trajectory\")):\n    frame = zarr_dataset.frames[idx_data]\n    coords[idx_coord] = frame[\"ego_translation\"][:2]\n\n\nplt.scatter(coords[:, 0], coords[:, 1], marker='.')\naxes = plt.gca()\naxes.set_xlim([-2500, 1600])\naxes.set_ylim([-2500, 1600])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Another easy thing to try is to get an idea of the agents types distribution.\n\nWe can get all the agents label_probabilities and get the argmax for each raw. because .zarr files map to numpy array we can use all the traditional numpy operations and functions.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"agents = zarr_dataset.agents\nprobabilities = agents[\"label_probabilities\"]\nlabels_indexes = np.argmax(probabilities, axis=1)\ncounts = []\nfor idx_label, label in enumerate(PERCEPTION_LABELS):\n    counts.append(np.sum(labels_indexes == idx_label))\n    \ntable = PrettyTable(field_names=[\"label\", \"counts\"])\nfor count, label in zip(counts, PERCEPTION_LABELS):\n    table.add_row([label, count])\nprint(table)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, however it's time for us to look at the scenes and analyze them in depth. Theoretically, we could create a nifty little data-loader to do some heavy lifting for us.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import numpy as np\nfrom IPython.display import display, clear_output\nimport PIL\n \ncfg[\"raster_params\"][\"map_type\"] = \"py_semantic\"\nrast = build_rasterizer(cfg, dm)\ndataset = EgoDataset(cfg, zarr_dataset, rast)\nscene_idx = 2\nindexes = dataset.get_scene_indices(scene_idx)\nimages = []\n\nfor idx in indexes:\n    \n    data = dataset[idx]\n    im = data[\"image\"].transpose(1, 2, 0)\n    im = dataset.rasterizer.to_rgb(im)\n    target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n    center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n    draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n    clear_output(wait=True)\n    display(PIL.Image.fromarray(im[::-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, there's a lot of information in this one image. I'll try my best to point everything out, but do notify me if I make any errors. OK, let's get started with dissecting the image:\n+ We have an intersection of four roads over here.\n+ The green blob represents the AV's motion, and we would require to predict the movement of the AV in these traffic conditions as a sample.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I don't exactly know what other inferences we can make without more detail on this data, so let's try a satellite-format viewing of these images. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Satellite View","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data = dataset[80]\n\nim = data[\"image\"].transpose(1, 2, 0)\nim = dataset.rasterizer.to_rgb(im)\ntarget_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\ndraw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nfrom IPython.display import display, clear_output\nimport PIL\n \ncfg[\"raster_params\"][\"map_type\"] = \"py_satellite\"\nrast = build_rasterizer(cfg, dm)\ndataset = EgoDataset(cfg, zarr_dataset, rast)\nscene_idx = 2\nindexes = dataset.get_scene_indices(scene_idx)\nimages = []\n\nfor idx in indexes:\n    \n    data = dataset[idx]\n    im = data[\"image\"].transpose(1, 2, 0)\n    im = dataset.rasterizer.to_rgb(im)\n    target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n    center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n    draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n    clear_output(wait=True)\n    display(PIL.Image.fromarray(im[::-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"+ Green still represents the autonomous vehicle (AV), and blue is primarily all the other cars/vehicles/exogenous factors we need to predict for","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import display, clear_output\nfrom IPython.display import HTML\n\nimport PIL\nimport matplotlib.pyplot as plt\nfrom matplotlib import animation, rc\ndef animate_solution(images):\n\n    def animate(i):\n        im.set_data(images[i])\n \n    fig, ax = plt.subplots()\n    im = ax.imshow(images[0])\n    \n    return animation.FuncAnimation(fig, animate, frames=len(images), interval=60)\ncfg[\"raster_params\"][\"map_type\"] = \"py_satellite\"\nrast = build_rasterizer(cfg, dm)\ndataset = EgoDataset(cfg, zarr_dataset, rast)\nscene_idx = 34\nindexes = dataset.get_scene_indices(scene_idx)\nimages = []\n\nfor idx in indexes:\n    \n    data = dataset[idx]\n    im = data[\"image\"].transpose(1, 2, 0)\n    im = dataset.rasterizer.to_rgb(im)\n    target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n    center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n    draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n    clear_output(wait=True)\n    images.append(PIL.Image.fromarray(im[::-1]))\nanim = animate_solution(images)\nHTML(anim.to_jshtml())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So this is a demonstration of the movement of the other vehicles and (in relation to the movement and placement of the other vehicles) the movement of the AV. The AV is currently taking only a straight path in its motion, and a straight path seems logical with the movement and placement of other vehicles.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# What if I want to visualise an agent?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = AgentDataset(cfg, zarr_dataset, rast)\ndata = dataset[0]\n\nim = data[\"image\"].transpose(1, 2, 0)\nim = dataset.rasterizer.to_rgb(im)\ntarget_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\ndraw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# What if I want to visualise the Autonomous Vehicle (AV)?¶\n## along with its tracjectory","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg[\"raster_params\"][\"map_type\"] = \"py_satellite\"\nrast = build_rasterizer(cfg, dm)\ndataset = EgoDataset(cfg, zarr_dataset, rast)\ndata = dataset[50]\n\nim = data[\"image\"].transpose(1, 2, 0)\nim = dataset.rasterizer.to_rgb(im)\ntarget_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\ndraw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n\nplt.imshow(im[::-1])\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import PIL\n \ncfg[\"raster_params\"][\"map_type\"] = \"py_semantic\"\nrast = build_rasterizer(cfg, dm)\ndataset = EgoDataset(cfg, zarr_dataset, rast)\nscene_idx = 2\nindexes = dataset.get_scene_indices(scene_idx)\nimages = []\n\nfor idx in indexes:\n    \n    data = dataset[idx]\n    im = data[\"image\"].transpose(1, 2, 0)\n    im = dataset.rasterizer.to_rgb(im)\n    target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n    center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n    draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n    clear_output(wait=True)\n    display(PIL.Image.fromarray(im[::-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Source: https://github.com/lyft/l5kit/blob/master/examples/visualisation/visualise_data.ipynb\nhttps://github.com/lyft/l5kit\n\n**WORK IN PROGRESS - MORE TO COME.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}