{"cells":[{"metadata":{},"cell_type":"markdown","source":"#             ***Understanding the problem***\n\nAutonomous vehicles (AVs) are expected to dramatically redefine the future of transportation. However, there are still significant engineering challenges to be solved before one can fully realize the benefits of self-driving cars. One such challenge is building models that reliably predict the movement of traffic agents around the AV, such as cars, cyclists, and pedestrians.\n\nThe ridesharing company Lyft started Level 5 to take on the self-driving challenge and build a full self-driving system. Their previous competition tasked participants with identifying 3D objects, an important step prior to detecting their movement. Now, they’re challenging to predict the motion of these traffic agents.\n\nSo we are trying to train motion prediction models with the largest collection of prediction data released to date. The dataset includes the logs of over 1,000 hours of movement of various traffic agents—such as cars, cyclists, and pedestrians—that is captured by autonomous fleet encountered on Palo Alto routes.\n\nThis articles gives you more understanding the problem\n\nhttps://medium.com/lyftlevel5/fueling-self-driving-research-with-level-5s-open-prediction-dataset-f0175e2b0cf8\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## The Road Map to Autonomus Driving is","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from IPython.display import Image\nimport os\n\n\nImage(\"../input/images-lyft/01.JPG\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Problem we are trying to solving","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"Image(\"../input/images-lyft/02.JPG\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Format\n\n* The dataset is provided in zarr format. The zarr files are flat, compact, and highly performant for loading.\n\n* The dataset consists of frames and agent states. A frame is a snapshot in time which consists of ego pose, time, and multiple agent states. Each agent state describes the position, orientation, bounds, and type.\n   \n \nDocumentation of the library created by Level 5 to read the dataset\n\n1. http://www.l5kit.org/\n2. https://github.com/lyft/l5kit/blob/master/examples/visualisation/visualise_data.ipynb","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"DIR_PATH = '../input/lyft-motion-prediction-autonomous-vehicles/'\nos.listdir(DIR_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir('../input/lyft-motion-prediction-autonomous-vehicles/scenes')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Importing required libraries ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# !pip install --upgrade pip\n# !pip install pymap3d==2.1.0\n# !pip install -U l5kit","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nimport numpy as np\n\nfrom l5kit.data import ChunkedDataset, LocalDataManager\nfrom l5kit.dataset import EgoDataset, AgentDataset\n\nfrom l5kit.rasterization import build_rasterizer\nfrom l5kit.configs import load_config_data\nfrom l5kit.visualization import draw_trajectory, TARGET_POINTS_COLOR\nfrom l5kit.geometry import transform_points\nfrom tqdm import tqdm\nfrom collections import Counter\nfrom l5kit.data import PERCEPTION_LABELS\nfrom prettytable import PrettyTable","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"By setting the L5KIT_DATA_FOLDER variable, we can point the script to the folder where the data lies","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# set env variable for data\nimport os\nos.environ[\"L5KIT_DATA_FOLDER\"] = \"../input/lyft-motion-prediction-autonomous-vehicles\"\n\n# reading the data\nzarr_dataset = ChunkedDataset('../input/lyft-motion-prediction-autonomous-vehicles/scenes/train.zarr')\nzarr_dataset.open()\nprint(zarr_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":".zarr files support most of the traditional numpy array operations. In the following cell we iterate over the frames to get a scatter plot of the AV locations to understand the distributions of various agent type distribution.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg={\n    \n    'model_params':{\n        'model_architecture': \"resnet50\",\n        'history_num_frames': 0,\n        'history_step_size': 1,\n        'history_delta_time': 0.1,\n        'future_num_frames': 50,\n        'future_step_size': 1,\n        'future_delta_time': 0.1\n        \n    },\n\n    \n    \n    'raster_params': {\n        'raster_size': [224, 224],\n        'pixel_size': [0.5, 0.5],\n        'ego_center': [0.25, 0.5],\n        'map_type': 'py_semantic',\n        'satellite_map_key': 'aerial_map/aerial_map.png',\n        'semantic_map_key': 'semantic_map/semantic_map.pb',\n        'dataset_meta_key': 'meta.json',\n        'filter_agents_threshold': 0.5\n    },\n    \n    'test_data_loader': {\n        'key': 'scenes/train.zarr',\n        'batch_size': 8,\n        'shuffle': False,\n        'num_workers': 4\n    }\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dm = LocalDataManager(None)\nrast = build_rasterizer(cfg, dm)\ndataset = EgoDataset(cfg, zarr_dataset, rast)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualize the Autonomous vehicles","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data = dataset[1]\n\nim = data[\"image\"].transpose(1, 2, 0)\nim = dataset.rasterizer.to_rgb(im)\ntarget_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\ndraw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Satilite Imagary","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg[\"raster_params\"][\"map_type\"] = \"py_satellite\"\nrast = build_rasterizer(cfg, dm)\ndataset = EgoDataset(cfg, zarr_dataset, rast)\ndata = dataset[1]\n\nim = data[\"image\"].transpose(1, 2, 0)\nim = dataset.rasterizer.to_rgb(im)\ntarget_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\ndraw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualise the agent","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = AgentDataset(cfg, zarr_dataset, rast)\ndata = dataset[1]\n\nim = data[\"image\"].transpose(1, 2, 0)\nim = dataset.rasterizer.to_rgb(im)\ntarget_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\ndraw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Entire picture","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import display, clear_output\nimport PIL\n \ncfg[\"raster_params\"][\"map_type\"] = \"py_semantic\"\nrast = build_rasterizer(cfg, dm)\ndataset = EgoDataset(cfg, zarr_dataset, rast)\nscene_idx = 2\nindexes = dataset.get_scene_indices(scene_idx)\nimages = []\n\nfor idx in indexes:\n    \n    data = dataset[idx]\n    im = data[\"image\"].transpose(1, 2, 0)\n    im = dataset.rasterizer.to_rgb(im)\n    target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n    center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n    draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n    clear_output(wait=True)\n    display(PIL.Image.fromarray(im[::-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build model for motion prediction\n\nWORK IN PROGRESS","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\n### <span style=\"color:red\">*If you find this kernel useful, Please consider Upvoting it , it motivates me to write more content.* </span>\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}