{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Note: As part of this notebook, I'll try to cover the data exploration aspects which are not covered by existing EDA notebooks and were confusing to me.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport os\nimport torch\nimport random\nimport cv2\n\nfrom torch import nn, optim\nfrom torch.utils.data import DataLoader\nfrom torchvision.models.resnet import resnet18\nfrom tqdm import tqdm\nfrom typing import Dict\nfrom typing import Tuple\n\nimport matplotlib.pyplot as plt\n\n# Add this notebook output as utility script instead of pip installing it:\n# https://www.kaggle.com/philculliton/kaggle-l5kit. Search this by \"philculliton/kaggle-l5kit\".\nfrom l5kit.data import LocalDataManager, ChunkedDataset\nfrom l5kit.dataset import AgentDataset, EgoDataset\nfrom l5kit.evaluation import write_pred_csv\nfrom l5kit.rasterization import build_rasterizer\n\n# Seed everything\ntorch.manual_seed(28)\ntorch.cuda.manual_seed(28)\nnp.random.seed(28)\nrandom.seed(28)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BASE_DIR = '/kaggle/input/lyft-motion-prediction-autonomous-vehicles'\nos.environ['L5KIT_DATA_FOLDER'] = BASE_DIR\n\nconfig = {\n    'format_version': 4,\n    'model_params': {\n        'model_architecture': 'resnet50',\n        'history_num_frames': 10,\n        'history_step_size': 1,\n        'history_delta_time': 0.1,\n        'future_num_frames': 50,\n        'future_step_size': 1,\n        'future_delta_time': 0.1\n    },\n    \n    'raster_params': {\n        'raster_size': [224, 224],\n        'pixel_size': [0.5, 0.5],\n        'ego_center': [0.25, 0.5],\n        'map_type': 'py_semantic',\n        'satellite_map_key': 'aerial_map/aerial_map.png',\n        'semantic_map_key': 'semantic_map/semantic_map.pb',\n        'dataset_meta_key': 'meta.json',\n        'filter_agents_threshold': 0.5\n    },\n    \n    'train_data_loader': {\n        'key': 'scenes/sample.zarr',\n        'batch_size': 20,\n        'shuffle': False,\n        'num_workers': 0\n    },\n    \n    'train_params': {\n        'max_num_steps': 100,\n        'checkpoint_every_n_steps': 5000\n    }\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Initialize local data manager\ndata_manager = LocalDataManager()\n\ntrain_config = config['train_data_loader']\n\n# Train dataset/dataloader\ntrain_zarr = ChunkedDataset(data_manager.require(train_config['key'])).open()\n\n\ndef load_dataset():\n    # Build Rasterizer\n    rasterizer = build_rasterizer(config, data_manager)\n    \n    train_dataset = AgentDataset(config, train_zarr, rasterizer)\n    return train_dataset[100]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Understanding the data we have for training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data = load_dataset()\n# batch = next(iter(train_dataloader))\nprint('List of available features:\\n\\n{}'.format('\\n'.join(data.keys())))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plotting all the image channels separately","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"f, ax = plt.subplots(5, 5, figsize=(20, 20))\nax = ax.flatten()\n\nfor i in range(25):\n    ax[i].imshow(data['image'][i], cmap='Greys')\n    ax[i].get_xaxis().set_visible(False)\n    ax[i].get_yaxis().set_visible(False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here are some important observations about the above data:\n\n- We have 25 channels per frame.\n- The first 11 channels(plots) are location of agents in the given frame. The next 11 channels(plots) are location of ego in the same frame. The last 3 channels(plots) are for semantic map (int RGB format). *Note: Credits of this explanation goes to @pestipeti.*\n- Pixel size in \"raster_params\" determines the raster's spatial resolution in meters per pixel. In the above plots, we have selected 0.5 meters per pixels. Actually this will be a hyperparameter. Below is the plot with pixel_size [0.25, 0.25], which would be zoomed version on the above plots.\n- Ego center is from 0 to 1 per axis. We have selected ego position at 25% on X-axis and 50% on Y-axis. [0.5,0.5] would show the ego centered in the image.\n- Map type would be either semantic or satellite.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Zoomed in images using pixel_size parameter","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"config['raster_params']['pixel_size'] = [0.3, 0.3]\ndata = load_dataset()\n\nf, ax = plt.subplots(5, 5, figsize=(20, 20))\nax = ax.flatten()\n\nfor i in range(25):\n    ax[i].imshow(data['image'][i], cmap='Greys')\n    ax[i].get_xaxis().set_visible(False)\n    ax[i].get_yaxis().set_visible(False)\n\n# Revert back the pixel_size\nconfig['raster_params']['pixel_size'] = [0.5, 0.5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note that, zooming in removed some of the agent annotations from the frame. So we'll have to choose this wisely.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Changing the Raster size","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sizes = [[150, 150], [224, 224], [250, 250], [350, 350], [450, 450], [500, 500]]\n\nf, ax = plt.subplots(2, 3, figsize=(20, 12))\nax = ax.flatten()\n\nfor i in range(6):\n    config['raster_params']['raster_size'] = sizes[i]\n    data = load_dataset()\n    \n    ax[i].imshow(data['image'][-3:].transpose(1, 2, 0), cmap='Greys')\n    ax[i].get_xaxis().set_visible(False)\n    ax[i].get_yaxis().set_visible(False)\n\n# Revert back the pixel_size\nconfig['raster_params']['raster_size'] = [224, 224]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see, increasing the raster size increases the region what models get to see. So that is also an important hyperparameter.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Changing the position of Ego","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We can also adjust the center of the ego using \"ego_center\" parameter of rasterization config. Let's center the ego using [0.5, 0.5] as \"ego_center\".","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"config['raster_params']['ego_center'] = [0.5, 0.5]\ndata = load_dataset()\n\nf, ax = plt.subplots(5, 5, figsize=(20, 20))\nax = ax.flatten()\n\nfor i in range(25):\n    ax[i].imshow(data['image'][i], cmap='Greys')\n\n# Revert back the ego_center\nconfig['raster_params']['ego_center'] = [0.25, 0.5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, the ego is in the center of the each channel image.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Visualizing the traffic lights on map","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(8, 8))\nplt.imshow(data['image'][-3:].transpose(1, 2, 0))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Traffic light statuses are annotated using lane coloring. There will be 3 colored lanes (Red, Green, Yellow) on lanes of semantic maps. Refer this: https://github.com/lyft/l5kit/blob/master/l5kit/l5kit/rasterization/semantic_rasterizer.py#L186-L196","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\"target_positions\" are labels on which we are training and predicting. They are target positions of agents in the given frame, for next 50 frames.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}