{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Lyft: Complete train and prediction pipeline (update for l5kit 1.1.0)\n\n![](http://www.l5kit.org/_images/av.jpg)\nThe image from L5Kit official document: http://www.l5kit.org/README.html\n\n#### This notebook is updated to be compatible with the new lyft environment, see the discussion [We did it all wrong](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/186492) and [l5kit 1.1.0 release](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/187825).\n\nFor a high level overview, check out this article [how to build a motion prediction model for autonomous vehicles](https://medium.com/lyftlevel5/how-to-build-a-motion-prediction-model-for-autonomous-vehicles-29f7f81f1580).\n\nIn this notebook I present an **end-to-end** train and prediction pipeline to predict vehicle motions with a pretrained model included. Unfortunately because of Kaggle memory constraint we can only either run the train part or the prediction part by toggling the parameters in the config part. \n\nSome of the code here are taken from the [tutorial notebook](https://github.com/lyft/l5kit/tree/master/examples/agent_motion_prediction) and the following kernels:\n\n- [Lyft: Training with multi-mode confidence](https://www.kaggle.com/corochann/lyft-training-with-multi-mode-confidence)\n- [Lyft: Prediction with multi-mode confidence](https://www.kaggle.com/corochann/lyft-prediction-with-multi-mode-confidence)\n\nwhich is part of a wonderful series of introductory notebooks by [corochann](https://www.kaggle.com/corochann)\n\n - [Lyft: Comprehensive guide to start competition](https://www.kaggle.com/corochann/lyft-comprehensive-guide-to-start-competition)\n - [Lyft: Deep into the l5kit library](https://www.kaggle.com/corochann/lyft-deep-into-the-l5kit-library)\n - [Save your time, submit without kernel inference](https://www.kaggle.com/corochann/save-your-time-submit-without-kernel-inference)\n - [Lyft: pytorch implementation of evaluation metric](https://www.kaggle.com/corochann/lyft-pytorch-implementation-of-evaluation-metric)\n - [Lyft: Training with multi-mode confidence](https://www.kaggle.com/corochann/lyft-training-with-multi-mode-confidence)\n - [Lyft: Prediction with multi-mode confidence](https://www.kaggle.com/corochann/lyft-prediction-with-multi-mode-confidence)\n \n**Note:** This notebook aims to create the best single possible, i.e. without any ensemeble, which should only be done near the end of the competition. Since it is still early in the competition, this notebook is still a baseline, any suggestion to improve is appreciated."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# Environment setup\n\n - Please add [pestipeti/lyft-l5kit-unofficial-fix](https://www.kaggle.com/pestipeti/lyft-l5kit-unofficial-fix) as utility script.\n - Official utility script \"[philculliton/kaggle-l5kit](https://www.kaggle.com/mathurinache/kaggle-l5kit)\" does not work with pytorch GPU.\n\nClick \"File\" botton on top-left, and choose \"Add utility script\". For the pop-up search window, you need to remove \"Your Work\" filter, and search [pestipeti/lyft-l5kit-unofficial-fix](https://www.kaggle.com/pestipeti/lyft-l5kit-unofficial-fix) on top-right of the search window. Then you can add the kaggle-l5kit utility script. It is much faster to do this rather than !pip install l5kit every time you run the notebook. \n\nIf successful, you can see \"usr/lib/lyft-l5kit-unofficial-fix\" is added to the \"Data\" section of this kernel page on right side of the kernel.\n\n- Also please add [pretrained baseline model](https://www.kaggle.com/huanvo/lyft-pretrained-model-hv)\n\nClick on the button \"Add data\" in the \"Data\" section and search for lyft-pretrained-model-hv. If you find the model useful, please upvote it as well. "},{"metadata":{"trusted":true},"cell_type":"code","source":"from typing import Dict\n\nfrom tempfile import gettempdir\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport torch\nfrom torch import nn, optim\nfrom torch.utils.data import DataLoader\nfrom torchvision.models.resnet import resnet50, resnet18, resnet34, resnet101\nfrom tqdm import tqdm\n\nimport l5kit\nfrom l5kit.configs import load_config_data\nfrom l5kit.data import LocalDataManager, ChunkedDataset\nfrom l5kit.dataset import AgentDataset, EgoDataset\nfrom l5kit.rasterization import build_rasterizer\nfrom l5kit.evaluation import write_pred_csv, compute_metrics_csv, read_gt_csv, create_chopped_dataset\nfrom l5kit.evaluation.chop_dataset import MIN_FUTURE_STEPS\nfrom l5kit.evaluation.metrics import neg_multi_log_likelihood, time_displace\nfrom l5kit.geometry import transform_points\nfrom l5kit.visualization import PREDICTED_POINTS_COLOR, TARGET_POINTS_COLOR, draw_trajectory\nfrom prettytable import PrettyTable\nfrom pathlib import Path\n\nimport matplotlib.pyplot as plt\n\nimport os\nimport random\nimport time\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"l5kit.__version__","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def set_seed(seed):\n random.seed(seed)\n np.random.seed(seed)\n os.environ[\"PYTHONHASHSEED\"] = str(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n \nset_seed(42)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Configs"},{"metadata":{"trusted":true},"cell_type":"code","source":"# --- Lyft configs ---\ncfg = {\n 'format_version': 4,\n 'data_path': \"/kaggle/input/lyft-motion-prediction-autonomous-vehicles\",\n 'model_params': {\n 'model_architecture': 'resnet34',\n 'history_num_frames': 10,\n 'history_step_size': 1,\n 'history_delta_time': 0.1,\n 'future_num_frames': 50,\n 'future_step_size': 1,\n 'future_delta_time': 0.1,\n 'model_name': \"model_resnet34_output\",\n 'lr': 1e-3,\n 'weight_path': \"/kaggle/input/lyft-pretrained-model-hv/model_multi_update_lyft_public.pth\",\n 'train': False,\n 'predict': True\n },\n\n 'raster_params': {\n 'raster_size': [224, 224],\n 'pixel_size': [0.5, 0.5],\n 'ego_center': [0.25, 0.5],\n 'map_type': 'py_semantic',\n 'satellite_map_key': 'aerial_map/aerial_map.png',\n 'semantic_map_key': 'semantic_map/semantic_map.pb',\n 'dataset_meta_key': 'meta.json',\n 'filter_agents_threshold': 0.5\n },\n\n 'train_data_loader': {\n 'key': 'scenes/train.zarr',\n 'batch_size': 16,\n 'shuffle': True,\n 'num_workers': 4\n },\n \n 'test_data_loader': {\n 'key': 'scenes/test.zarr',\n 'batch_size': 32,\n 'shuffle': False,\n 'num_workers': 4\n },\n\n 'train_params': {\n 'max_num_steps': 101,\n 'checkpoint_every_n_steps': 20,\n }\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Couple of things to note:\n\n - **model_architecture:** you can put 'resnet18', 'resnet34' or 'resnet50'. For the pretrained model we use resnet18 so we need to use 'resnet18' in the config.\n - **weight_path:** path to the pretrained model. If you don't have a pretrained model and want to train from scratch, put **weight_path** = False. \n - **model_name:** the name of the model that will be saved as output, this is only when **train**= True.\n - **train:** True if you want to continue to train the model. Unfortunately due to Kaggle memory constraint if **train**=True then you should put **predict** = False.\n - **predict:** True if you want to predict and submit to Kaggle. Unfortunately due to Kaggle memory constraint if you want to predict then you need to put **train** = False.\n - **lr:** learning rate of the model, feel free to change as you see fit. In the future I also plan to implement learning rate decay. \n - **raster_size:** specify the size of the image, the default is [224,224]. Increase **raster_size** can improve the score. However the training time will be significantly longer. \n - **batch_size:** number of inputs for one forward pass, again one of the parameters to tune. \n - **max_num_steps:** the number of iterations to train, i.e. number of epochs.\n - **checkpoint_every_n_steps:** the model will be saved at every n steps, again change this number as to how you want to keep track of the model."},{"metadata":{},"cell_type":"markdown","source":"## Load the train and test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# set env variable for data\nDIR_INPUT = cfg[\"data_path\"]\nos.environ[\"L5KIT_DATA_FOLDER\"] = DIR_INPUT\ndm = LocalDataManager(None)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ===== INIT TRAIN DATASET============================================================\ntrain_cfg = cfg[\"train_data_loader\"]\nrasterizer = build_rasterizer(cfg, dm)\ntrain_zarr = ChunkedDataset(dm.require(train_cfg[\"key\"])).open()\ntrain_dataset = AgentDataset(cfg, train_zarr, rasterizer)\ntrain_dataloader = DataLoader(train_dataset, shuffle=train_cfg[\"shuffle\"], batch_size=train_cfg[\"batch_size\"], \n num_workers=train_cfg[\"num_workers\"])\nprint(\"==================================TRAIN DATA==================================\")\nprint(train_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#====== INIT TEST DATASET=============================================================\ntest_cfg = cfg[\"test_data_loader\"]\nrasterizer = build_rasterizer(cfg, dm)\ntest_zarr = ChunkedDataset(dm.require(test_cfg[\"key\"])).open()\ntest_mask = np.load(f\"{DIR_INPUT}/scenes/mask.npz\")[\"arr_0\"]\ntest_dataset = AgentDataset(cfg, test_zarr, rasterizer, agents_mask=test_mask)\ntest_dataloader = DataLoader(test_dataset,shuffle=test_cfg[\"shuffle\"],batch_size=test_cfg[\"batch_size\"],\n num_workers=test_cfg[\"num_workers\"])\nprint(\"==================================TEST DATA==================================\")\nprint(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Simple visualization"},{"metadata":{},"cell_type":"markdown","source":"Let us visualize how an input to the model looks like."},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_trajectory(dataset, index, title=\"target_positions movement with draw_trajectory\"):\n data = dataset[index]\n im = data[\"image\"].transpose(1, 2, 0)\n im = dataset.rasterizer.to_rgb(im)\n target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n draw_trajectory(im, target_positions_pixels, TARGET_POINTS_COLOR, radius=1, yaws=data[\"target_yaws\"])\n\n plt.title(title)\n plt.imshow(im[::-1])\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (8,6))\nvisualize_trajectory(train_dataset, index=90)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loss function"},{"metadata":{},"cell_type":"markdown","source":"For this competition it is important to use the correct loss function when train the model. Our goal is to predict three possible paths together with the confidence score, so we need to use the loss function that takes that into account, simply using RMSE will not lead to an accurate model. More information about the loss function can be found here [negative log likelihood](https://github.com/lyft/l5kit/blob/master/competition.md)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# --- Function utils ---\n# Original code from https://github.com/lyft/l5kit/blob/20ab033c01610d711c3d36e1963ecec86e8b85b6/l5kit/l5kit/evaluation/metrics.py\nimport numpy as np\n\nimport torch\nfrom torch import Tensor\n\n\ndef pytorch_neg_multi_log_likelihood_batch(\n gt: Tensor, pred: Tensor, confidences: Tensor, avails: Tensor\n) -> Tensor:\n \"\"\"\n Compute a negative log-likelihood for the multi-modal scenario.\n log-sum-exp trick is used here to avoid underflow and overflow, For more information about it see:\n https://en.wikipedia.org/wiki/LogSumExp#log-sum-exp_trick_for_log-domain_calculations\n https://timvieira.github.io/blog/post/2014/02/11/exp-normalize-trick/\n https://leimao.github.io/blog/LogSumExp/\n Args:\n gt (Tensor): array of shape (bs)x(time)x(2D coords)\n pred (Tensor): array of shape (bs)x(modes)x(time)x(2D coords)\n confidences (Tensor): array of shape (bs)x(modes) with a confidence for each mode in each sample\n avails (Tensor): array of shape (bs)x(time) with the availability for each gt timestep\n Returns:\n Tensor: negative log-likelihood for this example, a single float number\n \"\"\"\n assert len(pred.shape) == 4, f\"expected 3D (MxTxC) array for pred, got {pred.shape}\"\n batch_size, num_modes, future_len, num_coords = pred.shape\n\n assert gt.shape == (batch_size, future_len, num_coords), f\"expected 2D (Time x Coords) array for gt, got {gt.shape}\"\n assert confidences.shape == (batch_size, num_modes), f\"expected 1D (Modes) array for gt, got {confidences.shape}\"\n assert torch.allclose(torch.sum(confidences, dim=1), confidences.new_ones((batch_size,))), \"confidences should sum to 1\"\n assert avails.shape == (batch_size, future_len), f\"expected 1D (Time) array for gt, got {avails.shape}\"\n # assert all data are valid\n assert torch.isfinite(pred).all(), \"invalid value found in pred\"\n assert torch.isfinite(gt).all(), \"invalid value found in gt\"\n assert torch.isfinite(confidences).all(), \"invalid value found in confidences\"\n assert torch.isfinite(avails).all(), \"invalid value found in avails\"\n\n # convert to (batch_size, num_modes, future_len, num_coords)\n gt = torch.unsqueeze(gt, 1) # add modes\n avails = avails[:, None, :, None] # add modes and cords\n\n # error (batch_size, num_modes, future_len)\n error = torch.sum(((gt - pred) * avails) ** 2, dim=-1) # reduce coords and use availability\n\n with np.errstate(divide=\"ignore\"): # when confidence is 0 log goes to -inf, but we're fine with it\n # error (batch_size, num_modes)\n error = torch.log(confidences) - 0.5 * torch.sum(error, dim=-1) # reduce time\n\n # use max aggregator on modes for numerical stability\n # error (batch_size, num_modes)\n max_value, _ = error.max(dim=1, keepdim=True) # error are negative at this point, so max() gives the minimum one\n error = -torch.log(torch.sum(torch.exp(error - max_value), dim=-1, keepdim=True)) - max_value # reduce modes\n # print(\"error\", error)\n return torch.mean(error)\n\n\ndef pytorch_neg_multi_log_likelihood_single(\n gt: Tensor, pred: Tensor, avails: Tensor\n) -> Tensor:\n \"\"\"\n\n Args:\n gt (Tensor): array of shape (bs)x(time)x(2D coords)\n pred (Tensor): array of shape (bs)x(time)x(2D coords)\n avails (Tensor): array of shape (bs)x(time) with the availability for each gt timestep\n Returns:\n Tensor: negative log-likelihood for this example, a single float number\n \"\"\"\n # pred (bs)x(time)x(2D coords) --> (bs)x(mode=1)x(time)x(2D coords)\n # create confidence (bs)x(mode=1)\n batch_size, future_len, num_coords = pred.shape\n confidences = pred.new_ones((batch_size, 1))\n return pytorch_neg_multi_log_likelihood_batch(gt, pred.unsqueeze(1), confidences, avails)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model"},{"metadata":{},"cell_type":"markdown","source":"Next we define the baseline model. Note that this model will return three possible trajectories together with confidence score for each trajectory."},{"metadata":{"trusted":true},"cell_type":"code","source":"class LyftMultiModel(nn.Module):\n\n def __init__(self, cfg: Dict, num_modes=3):\n super().__init__()\n\n architecture = cfg[\"model_params\"][\"model_architecture\"]\n backbone = eval(architecture)(pretrained=True, progress=True)\n self.backbone = backbone\n\n num_history_channels = (cfg[\"model_params\"][\"history_num_frames\"] + 1) * 2\n num_in_channels = 3 + num_history_channels\n\n self.backbone.conv1 = nn.Conv2d(\n num_in_channels,\n self.backbone.conv1.out_channels,\n kernel_size=self.backbone.conv1.kernel_size,\n stride=self.backbone.conv1.stride,\n padding=self.backbone.conv1.padding,\n bias=False,\n )\n\n # This is 512 for resnet18 and resnet34;\n # And it is 2048 for the other resnets\n \n if architecture == \"resnet50\":\n backbone_out_features = 2048\n else:\n backbone_out_features = 512\n\n # X, Y coords for the future positions (output shape: batch_sizex50x2)\n self.future_len = cfg[\"model_params\"][\"future_num_frames\"]\n num_targets = 2 * self.future_len\n\n # You can add more layers here.\n self.head = nn.Sequential(\n # nn.Dropout(0.2),\n nn.Linear(in_features=backbone_out_features, out_features=4096),\n )\n\n self.num_preds = num_targets * num_modes\n self.num_modes = num_modes\n\n self.logit = nn.Linear(4096, out_features=self.num_preds + num_modes)\n\n def forward(self, x):\n x = self.backbone.conv1(x)\n x = self.backbone.bn1(x)\n x = self.backbone.relu(x)\n x = self.backbone.maxpool(x)\n\n x = self.backbone.layer1(x)\n x = self.backbone.layer2(x)\n x = self.backbone.layer3(x)\n x = self.backbone.layer4(x)\n\n x = self.backbone.avgpool(x)\n x = torch.flatten(x, 1)\n\n x = self.head(x)\n x = self.logit(x)\n\n # pred (batch_size)x(modes)x(time)x(2D coords)\n # confidences (batch_size)x(modes)\n bs, _ = x.shape\n pred, confidences = torch.split(x, self.num_preds, dim=1)\n pred = pred.view(bs, self.num_modes, self.future_len, 2)\n assert confidences.shape == (bs, self.num_modes)\n confidences = torch.softmax(confidences, dim=1)\n return pred, confidences","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def forward(data, model, device, criterion = pytorch_neg_multi_log_likelihood_batch):\n inputs = data[\"image\"].to(device)\n target_availabilities = data[\"target_availabilities\"].to(device)\n targets = data[\"target_positions\"].to(device)\n # Forward pass\n preds, confidences = model(inputs)\n loss = criterion(targets, preds, confidences, target_availabilities)\n return loss, preds, confidences","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let us initialize the model and load the pretrained weights. Note that since the pretrained model was trained on GPU, you also need to enable GPU when running this notebook."},{"metadata":{"trusted":true},"cell_type":"code","source":"# ==== INIT MODEL=================\ndevice = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\nmodel = LyftMultiModel(cfg)\n\n#load weight if there is a pretrained model\nweight_path = cfg[\"model_params\"][\"weight_path\"]\nif weight_path:\n model.load_state_dict(torch.load(weight_path))\n\nmodel.to(device)\noptimizer = optim.Adam(model.parameters(), lr=cfg[\"model_params\"][\"lr\"])\nprint(f'device {device}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training loop"},{"metadata":{},"cell_type":"markdown","source":"Next let us implement the training loop, when the **train** parameter is set to True. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# ==== TRAINING LOOP =========================================================\nif cfg[\"model_params\"][\"train\"]:\n \n tr_it = iter(train_dataloader)\n progress_bar = tqdm(range(cfg[\"train_params\"][\"max_num_steps\"]))\n num_iter = cfg[\"train_params\"][\"max_num_steps\"]\n losses_train = []\n iterations = []\n metrics = []\n times = []\n model_name = cfg[\"model_params\"][\"model_name\"]\n start = time.time()\n for i in progress_bar:\n try:\n data = next(tr_it)\n except StopIteration:\n tr_it = iter(train_dataloader)\n data = next(tr_it)\n model.train()\n torch.set_grad_enabled(True)\n \n loss, _, _ = forward(data, model, device)\n\n # Backward pass\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n\n losses_train.append(loss.item())\n\n progress_bar.set_description(f\"loss: {loss.item()} loss(avg): {np.mean(losses_train)}\")\n if i % cfg['train_params']['checkpoint_every_n_steps'] == 0:\n torch.save(model.state_dict(), f'{model_name}_{i}.pth')\n iterations.append(i)\n metrics.append(np.mean(losses_train))\n times.append((time.time()-start)/60)\n\n results = pd.DataFrame({'iterations': iterations, 'metrics (avg)': metrics, 'elapsed_time (mins)': times})\n results.to_csv(f\"train_metrics_{model_name}_{num_iter}.csv\", index = False)\n print(f\"Total training time is {(time.time()-start)/60} mins\")\n print(results.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prediction"},{"metadata":{},"cell_type":"markdown","source":"Finally we implement the inference to submit to Kaggle when **predict** param is set to True."},{"metadata":{"trusted":true},"cell_type":"code","source":"# ==== EVAL LOOP ================================================================\nif cfg[\"model_params\"][\"predict\"]:\n \n model.eval()\n torch.set_grad_enabled(False)\n\n # store information for evaluation\n future_coords_offsets_pd = []\n timestamps = []\n confidences_list = []\n agent_ids = []\n\n progress_bar = tqdm(test_dataloader)\n \n for data in progress_bar:\n \n _, preds, confidences = forward(data, model, device)\n \n #fix for the new environment\n preds = preds.cpu().numpy()\n world_from_agents = data[\"world_from_agent\"].numpy()\n centroids = data[\"centroid\"].numpy()\n coords_offset = []\n \n # convert into world coordinates and compute offsets\n for idx in range(len(preds)):\n for mode in range(3):\n preds[idx, mode, :, :] = transform_points(preds[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]\n \n future_coords_offsets_pd.append(preds.copy())\n confidences_list.append(confidences.cpu().numpy().copy())\n timestamps.append(data[\"timestamp\"].numpy().copy())\n agent_ids.append(data[\"track_id\"].numpy().copy()) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#create submission to submit to Kaggle\npred_path = 'submission.csv'\nwrite_pred_csv(pred_path,\n timestamps=np.concatenate(timestamps),\n track_ids=np.concatenate(agent_ids),\n coords=np.concatenate(future_coords_offsets_pd),\n confs = np.concatenate(confidences_list)\n )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

That is it. Feel free to modify this kernel as you wish. Let me know your suggestions in the comments below.

\n

Please upvote if this kernel helps you. Thanks!

"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}