{"cells":[{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#python basics\nfrom matplotlib import pyplot as plt\nimport math, os, re, time, random\nimport numpy as np, pandas as pd, seaborn as sns\nfrom tqdm import tqdm\nfrom typing import Dict\n\n#for deep learning\nimport torch\nfrom torch import nn, optim, Tensor\nfrom torchvision.models.resnet import resnet18\n\n#for scene visualization\nfrom IPython.display import display, clear_output\nfrom IPython.display import HTML\nimport PIL\nfrom matplotlib import animation, rc\n\nos.chdir('/kaggle/input')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Overview\n\n**Most of the contents of this notebook can be found on the `l5kit` GitHub repository in the `examples` folder [here](https://github.com/lyft/l5kit/blob/master/examples). It is a great place to start for this competition.**\n\n# Data Format\n\n**The data in this competition is packed into `.zarr` files, which can be loaded with the Python `zarr` module. Taken from the competition Data description, each `.zarr` file contains a set of:**\n\n* scenes: driving episodes acquired from a given vehicle.\n* frames: snapshots in time of the pose of the vehicle.\n* agents: a generic entity captured by the vehicle's sensors. Note that only 4 of the 17 possible agent label_probabilities are present in this dataset.\n* agents_mask: a mask that (for train and validation) masks out objects that aren't useful for training. In test, the mask (provided in files as mask.npz) masks out any test object for which predictions are NOT required.\n* traffic_light_faces: traffic light information.\n\n# l5kit\n\n**`l5kit` is a Python library developed by Lyft Level 5 with functionality for developing and training *learned prediction, planning and simulation* models for autonomous driving applications. Among other things, this library allows us to predict the future movement of cars based on historical observations. In particular, `l5kit` allows us to:**\n\n* Load driving scenes from zarr files\n* Read aerial and semantic maps\n* Create birds-eye-view images that repesent a scene around an automonous vehicle/other vehicles\n* And most importantly, train neural networks and visualize their results\n\n**Let's add `l5kit` now and start to explore how we can use it for the task at hand. Note that as per the competition rules, internet is *not* allowed in this competition, so we cannot use `pip` to install packages. This is why Lyft has provided us the `l5kit` module as a utility script called `kaggle_l5kit`.**"},{"metadata":{},"cell_type":"markdown","source":"## Data\n\n**Data is expected to live in the folder that we set using the `L5KIT_DATA_FOLD` env variable. This folder should contain subfolders for the aerial and semantic maps as well as the `.zarr` files (the scenes). Luckily, this is exactly how our data is set up in this Kaggle kernel so all we need to do is this:** \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# set env variable for data\nPATH_TO_DATA = \"../input/lyft-motion-prediction-autonomous-vehicles\"\nos.environ[\"L5KIT_DATA_FOLDER\"] = PATH_TO_DATA","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Now, we can iteract with `l5kit` via the below configuration. It will become clear how to manipulate this `cfg` as we go through the notebook:**"},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg = {\n    'model_params': {\n        'model_architecture': 'resnet18',\n        \n        'history_num_frames': 10,\n        'history_step_size': 1,\n        'history_delta_time': 0.1,\n        \n        'future_num_frames': 50,\n        'future_step_size': 1,\n        'future_delta_time': 0.1\n    },\n    \n    'raster_params': {\n        'raster_size': [300, 300],\n        'pixel_size': [0.5, 0.5],\n        'ego_center': [0.25, 0.5],\n        'map_type': 'py_semantic',\n        'satellite_map_key': 'aerial_map/aerial_map.png',\n        'semantic_map_key': 'semantic_map/semantic_map.pb',\n        'dataset_meta_key': 'meta.json',\n        'filter_agents_threshold': 0.5\n    },\n    \n    'train_data_loader': {\n        'key': 'scenes/train.zarr',\n        'batch_size': 32,\n        'shuffle': True,\n        'num_workers': 4\n    },\n    \n    'sample_data_loader': {\n        'key': 'scenes/sample.zarr',\n        'batch_size': 32,\n        'shuffle': True,\n        'num_workers': 4\n    },\n    \n    'test_data_loader': {\n        'key': 'scenes/test.zarr',\n        'batch_size': 8,\n        'shuffle': False,\n        'num_workers': 4\n    },\n    \n    'train_params': {\n        'checkpoint_every_n_steps': 5000,\n        'max_num_steps': 25,\n        'image_coords': True\n        \n    },\n        \n    'test_params': {\n        'image_coords': True\n        \n    }\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**This configuration file is used to load the data by using a `LocalDataManager` object, which resolves relative paths from the configuration file using the `L5KIT_DATA_FOLDER` env that we set earlier, to extract the `.zarr` file and store it as a `ChunkedDataset` object. When looking at the above configuration dictionary, we see that the `.zarr` files of interest are in `train_data_loader : key`:**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from l5kit.data import LocalDataManager, ChunkedDataset\n\ndm = LocalDataManager()\ntrain_zarr = ChunkedDataset(dm.require(cfg[\"train_data_loader\"][\"key\"])).open()\n\n#let's see what one of the objects looks like\nprint(train_zarr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualization\n\n**Main source: https://github.com/lyft/l5kit/tree/master/examples/visualisation**\n\n### Raw Data\n\n**`.zarr` files support most traditional NumPy array operations. For example, we can iterate over the frames in our `.zarr` file to create a scatter plot of all the AV (henceforth AV) locations like so:**\n\n**I have this #-d out because it takes around 40 minutes to complete and is not nearly as insightful as the below visualizations will be:**"},{"metadata":{"trusted":true},"cell_type":"code","source":"#frames = train_zarr.frames\n#coords = np.zeros((len(frames), 2))\n#for idx_coord, idx_data in enumerate(tqdm(range(len(frames)), desc=\"getting centroid to plot trajectory\")):\n    #frame = train_zarr.frames[idx_data]\n    #coords[idx_coord] = frame[\"ego_translation\"][:2]\n\n\n#plt.scatter(coords[:, 0], coords[:, 1], marker='.')\n#axes = plt.gca()\n#axes.set_xlim([-2500, 1600])\n#axes.set_ylim([-2500, 1600])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Abstraction\n\n**Working with raw `.zarr` files is fine, but `l5kit` also provides abstract data classes to generate inputs and targets with ease. Most importantly, we can use a `rasterizer`, which takes these chunked `.zarr` files and processes them into rectangular grid of pixels (a raster image) so that we can view them regularly on a computer screen.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from l5kit.rasterization import build_rasterizer\n\nrasterizer = build_rasterizer(cfg, dm)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**We can then use the `EgoDataset` and `AgentDataset` objects in `l5kit` to iterate over the rasterizer object we just defined to return RGB images. The `EgoDataset` iterates only over the AV annotations and the `AgentDataset` iterates everything but the AV annotations. Let's first visualize the AV using an `EgoDataset`:**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from l5kit.dataset import EgoDataset\n\nAV_ds = EgoDataset(cfg, train_zarr, rasterizer)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Once in an `EgoDataset`, we can plot the ground truth trajectory by converting the `EgoDataset.target_position` into pixel coordinates and then call `draw_trajectory`, which can be used to draw predicted trajectories as well.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from l5kit.visualization import draw_trajectory, TARGET_POINTS_COLOR\nfrom l5kit.geometry import transform_points\n\nsample = AV_ds[50]\n\nim = sample[\"image\"].transpose(1, 2, 0)\nim = AV_ds.rasterizer.to_rgb(im)\ntarget_positions_pixels = transform_points(sample[\"target_positions\"] + sample[\"centroid\"][:2], sample[\"world_to_image\"])\ndraw_trajectory(im, target_positions_pixels, sample[\"target_yaws\"], TARGET_POINTS_COLOR)\n\n_, ax = plt.subplots(figsize = (7, 7))\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**We can change the `rasterizer` by building a new one and an accomodating dataset for it like so:**"},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg[\"raster_params\"][\"map_type\"] = \"py_satellite\"\nrasterizer = build_rasterizer(cfg, dm)\nAV_ds = EgoDataset(cfg, train_zarr, rasterizer)\nsample = AV_ds[50]\n\nim = sample[\"image\"].transpose(1, 2, 0)\nim = AV_ds.rasterizer.to_rgb(im)\ntarget_positions_pixels = transform_points(sample[\"target_positions\"] + sample[\"centroid\"][:2], sample[\"world_to_image\"])\ndraw_trajectory(im, target_positions_pixels, sample[\"target_yaws\"], TARGET_POINTS_COLOR)\n\n_, ax = plt.subplots(figsize = (7, 7))\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**And now we can do the exact same thing to view the agents by switching from an `EgoDataset` to an `AgentDataset`:**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from l5kit.dataset import AgentDataset\n\ncfg[\"raster_params\"][\"map_type\"] = \"py_semantic\"\nA_ds = AgentDataset(cfg, train_zarr, rasterizer)\nsample = A_ds[50]\n\nim = sample[\"image\"].transpose(1, 2, 0)\nim = A_ds.rasterizer.to_rgb(im)\ntarget_positions_pixels = transform_points(sample[\"target_positions\"] + sample[\"centroid\"][:2], sample[\"world_to_image\"])\ndraw_trajectory(im, target_positions_pixels, sample[\"target_yaws\"], TARGET_POINTS_COLOR)\n\n_, ax = plt.subplots(figsize = (7, 7))\nplt.imshow(im[::-1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**The below animation code is taken from [here](https://www.kaggle.com/nxrprime/lyft-understanding-the-data-and-eda):**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def animate_solution(images):\n    def animate(i):\n        im.set_data(images[i])\n        \n    fig, ax = plt.subplots()\n    im = ax.imshow(images[0])\n    \n    return animation.FuncAnimation(fig, animate, frames = len(images), interval = 60)\n \ncfg[\"raster_params\"][\"map_type\"] = \"py_semantic\"\nrasterizer = build_rasterizer(cfg, dm)\nA_ds = EgoDataset(cfg, train_zarr, rasterizer)\nscene_idx = 34\nindexes = AV_ds.get_scene_indices(scene_idx)\nimages = []\n\nfor idx in indexes:\n    data = A_ds[idx]\n    im = data[\"image\"].transpose(1, 2, 0)\n    im = A_ds.rasterizer.to_rgb(im)\n    target_positions_pixels = transform_points(data[\"target_positions\"] + data[\"centroid\"][:2], data[\"world_to_image\"])\n    center_in_pixels = np.asarray(cfg[\"raster_params\"][\"ego_center\"]) * cfg[\"raster_params\"][\"raster_size\"]\n    draw_trajectory(im, target_positions_pixels, data[\"target_yaws\"], TARGET_POINTS_COLOR)\n    clear_output(wait=True)\n    images.append(PIL.Image.fromarray(im[::-1]))\n    \nHTML(animate_solution(images).to_jshtml())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Pretty cool right? We are able to follow our AV as it traverses threw this scene**"},{"metadata":{},"cell_type":"markdown","source":"# PyTorch Training\n\n**Main source: https://github.com/lyft/l5kit/tree/master/examples/agent_motion_prediction**\n\n**And now we are ready to explore how to train a neural network with this `.zarr` data. To prepare a rar `.zarr` file for training, we follow the below procedure:**\n\n1. Load `.zarr` file into a `ChunkedDataset` object\n2. Wrap `ChunkedDataset` object into an `AgentDataset`, which inhereits from the PyTorch `Dataset` class\n3. Pass `AgentDataset` into a torch `Dataloader`\n\n**Let's do this now:**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from torch.utils.data import DataLoader\n\ntrain_cfg = cfg[\"train_data_loader\"]\nrasterizer = build_rasterizer(cfg, dm)\ntrain_zarr = ChunkedDataset(dm.require(train_cfg[\"key\"])).open()\ntrain_dataset = AgentDataset(cfg, train_zarr, rasterizer)\ntrain_dataloader = DataLoader(train_dataset,\n                              shuffle = cfg[\"train_data_loader\"][\"shuffle\"],\n                              batch_size = cfg[\"train_data_loader\"][\"batch_size\"],\n                              num_workers = cfg[\"train_data_loader\"][\"num_workers\"])\n\nprint(len(train_dataloader))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model\n\n**We can create a simple model to feed this `DataLoader` to. The below is a ResNet18 pre-trained on ImagNet taken from [Peter](https://www.kaggle.com/pestipeti)'s notebook [here](https://www.kaggle.com/pestipeti/pytorch-baseline-train). We need to change the input channels of the ResNet to match the `rasterizer` output. We also need to change the output size to `(X, Y) * number of future states`:**\n\n**Update: see discussion post [here](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/186492) for explanation of new architecture and training loop changes:**"},{"metadata":{"trusted":true},"cell_type":"code","source":"class LyftModel(nn.Module):\n\n    def __init__(self, cfg: Dict, num_modes=3):\n        super().__init__()\n\n        architecture = cfg[\"model_params\"][\"model_architecture\"]\n        backbone = eval(architecture)(pretrained=True, progress=True)\n        self.backbone = backbone\n\n        num_history_channels = (cfg[\"model_params\"][\"history_num_frames\"] + 1) * 2\n        num_in_channels = 3 + num_history_channels\n\n        self.backbone.conv1 = nn.Conv2d(\n            num_in_channels,\n            self.backbone.conv1.out_channels,\n            kernel_size=self.backbone.conv1.kernel_size,\n            stride=self.backbone.conv1.stride,\n            padding=self.backbone.conv1.padding,\n            bias=False,\n        )\n        \n        if architecture == \"resnet50\":\n            backbone_out_features = 2048\n        else:\n            backbone_out_features = 512\n\n        # X, Y coords for the future positions (output shape: batch_sizex50x2)\n        self.future_len = cfg[\"model_params\"][\"future_num_frames\"]\n        num_targets = 2 * self.future_len\n\n        # You can add more layers here.\n        self.head = nn.Sequential(\n            # nn.Dropout(0.2),\n            nn.Linear(in_features=backbone_out_features, out_features=4096),\n        )\n\n        self.num_preds = num_targets * num_modes\n        self.num_modes = num_modes\n\n        self.logit = nn.Linear(4096, out_features=self.num_preds + num_modes)\n\n    def forward(self, x):\n        x = self.backbone.conv1(x)\n        x = self.backbone.bn1(x)\n        x = self.backbone.relu(x)\n        x = self.backbone.maxpool(x)\n\n        x = self.backbone.layer1(x)\n        x = self.backbone.layer2(x)\n        x = self.backbone.layer3(x)\n        x = self.backbone.layer4(x)\n\n        x = self.backbone.avgpool(x)\n        x = torch.flatten(x, 1)\n\n        x = self.head(x)\n        x = self.logit(x)\n        \n        return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def forward(data, model, device, criterion):\n    inputs = data[\"image\"].to(device)\n    target_availabilities = data[\"target_availabilities\"].to(device)\n    targets = data[\"target_positions\"].to(device)\n    matrix = data[\"world_to_image\"].to(device)\n    centroid = data[\"centroid\"].to(device)[:,None,:].to(torch.float)\n\n    # Forward pass\n    outputs = model(inputs)\n\n    bs,tl,_ = targets.shape\n    assert tl == cfg[\"model_params\"][\"future_num_frames\"]\n\n    if cfg['train_params']['image_coords']:\n        targets = targets + centroid\n        targets = torch.cat([targets,torch.ones((bs,tl,1)).to(device)], dim=2)\n        targets = torch.matmul(matrix.to(torch.float), targets.transpose(1,2))\n        targets = targets.transpose(1,2)[:,:,:2]\n        rs = cfg[\"raster_params\"][\"raster_size\"]\n        ec = cfg[\"raster_params\"][\"ego_center\"]\n\n        bias = torch.tensor([rs[0] * ec[0], rs[1] * ec[1]])[None, None, :].to(device)\n        targets = targets - bias\n\n    confidences, pred = outputs[:,:3], outputs[:,3:]\n    pred = pred.view(bs, 3, tl, 2)\n    assert confidences.shape == (bs, 3)\n    confidences = torch.softmax(confidences, dim=1)\n\n    loss = criterion(targets, pred, confidences, target_availabilities)\n    loss = torch.mean(loss)\n\n    if cfg['train_params']['image_coords']:\n        matrix_inv = torch.inverse(matrix)\n        pred = pred + bias[:,None,:,:]\n        pred = torch.cat([pred,torch.ones((bs,3,tl,1)).to(device)], dim=3)\n        pred = torch.stack([torch.matmul(matrix_inv.to(torch.float), pred[:,i].transpose(1,2)) \n                            for i in range(3)], dim=1)\n        pred = pred.transpose(2,3)[:,:,:,:2]\n        pred = pred - centroid[:,None,:,:]\n\n    return loss, pred, confidences","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# from https://www.kaggle.com/corochann/lyft-training-with-multi-mode-confidence\ndef pytorch_neg_multi_log_likelihood_batch(\n    gt: Tensor, pred: Tensor, confidences: Tensor, avails: Tensor\n) -> Tensor:\n\n    assert len(pred.shape) == 4, f\"expected 3D (MxTxC) array for pred, got {pred.shape}\"\n    batch_size, num_modes, future_len, num_coords = pred.shape\n\n    assert gt.shape == (batch_size, future_len, num_coords), f\"expected 2D (Time x Coords) array for gt, got {gt.shape}\"\n    assert confidences.shape == (batch_size, num_modes), f\"expected 1D (Modes) array for gt, got {confidences.shape}\"\n    assert torch.allclose(torch.sum(confidences, dim=1), confidences.new_ones((batch_size,))), \"confidences should sum to 1\"\n    assert avails.shape == (batch_size, future_len), f\"expected 1D (Time) array for gt, got {avails.shape}\"\n    assert torch.isfinite(pred).all(), \"invalid value found in pred\"\n    assert torch.isfinite(gt).all(), \"invalid value found in gt\"\n    assert torch.isfinite(confidences).all(), \"invalid value found in confidences\"\n    assert torch.isfinite(avails).all(), \"invalid value found in avails\"\n\n    # convert to (batch_size, num_modes, future_len, num_coords)\n    gt = torch.unsqueeze(gt, 1)  # add modes\n    avails = avails[:, None, :, None]  # add modes and cords\n\n    # error (batch_size, num_modes, future_len)\n    error = torch.sum(((gt - pred) * avails) ** 2, dim=-1)  # reduce coords and use availability\n    \n    if cfg['train_params']['image_coords']:\n        error = error / 4\n\n    with np.errstate(divide=\"ignore\"):  # when confidence is 0 log goes to -inf, but we're fine with it\n        # error (batch_size, num_modes)\n        error = torch.log(confidences) - 0.5 * torch.sum(error, dim=-1)  # reduce time\n\n    # use max aggregator on modes for numerical stability\n    # error (batch_size, num_modes)\n    max_value, _ = error.max(dim=1, keepdim=True)  # error are negative at this point, so max() gives the minimum one\n    error = -torch.log(torch.sum(torch.exp(error - max_value), dim=-1, keepdim=True)) - max_value  # reduce modes\n    # print(\"error\", error)\n    return torch.mean(error)\n\n\ndef pytorch_neg_multi_log_likelihood_single(\n    gt: Tensor, pred: Tensor, avails: Tensor\n) -> Tensor:\n\n    # pred (bs)x(time)x(2D coords) --> (bs)x(mode=1)x(time)x(2D coords)\n    # create confidence (bs)x(mode=1)\n    batch_size, future_len, num_coords = pred.shape\n    confidences = pred.new_ones((batch_size, 1))\n    return pytorch_neg_multi_log_likelihood_batch(gt, pred.unsqueeze(1), confidences, avails)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**We can speed up our training by placing our computations on the GPU with the `torch.device` object. We will define our optimizer and metric here as well:**"},{"metadata":{},"cell_type":"markdown","source":"### Training Loop"},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\nmodel = LyftModel(cfg).to(device)\noptimizer = optim.Adam(model.parameters(), lr = 1e-3)\ncriterion = pytorch_neg_multi_log_likelihood_batch","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if device.type == 'cpu': print('Training on CPU')\nif device.type == 'cuda': print('Training on GPU')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tr_it = iter(train_dataloader)\n\nprogress_bar = tqdm(range(cfg[\"train_params\"][\"max_num_steps\"]))\nlosses_train = []\n\nfor itr in progress_bar:\n\n    try:\n        data = next(tr_it)\n    except StopIteration:\n        tr_it = iter(train_dataloader)\n        data = next(tr_it)\n\n    model.train()\n    torch.set_grad_enabled(True)\n\n    loss, pred, confidences = forward(data, model, device, criterion)\n\n    # Backward pass\n    optimizer.zero_grad()\n    loss.backward()\n    optimizer.step()\n\n    losses_train.append(loss.item())\n        \n    #save model during training\n    if (itr+1) % cfg['train_params']['checkpoint_every_n_steps'] == 0 and not DEBUG:\n        torch.save(model.state_dict(), f'model_state_{itr}.pth')\n    \n    #display training progress\n    progress_bar.set_description(f\"loss: {loss.item()} loss(avg): {np.mean(losses_train)}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Once you are happy with your training, you can save the model state to your disk and upload the pre-trained weights offline for an inference submission using `torch.save`**"},{"metadata":{},"cell_type":"markdown","source":"# Inference\n\n**And now, after all that work, we can predict with this baseline model. The following code is (again) taken from [Peter](https://www.kaggle.com/pestipeti)'s notebook [here](https://www.kaggle.com/pestipeti/pytorch-baseline-inference):**\n\n**Since I have internet enabled in this notebook (which is not allowed for submission notebooks in this comp), I carried out the prediction part in [this notebook](https://www.kaggle.com/tuckerarrants/lyft-inference-resnet50/edit), but I included the general procedure here for completeness**"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_cfg = cfg[\"test_data_loader\"]\n\n# Rasterizer\nrasterizer = build_rasterizer(cfg, dm)\n\n# Test dataset/dataloader\ntest_zarr = ChunkedDataset(dm.require(test_cfg[\"key\"])).open()\ntest_mask = np.load(f\"{PATH_TO_DATA}/scenes/mask.npz\")[\"arr_0\"]\ntest_dataset = AgentDataset(cfg, test_zarr, rasterizer, agents_mask=test_mask)\ntest_dataloader = DataLoader(test_dataset,\n                             shuffle=test_cfg[\"shuffle\"],\n                             batch_size=test_cfg[\"batch_size\"],\n                             num_workers=test_cfg[\"num_workers\"])\n\n\nprint(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_prediction(predictor, data_loader):\n    predictor.eval()\n\n    pred_coords_list = []\n    confidences_list = []\n    timestamps_list = []\n    track_id_list = []\n\n    with torch.no_grad():\n        dataiter = tqdm(data_loader)\n        for data in dataiter:\n            image = data[\"image\"].to(device)\n            inputs = data[\"image\"].to(device)\n            target_availabilities = data[\"target_availabilities\"].to(device)\n            targets = data[\"target_positions\"].to(device)\n            matrix = data[\"world_to_image\"].to(device)\n            centroid = data[\"centroid\"].to(device)[:,None,:].to(torch.float)\n            \n            pred, confidences = predictor(image)\n            \n            if cfg['test_params']['image_coords']:\n                matrix_inv = torch.inverse(matrix)\n                pred = pred + bias[:,None,:,:]\n                pred = torch.cat([pred,torch.ones((bs,3,tl,1)).to(device)], dim=3)\n                pred = torch.stack([torch.matmul(matrix_inv.to(torch.float), pred[:,i].transpose(1,2)) \n                                    for i in range(3)], dim=1)\n                pred = pred.transpose(2,3)[:,:,:,:2]\n                pred = pred - centroid[:,None,:,:]\n\n            pred_coords_list.append(pred.cpu().numpy().copy())\n            confidences_list.append(confidences.cpu().numpy().copy())\n            timestamps_list.append(data[\"timestamp\"].numpy().copy())\n            track_id_list.append(data[\"track_id\"].numpy().copy())\n            \n    timestamps = np.concatenate(timestamps_list)\n    track_ids = np.concatenate(track_id_list)\n    coords = np.concatenate(pred_coords_list)\n    confs = np.concatenate(confidences_list)\n    return timestamps, track_ids, coords, confs","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Here you upload your freshly trained model:**"},{"metadata":{"trusted":true},"cell_type":"code","source":"LOAD_MODEL = False\n\nif LOAD_MODEL:\n    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n    predictor = LyftModel(cfg)\n    saved_model_path = \"\"\n    predictor.load_state_dict(torch.load(saved_model_path))\n    predictor.to(device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Now we get our predictions and save them as a `csv` file with `write_pred_csv`:**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from l5kit.evaluation import write_pred_csv\n\nINFER = False\n\nif INFER:\n    timestamps, track_ids, coords, confs = run_prediction(predictor, test_dataloader)\n    write_pred_csv('submission.csv',timestamps=timestamps,\n    track_ids=track_ids, coords=coords, confs=confs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training Parameter EDA\n\n**Let's see which parameters in our configuration file are relevant to training:**"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset[0].keys()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Let's explore two important training specific parameters: `raster_size` and `pixel_size`:**"},{"metadata":{},"cell_type":"markdown","source":"### Raster Size"},{"metadata":{"trusted":true},"cell_type":"code","source":"def sample_dataset():\n    # Build Rasterizer\n    rasterizer = build_rasterizer(cfg, dm)\n    \n    train_dataset = AgentDataset(cfg, train_zarr, rasterizer)\n    return train_dataset[100]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sizes = [[200, 200], [224, 224], [250, 250], [350, 350], [450, 450], [500, 500]]\n\nf, ax = plt.subplots(2, 3, figsize=(20, 12))\nax = ax.flatten()\n\nfor i in range(6):\n    cfg['raster_params']['raster_size'] = sizes[i]\n    sample = sample_dataset()\n    \n    ax[i].imshow(sample['image'][-3:].transpose(1, 2, 0))\n    ax[i].get_xaxis().set_visible(False)\n    ax[i].get_yaxis().set_visible(False)\n    ax[i].set_title(f\"Raster size: {sizes[i]}\")\n\n#reset to default\ncfg['raster_params']['raster_size'] = [224, 224]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**We can see that different raster sizes affect how much of a scene is rasterized, meaning that our model sees different agents depending on the raster size. This is a powerful hyper parameter: should our model focus on agents far away? Or should it focus on agents closer to it?**"},{"metadata":{},"cell_type":"markdown","source":"### Pixel Size\n\n**From the `l5kit` GitHub repository, the pixel size of the raster image is the:**\n\n> raster's spatial resolution [meters per pixel]: the size in the real world one pixel corresponds to.\n\n**So this is the value to tweak if you want to change the resolution of the image. This is yet another import training parameter to experiment with, especially when we are feeding these rasterized images to a CNN**"},{"metadata":{"trusted":true},"cell_type":"code","source":"sizes = [[.2, .2,], [.3, .3], [.4, .4], [.5, .5], [.6, .6], [.7, .7]]\n\nf, ax = plt.subplots(2, 3, figsize=(20, 12))\nax = ax.flatten()\n\nfor i in range(6):\n    cfg['raster_params']['pixel_size'] = sizes[i]\n    sample = sample_dataset()\n    \n    ax[i].imshow(sample['image'][-3:].transpose(1, 2, 0))\n    ax[i].get_xaxis().set_visible(False)\n    ax[i].get_yaxis().set_visible(False)\n    ax[i].set_title(f\"Pixel size: {sizes[i]}\")\n\n#reset to default\ncfg['raster_params']['pixel_size'] = [0.5, 0.5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**If `pixel_size = [.5, .5]`, this means that we are at most able to capture objects no less than half a meter in size: if the object is smaller than half a meter, it will not be detected as a pixel, so to view these objects, we need a higher resolution (lower `pixel_size`)**\n\n**As you can read about in this [notebook](https://www.kaggle.com/ilu000/expected-error-inaccuracy-from-rasterization), you'll see that there is some inherent innacuracy from the rasterization process defined by `pixel_size`. Each history position, lane, and other agents are encoded into the pixels and if the raster has `pixel_size = .5`, we ought to expect a mean error of `.5/4` for each direction of each predicted position. We can combat this by using a smaller `pixel_size`:**"},{"metadata":{},"cell_type":"markdown","source":"### Changing Raster Size and Pixel Size"},{"metadata":{"trusted":true},"cell_type":"code","source":"sizes = [[[200, 200],[.5, .5]], [[250, 250],[.4, .4]], [[350, 350],[.3, .3]], [[500, 500],[.2, .2]]]\n\nf, ax = plt.subplots(1, 4, figsize=(20, 12))\nax = ax.flatten()\n\nfor i in range(4):\n    cfg['raster_params']['pixel_size'] = sizes[i][1]\n    cfg['raster_params']['raster_size'] = sizes[i][0]\n    sample = sample_dataset()\n    \n    ax[i].imshow(sample['image'][-3:].transpose(1, 2, 0))\n    ax[i].get_xaxis().set_visible(False)\n    ax[i].get_yaxis().set_visible(False)\n    ax[i].set_title(f\"Raster/Pixel size: {sizes[i]}\")\n\n#reset to default\ncfg['raster_params']['raster_size'] = [224, 224]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**If you want the model to see more of a scene, you can change either `raster_size` or `pixel_size`. More importantly, if you want the same scene in a higher resolution, you need to change *both* `raster_size` and `pixel_size`. For a more thorough explanation, see Peter's comment [here](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/178323)**"},{"metadata":{"trusted":true},"cell_type":"code","source":"sizes = [[[400, 400],[.2, .2]], [[500, 500],[.2, .2]], [[600, 600],[.2, .2]], [[700, 700],[.2, .2]]]\n\nf, ax = plt.subplots(1, 4, figsize=(20, 12))\nax = ax.flatten()\n\nfor i in range(4):\n    cfg['raster_params']['pixel_size'] = sizes[i][1]\n    cfg['raster_params']['raster_size'] = sizes[i][0]\n    sample = sample_dataset()\n    \n    ax[i].imshow(sample['image'][-3:].transpose(1, 2, 0))\n    ax[i].get_xaxis().set_visible(False)\n    ax[i].get_yaxis().set_visible(False)\n    ax[i].set_title(f\"Raster/Pixel size: {sizes[i]}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Augmentation\n\n**It seems there is a large gap between training loss and validation. This is largely because some images are very similar, e.g. the frames of the agent vehicle at a red light are much the same in comparison to frames in which it is moving at full speed.**\n\n**We can reduce this similarity between frames by introducing augmentations like CutOut, CourseDropout, and more. `l5kit` uses PyTorch `DataLoaders` so we will use `albumentations` to compose these augmentations.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"#for augmentations\nimport albumentations as A\n\n#for better visualization\ncfg['raster_params']['raster_size'] = [500, 500]\ncfg['raster_params']['pixle_size'] = [.25, .25]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def show_images(aug_cfg):\n    dm = LocalDataManager()\n    train_zarr = ChunkedDataset(dm.require(cfg[\"train_data_loader\"][\"key\"])).open()\n    cfg[\"raster_params\"][\"map_type\"] = 'py_semantic'\n    rasterizer = build_rasterizer(cfg, dm)\n    sem_ds = AgentDataset(cfg, train_zarr, rasterizer)\n    cfg[\"raster_params\"][\"map_type\"] = 'py_satellite'\n    rasterizer = build_rasterizer(cfg, dm)\n    sat_ds = AgentDataset(cfg, train_zarr, rasterizer)\n\n    #get a random sample\n    random_index = int(np.random.random()*len(AV_ds))\n    sat_sample = sat_ds[random_index]\n    sem_sample = sem_ds[random_index]\n\n    sat_im = sat_sample[\"image\"].transpose(1, 2, 0)\n    sat_im = sat_ds.rasterizer.to_rgb(sat_im)\n    sem_im = sem_sample[\"image\"].transpose(1, 2, 0)\n    sem_im = sem_ds.rasterizer.to_rgb(sem_im)\n    \n    fig, ax = plt.subplots(len(aug_cfg), 2, figsize=(15,15))\n    \n    for i, (key, aug) in enumerate(aug_cfg.items()):\n        if aug is None:\n            ax[i, 0].imshow(sat_im[::-1])\n            ax[i, 0].set_title(key)\n        else:\n            sat_im_ = aug(image=sat_im)['image']\n            ax[i, 0].imshow(sat_im_[::-1])\n            ax[i, 0].set_title(key)\n            \n    for i, (key, aug) in enumerate(aug_cfg.items()):\n        if aug is None:\n            ax[i, 1].imshow(sem_im[::-1])\n            ax[i, 1].set_title(key)\n        else:\n            sem_im_ = aug(image=sem_im)['image']\n            ax[i, 1].imshow(sem_im_[::-1])\n            ax[i, 1].set_title(key)\n\n    plt.tight_layout();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"aug_dict = {'Original': None,\n            \n            'Cutout':A.Cutout(num_holes=10, max_h_size=20, max_w_size=20, fill_value=0, \n                              always_apply=False, p=1),\n            \n            'CoarseDropout': A.CoarseDropout(max_holes=10, max_height=20, max_width=20, \n                                             min_holes=None, min_height=None, min_width=None,\n                                             fill_value=0, always_apply=False, p=1),\n            \n            'GridDropout': A.GridDropout(ratio=.4, unit_size_min=None, unit_size_max=None,\n                                         holes_number_x=None, holes_number_y=None,\n                                         shift_x=0, shift_y=0, p=1)}\n\nshow_images(aug_dict)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Now this probably isn't that useful, but it is cool so I will demo it anyway. We can add some  weather augmentations, which doesn't make much sense to do with the `py_semantic` raster images, but I guess one could use it for `py_satellite` raster images.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"aug_dict = {'Original': None,\n            \n            'RandomRain': A.RandomRain(slant_lower=-10, slant_upper=10, drop_length=20,\n                                      drop_width=1,blur_value=7, brightness_coefficient=0.7,\n                                      rain_type=None, always_apply=False, p=1),\n            \n            'RandomFog': A.RandomFog(fog_coef_lower=0.3, fog_coef_upper=1, alpha_coef=0.08,\n                                     always_apply=False, p=1),\n            \n            'RandomSnow': A.RandomSnow(snow_point_lower=0.1, snow_point_upper=0.3,\n                                        brightness_coeff=2.5, always_apply=False, p=1)}\n\nshow_images(aug_dict)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# To be continued..."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}