{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Lyft: Prediction with multi-mode confidence\n\n![](http://www.l5kit.org/_images/av.jpg)\n<cite>The image from L5Kit official document: <a href=\"http://www.l5kit.org/README.html\">http://www.l5kit.org/README.html</a></cite>\n\nContinued from the previous kernel:\n - [Lyft: Comprehensive guide to start competition](https://www.kaggle.com/corochann/lyft-comprehensive-guide-to-start-competition)\n - [Lyft: Deep into the l5kit library](https://www.kaggle.com/corochann/lyft-deep-into-the-l5kit-library)\n - [Lyft: Training with multi-mode confidence](https://www.kaggle.com/corochann/lyft-training-with-multi-mode-confidence)\n\nIn this kernel, I will run **prediction code for competition submission** where the model is trained in previous kernel.\n\n<span style=\"color:red\">I uploaded trained models as dataset <a href=\"https://www.kaggle.com/corochann/lyft-resnet18-baseline\">lyft-resnet18-baseline</a><br/>\nPlease <b>upvote the dataset</b> as well if this kernel helps you :)</span>"},{"metadata":{},"cell_type":"markdown","source":"# Environment setup\n\n - Please add [pestipeti/lyft-l5kit-unofficial-fix](https://www.kaggle.com/pestipeti/lyft-l5kit-unofficial-fix) as utility script.\n    - Official utility script \"[philculliton/kaggle-l5kit](https://www.kaggle.com/mathurinache/kaggle-l5kit)\" does not work with pytorch GPU.\n - Please add dataset:\n    - [lyft-config-files](https://www.kaggle.com/jpbremer/lyft-config-files)\n    - [lyft-resnet18-baseline](https://www.kaggle.com/corochann/lyft-resnet18-baseline)\n \nSee previous kernel [Lyft: Comprehensive guide to start competition](https://www.kaggle.com/corochann/lyft-comprehensive-guide-to-start-competition) for details."},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install pytorch-pfn-extras==0.3.1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import gc\nimport os\nfrom pathlib import Path\nimport random\nimport sys\n\nfrom tqdm.notebook import tqdm\nimport numpy as np\nimport pandas as pd\nimport scipy as sp\n\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom IPython.core.display import display, HTML\n\n# --- plotly ---\nfrom plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.express as px\nimport plotly.figure_factory as ff\nimport plotly.io as pio\npio.templates.default = \"plotly_dark\"\n\n# --- models ---\nfrom sklearn import preprocessing\nfrom sklearn.model_selection import KFold\nimport lightgbm as lgb\nimport xgboost as xgb\nimport catboost as cb\n\n# --- setup ---\npd.set_option('max_columns', 50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import zarr\n\nimport l5kit\nfrom l5kit.data import ChunkedDataset, LocalDataManager\nfrom l5kit.dataset import EgoDataset, AgentDataset\n\nfrom l5kit.rasterization import build_rasterizer\nfrom l5kit.configs import load_config_data\nfrom l5kit.visualization import draw_trajectory, TARGET_POINTS_COLOR\nfrom l5kit.geometry import transform_points\nfrom tqdm import tqdm\nfrom collections import Counter\nfrom l5kit.data import PERCEPTION_LABELS\nfrom prettytable import PrettyTable\nfrom l5kit.evaluation import write_pred_csv\n\nfrom matplotlib import animation, rc\nfrom IPython.display import HTML\n\nrc('animation', html='jshtml')\nprint(\"l5kit version:\", l5kit.__version__)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import torch\nfrom pathlib import Path\n\nimport pytorch_pfn_extras as ppe\nfrom math import ceil\nfrom pytorch_pfn_extras.training import IgniteExtensionsManager\nfrom pytorch_pfn_extras.training.triggers import MinValueTrigger\n\nfrom torch import nn, optim\nfrom torch.utils.data import DataLoader\nfrom torch.utils.data.dataset import Subset\nimport pytorch_pfn_extras.training.extensions as E","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model\n\npytorch model definition. Here model outputs both **multi-mode trajectory prediction & confidence of each trajectory**."},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"# --- Model utils ---\nimport torch\nfrom torchvision.models import resnet18\nfrom torch import nn\nfrom typing import Dict\n\n\nclass LyftMultiModel(nn.Module):\n\n    def __init__(self, cfg: Dict, num_modes=3):\n        super().__init__()\n\n        # TODO: support other than resnet18?\n        backbone = resnet18(pretrained=True, progress=True)\n        self.backbone = backbone\n\n        num_history_channels = (cfg[\"model_params\"][\"history_num_frames\"] + 1) * 2\n        num_in_channels = 3 + num_history_channels\n\n        self.backbone.conv1 = nn.Conv2d(\n            num_in_channels,\n            self.backbone.conv1.out_channels,\n            kernel_size=self.backbone.conv1.kernel_size,\n            stride=self.backbone.conv1.stride,\n            padding=self.backbone.conv1.padding,\n            bias=False,\n        )\n\n        # This is 512 for resnet18 and resnet34;\n        # And it is 2048 for the other resnets\n        backbone_out_features = 512\n\n        # X, Y coords for the future positions (output shape: Bx50x2)\n        self.future_len = cfg[\"model_params\"][\"future_num_frames\"]\n        num_targets = 2 * self.future_len\n\n        # You can add more layers here.\n        self.head = nn.Sequential(\n            # nn.Dropout(0.2),\n            nn.Linear(in_features=backbone_out_features, out_features=4096),\n        )\n\n        self.num_preds = num_targets * num_modes\n        self.num_modes = num_modes\n\n        self.logit = nn.Linear(4096, out_features=self.num_preds + num_modes)\n\n    def forward(self, x):\n        x = self.backbone.conv1(x)\n        x = self.backbone.bn1(x)\n        x = self.backbone.relu(x)\n        x = self.backbone.maxpool(x)\n\n        x = self.backbone.layer1(x)\n        x = self.backbone.layer2(x)\n        x = self.backbone.layer3(x)\n        x = self.backbone.layer4(x)\n\n        x = self.backbone.avgpool(x)\n        x = torch.flatten(x, 1)\n\n        x = self.head(x)\n        x = self.logit(x)\n\n        # pred (bs)x(modes)x(time)x(2D coords)\n        # confidences (bs)x(modes)\n        bs, _ = x.shape\n        pred, confidences = torch.split(x, self.num_preds, dim=1)\n        pred = pred.view(bs, self.num_modes, self.future_len, 2)\n        assert confidences.shape == (bs, self.num_modes)\n        confidences = torch.softmax(confidences, dim=1)\n        return pred, confidences\n\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# --- Utils ---\nimport yaml\n\n\ndef save_yaml(filepath, content, width=120):\n    with open(filepath, 'w') as f:\n        yaml.dump(content, f, width=width)\n\n\ndef load_yaml(filepath):\n    with open(filepath, 'r') as f:\n        content = yaml.safe_load(f)\n    return content\n\n\nclass DotDict(dict):\n    \"\"\"dot.notation access to dictionary attributes\n\n    Refer: https://stackoverflow.com/questions/2352181/how-to-use-a-dot-to-access-members-of-dictionary/23689767#23689767\n    \"\"\"  # NOQA\n\n    __getattr__ = dict.get\n    __setattr__ = dict.__setitem__\n    __delattr__ = dict.__delitem__\n\n    ","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Referred https://www.kaggle.com/pestipeti/pytorch-baseline-inference\ndef run_prediction(predictor, data_loader):\n    predictor.eval()\n\n    pred_coords_list = []\n    confidences_list = []\n    timestamps_list = []\n    track_id_list = []\n\n    with torch.no_grad():\n        dataiter = tqdm(data_loader)\n        for data in dataiter:\n            image = data[\"image\"].to(device)\n            # target_availabilities = data[\"target_availabilities\"].to(device)\n            # targets = data[\"target_positions\"].to(device)\n            pred, confidences = predictor(image)\n\n            pred_coords_list.append(pred.cpu().numpy().copy())\n            confidences_list.append(confidences.cpu().numpy().copy())\n            timestamps_list.append(data[\"timestamp\"].numpy().copy())\n            track_id_list.append(data[\"track_id\"].numpy().copy())\n    timestamps = np.concatenate(timestamps_list)\n    track_ids = np.concatenate(track_id_list)\n    coords = np.concatenate(pred_coords_list)\n    confs = np.concatenate(confidences_list)\n    return timestamps, track_ids, coords, confs\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Configs"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# --- Lyft configs ---\ncfg = {\n    'format_version': 4,\n    'model_params': {\n        'model_architecture': 'resnet50',\n        'history_num_frames': 10,\n        'history_step_size': 1,\n        'history_delta_time': 0.1,\n        'future_num_frames': 50,\n        'future_step_size': 1,\n        'future_delta_time': 0.1\n    },\n\n    'raster_params': {\n        'raster_size': [224, 224],\n        'pixel_size': [0.5, 0.5],\n        'ego_center': [0.25, 0.5],\n        'map_type': 'py_semantic',\n        'satellite_map_key': 'aerial_map/aerial_map.png',\n        'semantic_map_key': 'semantic_map/semantic_map.pb',\n        'dataset_meta_key': 'meta.json',\n        'filter_agents_threshold': 0.5\n    },\n\n    'train_data_loader': {\n        'key': 'scenes/train.zarr',\n        'batch_size': 12,\n        'shuffle': True,\n        'num_workers': 4\n    },\n\n    'valid_data_loader': {\n        'key': 'scenes/validate.zarr',\n        'batch_size': 32,\n        'shuffle': False,\n        'num_workers': 4\n    },\n    \n    'test_data_loader': {\n        'key': 'scenes/test.zarr',\n        'batch_size': 8,\n        'shuffle': False,\n        'num_workers': 4\n    },\n\n    'train_params': {\n        'max_num_steps': 10000,\n        'checkpoint_every_n_steps': 5000,\n\n        # 'eval_every_n_steps': -1\n    }\n}\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"flags_dict = {\n    \"debug\": False,\n    # --- Data configs ---\n    \"l5kit_data_folder\": \"/kaggle/input/lyft-motion-prediction-autonomous-vehicles\",\n    # --- Model configs ---\n    \"pred_mode\": \"multi\",\n    # --- Training configs ---\n    \"device\": \"cuda:0\",\n    \"out_dir\": \"results/multi_train\",\n    \"epoch\": 2,\n    \"snapshot_freq\": 50,\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Main script\n\nNow finished defining all the util codes. Let's start writing main script to train the model!"},{"metadata":{},"cell_type":"markdown","source":"## Loading data\n\nHere we will only use the first dataset from the sample set. (sample.zarr data is used for visualization, please use train.zarr / validate.zarr / test.zarr for actual model training/validation/prediction.)<br/>\nWe're building a `LocalDataManager` object. This will resolve relative paths from the config using the `L5KIT_DATA_FOLDER` env variable we have just set."},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"flags = DotDict(flags_dict)\nout_dir = Path(flags.out_dir)\nos.makedirs(str(out_dir), exist_ok=True)\nprint(f\"flags: {flags_dict}\")\nsave_yaml(out_dir / 'flags.yaml', flags_dict)\nsave_yaml(out_dir / 'cfg.yaml', cfg)\ndebug = flags.debug","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# set env variable for data\nl5kit_data_folder = \"/kaggle/input/lyft-motion-prediction-autonomous-vehicles\"\nos.environ[\"L5KIT_DATA_FOLDER\"] = l5kit_data_folder\ndm = LocalDataManager(None)\n\nprint(\"Load dataset...\")\ndefault_test_cfg = {\n    'key': 'scenes/test.zarr',\n    'batch_size': 32,\n    'shuffle': False,\n    'num_workers': 4\n}\ntest_cfg = cfg.get(\"test_data_loader\", default_test_cfg)\n\n# Rasterizer\nrasterizer = build_rasterizer(cfg, dm)\n\ntest_path = test_cfg[\"key\"]\nprint(f\"Loading from {test_path}\")\ntest_zarr = ChunkedDataset(dm.require(test_path)).open()\nprint(\"test_zarr\", type(test_zarr))\ntest_mask = np.load(f\"{l5kit_data_folder}/scenes/mask.npz\")[\"arr_0\"]\ntest_agent_dataset = AgentDataset(cfg, test_zarr, rasterizer, agents_mask=test_mask)\ntest_dataset = test_agent_dataset\nif debug:\n    # Only use 100 dataset for fast check...\n    test_dataset = Subset(test_dataset, np.arange(100))\ntest_loader = DataLoader(\n    test_dataset,\n    shuffle=test_cfg[\"shuffle\"],\n    batch_size=test_cfg[\"batch_size\"],\n    num_workers=test_cfg[\"num_workers\"],\n    pin_memory=True,\n)\n\nprint(test_agent_dataset)\nprint(\"# AgentDataset test:\", len(test_agent_dataset))\nprint(\"# ActualDataset test:\", len(test_dataset))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare model & optimizer"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"device = torch.device(flags.device)\n\nif flags.pred_mode == \"multi\":\n    predictor = LyftMultiModel(cfg)\nelse:\n    raise ValueError(f\"[ERROR] Unexpected value flags.pred_mode={flags.pred_mode}\")\n\npt_path = \"/kaggle/input/lyft-resnet18-baseline/0918_predictor_full.pt\"\nprint(f\"Loading from {pt_path}\")\npredictor.load_state_dict(torch.load(pt_path))\npredictor.to(device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference!"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"# --- Inference ---\ntimestamps, track_ids, coords, confs = run_prediction(predictor, test_loader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"csv_path = \"submission.csv\"\nwrite_pred_csv(\n    csv_path,\n    timestamps=timestamps,\n    track_ids=track_ids,\n    coords=coords,\n    confs=confs)\nprint(f\"Saved to {csv_path}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Next to go\n\nTo understand the competition in more detail, please refer my other kernels too.\n - [Lyft: Comprehensive guide to start competition](https://www.kaggle.com/corochann/lyft-comprehensive-guide-to-start-competition)\n - [Lyft: Deep into the l5kit library](https://www.kaggle.com/corochann/lyft-deep-into-the-l5kit-library)\n - [Save your time, submit without kernel inference](https://www.kaggle.com/corochann/save-your-time-submit-without-kernel-inference)\n - [Lyft: pytorch implementation of evaluation metric](https://www.kaggle.com/corochann/lyft-pytorch-implementation-of-evaluation-metric)\n - [Lyft: Training with multi-mode confidence](https://www.kaggle.com/corochann/lyft-training-with-multi-mode-confidence)\n\nDiscussions:\n - [Is this the way how the autonomous car predicts motions?](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/179236)\n - [How to ensemble multi-trajectory predictions?](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/180931)"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Further reference\n\n - Paper of this Lyft Level 5 prediction dataset: [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://arxiv.org/abs/2006.14480)\n - [jpbremer/lyft-scene-visualisations](https://www.kaggle.com/jpbremer/lyft-scene-visualisations)"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"<h3 style=\"color:red\">If this kernel helps you, please upvote to keep me motivated :)<br>Thanks!</h3>"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}