{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Lyft: Training with multi-mode confidence\n\n\nThe image from L5Kit official document: http://www.l5kit.org/README.html\n\nContinued from the previous kernel:\n - [Lyft: Comprehensive guide to start competition](https://www.kaggle.com/corochann/lyft-comprehensive-guide-to-start-competition)\n - [Lyft: Deep into the l5kit library](https://www.kaggle.com/corochann/lyft-deep-into-the-l5kit-library)\n\nIn this kernel, I will run **pytorch CNN model training**. Especially, followings are new items to try:\n - Predict **multi-mode with confidence**: As written in [evaluation metric](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/overview/evaluation) page, we can predict **3 modes** of motion trajectory.\n - Training loss with **competition evaluation metric**\n - Use Training abstraction library **`pytorch-ignite` and `pytorch-pfn-extras`**.\n\n\n[Update 2020/9/6]
\nPublished prediction kernel: [Lyft: Prediction with multi-mode confidence](https://www.kaggle.com/corochann/lyft-prediction-with-multi-mode-confidence)
\nTry yourself how good score you can get using only single model without ensemble! :)"},{"metadata":{},"cell_type":"markdown","source":"# Environment setup\n\n - Please add [pestipeti/lyft-l5kit-unofficial-fix](https://www.kaggle.com/pestipeti/lyft-l5kit-unofficial-fix) as utility script.\n - Official utility script \"[philculliton/kaggle-l5kit](https://www.kaggle.com/mathurinache/kaggle-l5kit)\" does not work with pytorch GPU.\n - Please add [lyft-config-files](https://www.kaggle.com/jpbremer/lyft-config-files) as dataset\n \nSee previous kernel [Lyft: Comprehensive guide to start competition](https://www.kaggle.com/corochann/lyft-comprehensive-guide-to-start-competition) for details."},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# https://github.com/pfnet/pytorch-pfn-extras/releases/tag/v0.3.1\n!pip install pytorch-pfn-extras==0.3.1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import gc\nimport os\nfrom pathlib import Path\nimport random\nimport sys\n\nfrom tqdm.notebook import tqdm\nimport numpy as np\nimport pandas as pd\nimport scipy as sp\n\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom IPython.core.display import display, HTML\n\n# --- plotly ---\nfrom plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.express as px\nimport plotly.figure_factory as ff\nimport plotly.io as pio\npio.templates.default = \"plotly_dark\"\n\n# --- models ---\nfrom sklearn import preprocessing\nfrom sklearn.model_selection import KFold\nimport lightgbm as lgb\nimport xgboost as xgb\nimport catboost as cb\n\n# --- setup ---\npd.set_option('max_columns', 50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import zarr\n\nimport l5kit\nfrom l5kit.data import ChunkedDataset, LocalDataManager\nfrom l5kit.dataset import EgoDataset, AgentDataset\n\nfrom l5kit.rasterization import build_rasterizer\nfrom l5kit.configs import load_config_data\nfrom l5kit.visualization import draw_trajectory, TARGET_POINTS_COLOR\nfrom l5kit.geometry import transform_points\nfrom tqdm import tqdm\nfrom collections import Counter\nfrom l5kit.data import PERCEPTION_LABELS\nfrom prettytable import PrettyTable\n\nfrom matplotlib import animation, rc\nfrom IPython.display import HTML\n\nrc('animation', html='jshtml')\nprint(\"l5kit version:\", l5kit.__version__)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import torch\nfrom pathlib import Path\n\nimport pytorch_pfn_extras as ppe\nfrom math import ceil\nfrom pytorch_pfn_extras.training import IgniteExtensionsManager\nfrom pytorch_pfn_extras.training.triggers import MinValueTrigger\n\nfrom torch import nn, optim\nfrom torch.utils.data import DataLoader\nfrom torch.utils.data.dataset import Subset\nimport pytorch_pfn_extras.training.extensions as E","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# --- Dataset utils ---\nfrom typing import Callable\n\nfrom torch.utils.data.dataset import Dataset\n\n\nclass TransformDataset(Dataset):\n def __init__(self, dataset: Dataset, transform: Callable):\n self.dataset = dataset\n self.transform = transform\n\n def __getitem__(self, index):\n batch = self.dataset[index]\n return self.transform(batch)\n\n def __len__(self):\n return len(self.dataset)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Function\n\nTo define loss function to calculate competition evaluation metric **in batch**.
\nIt works with **pytorch tensor, so it is differentiable** and can be used for training Neural Network."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# --- Function utils ---\n# Original code from https://github.com/lyft/l5kit/blob/20ab033c01610d711c3d36e1963ecec86e8b85b6/l5kit/l5kit/evaluation/metrics.py\nimport numpy as np\n\nimport torch\nfrom torch import Tensor\n\n\ndef pytorch_neg_multi_log_likelihood_batch(\n gt: Tensor, pred: Tensor, confidences: Tensor, avails: Tensor\n) -> Tensor:\n \"\"\"\n Compute a negative log-likelihood for the multi-modal scenario.\n log-sum-exp trick is used here to avoid underflow and overflow, For more information about it see:\n https://en.wikipedia.org/wiki/LogSumExp#log-sum-exp_trick_for_log-domain_calculations\n https://timvieira.github.io/blog/post/2014/02/11/exp-normalize-trick/\n https://leimao.github.io/blog/LogSumExp/\n Args:\n gt (Tensor): array of shape (bs)x(time)x(2D coords)\n pred (Tensor): array of shape (bs)x(modes)x(time)x(2D coords)\n confidences (Tensor): array of shape (bs)x(modes) with a confidence for each mode in each sample\n avails (Tensor): array of shape (bs)x(time) with the availability for each gt timestep\n Returns:\n Tensor: negative log-likelihood for this example, a single float number\n \"\"\"\n assert len(pred.shape) == 4, f\"expected 3D (MxTxC) array for pred, got {pred.shape}\"\n batch_size, num_modes, future_len, num_coords = pred.shape\n\n assert gt.shape == (batch_size, future_len, num_coords), f\"expected 2D (Time x Coords) array for gt, got {gt.shape}\"\n assert confidences.shape == (batch_size, num_modes), f\"expected 1D (Modes) array for gt, got {confidences.shape}\"\n assert torch.allclose(torch.sum(confidences, dim=1), confidences.new_ones((batch_size,))), \"confidences should sum to 1\"\n assert avails.shape == (batch_size, future_len), f\"expected 1D (Time) array for gt, got {avails.shape}\"\n # assert all data are valid\n assert torch.isfinite(pred).all(), \"invalid value found in pred\"\n assert torch.isfinite(gt).all(), \"invalid value found in gt\"\n assert torch.isfinite(confidences).all(), \"invalid value found in confidences\"\n assert torch.isfinite(avails).all(), \"invalid value found in avails\"\n\n # convert to (batch_size, num_modes, future_len, num_coords)\n gt = torch.unsqueeze(gt, 1) # add modes\n avails = avails[:, None, :, None] # add modes and cords\n\n # error (batch_size, num_modes, future_len)\n error = torch.sum(((gt - pred) * avails) ** 2, dim=-1) # reduce coords and use availability\n\n with np.errstate(divide=\"ignore\"): # when confidence is 0 log goes to -inf, but we're fine with it\n # error (batch_size, num_modes)\n error = torch.log(confidences) - 0.5 * torch.sum(error, dim=-1) # reduce time\n\n # use max aggregator on modes for numerical stability\n # error (batch_size, num_modes)\n max_value, _ = error.max(dim=1, keepdim=True) # error are negative at this point, so max() gives the minimum one\n error = -torch.log(torch.sum(torch.exp(error - max_value), dim=-1, keepdim=True)) - max_value # reduce modes\n # print(\"error\", error)\n return torch.mean(error)\n\n\ndef pytorch_neg_multi_log_likelihood_single(\n gt: Tensor, pred: Tensor, avails: Tensor\n) -> Tensor:\n \"\"\"\n\n Args:\n gt (Tensor): array of shape (bs)x(time)x(2D coords)\n pred (Tensor): array of shape (bs)x(time)x(2D coords)\n avails (Tensor): array of shape (bs)x(time) with the availability for each gt timestep\n Returns:\n Tensor: negative log-likelihood for this example, a single float number\n \"\"\"\n # pred (bs)x(time)x(2D coords) --> (bs)x(mode=1)x(time)x(2D coords)\n # create confidence (bs)x(mode=1)\n batch_size, future_len, num_coords = pred.shape\n confidences = pred.new_ones((batch_size, 1))\n return pytorch_neg_multi_log_likelihood_batch(gt, pred.unsqueeze(1), confidences, avails)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model\n\npytorch model definition. Here model outputs both **multi-mode trajectory prediction & confidence of each trajectory**."},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"# --- Model utils ---\nimport torch\nfrom torchvision.models import resnet18\nfrom torch import nn\nfrom typing import Dict\n\n\nclass LyftMultiModel(nn.Module):\n\n def __init__(self, cfg: Dict, num_modes=3):\n super().__init__()\n\n # TODO: support other than resnet18?\n backbone = resnet18(pretrained=True, progress=True)\n self.backbone = backbone\n\n num_history_channels = (cfg[\"model_params\"][\"history_num_frames\"] + 1) * 2\n num_in_channels = 3 + num_history_channels\n\n self.backbone.conv1 = nn.Conv2d(\n num_in_channels,\n self.backbone.conv1.out_channels,\n kernel_size=self.backbone.conv1.kernel_size,\n stride=self.backbone.conv1.stride,\n padding=self.backbone.conv1.padding,\n bias=False,\n )\n\n # This is 512 for resnet18 and resnet34;\n # And it is 2048 for the other resnets\n backbone_out_features = 512\n\n # X, Y coords for the future positions (output shape: Bx50x2)\n self.future_len = cfg[\"model_params\"][\"future_num_frames\"]\n num_targets = 2 * self.future_len\n\n # You can add more layers here.\n self.head = nn.Sequential(\n # nn.Dropout(0.2),\n nn.Linear(in_features=backbone_out_features, out_features=4096),\n )\n\n self.num_preds = num_targets * num_modes\n self.num_modes = num_modes\n\n self.logit = nn.Linear(4096, out_features=self.num_preds + num_modes)\n\n def forward(self, x):\n x = self.backbone.conv1(x)\n x = self.backbone.bn1(x)\n x = self.backbone.relu(x)\n x = self.backbone.maxpool(x)\n\n x = self.backbone.layer1(x)\n x = self.backbone.layer2(x)\n x = self.backbone.layer3(x)\n x = self.backbone.layer4(x)\n\n x = self.backbone.avgpool(x)\n x = torch.flatten(x, 1)\n\n x = self.head(x)\n x = self.logit(x)\n\n # pred (bs)x(modes)x(time)x(2D coords)\n # confidences (bs)x(modes)\n bs, _ = x.shape\n pred, confidences = torch.split(x, self.num_preds, dim=1)\n pred = pred.view(bs, self.num_modes, self.future_len, 2)\n assert confidences.shape == (bs, self.num_modes)\n confidences = torch.softmax(confidences, dim=1)\n return pred, confidences\n\n ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class LyftMultiRegressor(nn.Module):\n \"\"\"Single mode prediction\"\"\"\n\n def __init__(self, predictor, lossfun=pytorch_neg_multi_log_likelihood_batch):\n super().__init__()\n self.predictor = predictor\n self.lossfun = lossfun\n\n def forward(self, image, targets, target_availabilities):\n pred, confidences = self.predictor(image)\n loss = self.lossfun(targets, pred, confidences, target_availabilities)\n metrics = {\n \"loss\": loss.item(),\n \"nll\": pytorch_neg_multi_log_likelihood_batch(targets, pred, confidences, target_availabilities).item()\n }\n ppe.reporting.report(metrics, self)\n return loss, metrics\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training with Ignite\n\nI use [pytorch-ignite](https://github.com/pytorch/ignite) for training abstraction.
\n`Engine` defines the 1 iteration of training update.\n\n - [Official Document: IGNITE YOUR NETWORKS!](https://pytorch.org/ignite/)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# --- Training utils ---\nfrom ignite.engine import Engine\n\n\ndef create_trainer(model, optimizer, device) -> Engine:\n model.to(device)\n\n def update_fn(engine, batch):\n model.train()\n optimizer.zero_grad()\n loss, metrics = model(*[elem.to(device) for elem in batch])\n loss.backward()\n optimizer.step()\n return metrics\n trainer = Engine(update_fn)\n return trainer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# --- Utils ---\nimport yaml\n\n\ndef save_yaml(filepath, content, width=120):\n with open(filepath, 'w') as f:\n yaml.dump(content, f, width=width)\n\n\ndef load_yaml(filepath):\n with open(filepath, 'r') as f:\n content = yaml.safe_load(f)\n return content\n\n\nclass DotDict(dict):\n \"\"\"dot.notation access to dictionary attributes\n\n Refer: https://stackoverflow.com/questions/2352181/how-to-use-a-dot-to-access-members-of-dictionary/23689767#23689767\n \"\"\" # NOQA\n\n __getattr__ = dict.get\n __setattr__ = dict.__setitem__\n __delattr__ = dict.__delitem__\n\n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Configs"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# --- Lyft configs ---\ncfg = {\n 'format_version': 4,\n 'model_params': {\n 'model_architecture': 'resnet50',\n 'history_num_frames': 10,\n 'history_step_size': 1,\n 'history_delta_time': 0.1,\n 'future_num_frames': 50,\n 'future_step_size': 1,\n 'future_delta_time': 0.1\n },\n\n 'raster_params': {\n 'raster_size': [224, 224],\n 'pixel_size': [0.5, 0.5],\n 'ego_center': [0.25, 0.5],\n 'map_type': 'py_semantic',\n 'satellite_map_key': 'aerial_map/aerial_map.png',\n 'semantic_map_key': 'semantic_map/semantic_map.pb',\n 'dataset_meta_key': 'meta.json',\n 'filter_agents_threshold': 0.5\n },\n\n 'train_data_loader': {\n 'key': 'scenes/train.zarr',\n 'batch_size': 12,\n 'shuffle': True,\n 'num_workers': 4\n },\n\n 'valid_data_loader': {\n 'key': 'scenes/validate.zarr',\n 'batch_size': 32,\n 'shuffle': False,\n 'num_workers': 4\n },\n\n 'train_params': {\n 'max_num_steps': 10000,\n 'checkpoint_every_n_steps': 5000,\n\n # 'eval_every_n_steps': -1\n }\n}\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"flags_dict = {\n \"debug\": True,\n # --- Data configs ---\n \"l5kit_data_folder\": \"/kaggle/input/lyft-motion-prediction-autonomous-vehicles\",\n # --- Model configs ---\n \"pred_mode\": \"multi\",\n # --- Training configs ---\n \"device\": \"cuda:0\",\n \"out_dir\": \"results/multi_train\",\n \"epoch\": 2,\n \"snapshot_freq\": 50,\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Main script\n\nNow finished defining all the util codes. Let's start writing main script to train the model!"},{"metadata":{},"cell_type":"markdown","source":"## Loading data\n\nHere we will only use the first dataset from the sample set. (sample.zarr data is used for visualization, please use train.zarr / validate.zarr / test.zarr for actual model training/validation/prediction.)
\nWe're building a `LocalDataManager` object. This will resolve relative paths from the config using the `L5KIT_DATA_FOLDER` env variable we have just set."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"flags = DotDict(flags_dict)\nout_dir = Path(flags.out_dir)\nos.makedirs(str(out_dir), exist_ok=True)\nprint(f\"flags: {flags_dict}\")\nsave_yaml(out_dir / 'flags.yaml', flags_dict)\nsave_yaml(out_dir / 'cfg.yaml', cfg)\ndebug = flags.debug\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"# set env variable for data\nos.environ[\"L5KIT_DATA_FOLDER\"] = flags.l5kit_data_folder\ndm = LocalDataManager(None)\n\nprint(\"Load dataset...\")\ntrain_cfg = cfg[\"train_data_loader\"]\nvalid_cfg = cfg[\"valid_data_loader\"]\n\n# Rasterizer\nrasterizer = build_rasterizer(cfg, dm)\n\n# Train dataset/dataloader\ndef transform(batch):\n return batch[\"image\"], batch[\"target_positions\"], batch[\"target_availabilities\"]\n\n\ntrain_path = \"scenes/sample.zarr\" if debug else train_cfg[\"key\"]\ntrain_zarr = ChunkedDataset(dm.require(train_path)).open()\nprint(\"train_zarr\", type(train_zarr))\ntrain_agent_dataset = AgentDataset(cfg, train_zarr, rasterizer)\ntrain_dataset = TransformDataset(train_agent_dataset, transform)\nif debug:\n # Only use 1000 dataset for fast check...\n train_dataset = Subset(train_dataset, np.arange(1000))\ntrain_loader = DataLoader(train_dataset,\n shuffle=train_cfg[\"shuffle\"],\n batch_size=train_cfg[\"batch_size\"],\n num_workers=train_cfg[\"num_workers\"])\nprint(train_agent_dataset)\n\nvalid_path = \"scenes/sample.zarr\" if debug else valid_cfg[\"key\"]\nvalid_zarr = ChunkedDataset(dm.require(valid_path)).open()\nprint(\"valid_zarr\", type(train_zarr))\nvalid_agent_dataset = AgentDataset(cfg, valid_zarr, rasterizer)\nvalid_dataset = TransformDataset(valid_agent_dataset, transform)\nif debug:\n # Only use 100 dataset for fast check...\n valid_dataset = Subset(valid_dataset, np.arange(100))\nelse:\n # Only use 1000 dataset for fast check...\n valid_dataset = Subset(valid_dataset, np.arange(1000))\nvalid_loader = DataLoader(\n valid_dataset,\n shuffle=valid_cfg[\"shuffle\"],\n batch_size=valid_cfg[\"batch_size\"],\n num_workers=valid_cfg[\"num_workers\"]\n)\n\nprint(valid_agent_dataset)\nprint(\"# AgentDataset train:\", len(train_agent_dataset), \"#valid\", len(valid_agent_dataset))\nprint(\"# ActualDataset train:\", len(train_dataset), \"#valid\", len(valid_dataset))\n# AgentDataset train: 22496709 #valid 21624612\n# ActualDataset train: 100 #valid 100","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare model & optimizer"},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(flags.device)\n\nif flags.pred_mode == \"multi\":\n predictor = LyftMultiModel(cfg)\n model = LyftMultiRegressor(predictor)\nelse:\n raise ValueError(f\"[ERROR] Unexpected value flags.pred_mode={flags.pred_mode}\")\n\nmodel.to(device)\noptimizer = optim.Adam(model.parameters(), lr=1e-3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Write training code\n\npytorch-ignite & pytorch-pfn-extras are used here.\n\n - [pytorch/ignite](https://github.com/pytorch/ignite): It provides abstraction for writing training loop.\n - [pfnet/pytorch-pfn-extras](https://github.com/pfnet/pytorch-pfn-extras): It provides several \"extensions\" useful for training. Useful for **logging, printing, evaluating, saving the model, scheduling the learning rate** during training.\n \n**[Note] Why training abstraction library is used?**\n\nYou may feel understanding training abstraction code below is a bit unintuitive compared to writing \"raw\" training loop.
\nThe advantage of abstracting the code is that we can re-use implemented handler class for other training, other competition.
\nYou don't need to write code for saving models, logging training loss/metric, show progressbar etc.\nThese are done by provided util classes in `pytorch-pfn-extras` library!\n\nYou may refer my other kernel in previous competition too: [Bengali: SEResNeXt training with pytorch](https://www.kaggle.com/corochann/bengali-seresnext-training-with-pytorch)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Train setup\ntrainer = create_trainer(model, optimizer, device)\n\n\ndef eval_func(*batch):\n loss, metrics = model(*[elem.to(device) for elem in batch])\n\n\nvalid_evaluator = E.Evaluator(\n valid_loader,\n model,\n progress_bar=False,\n eval_func=eval_func,\n)\n\nlog_trigger = (10 if debug else 1000, \"iteration\")\nlog_report = E.LogReport(trigger=log_trigger)\n\n\nextensions = [\n log_report, # Save `log` to file\n valid_evaluator, # Run evaluation for valid dataset in each epoch.\n # E.FailOnNonNumber() # Stop training when nan is detected.\n E.ProgressBarNotebook(update_interval=10 if debug else 100), # Show progress bar during training\n E.PrintReportNotebook(), # Show \"log\" on jupyter notebook \n]\n\nepoch = flags.epoch\n\nmodels = {\"main\": model}\noptimizers = {\"main\": optimizer}\nmanager = IgniteExtensionsManager(\n trainer,\n models,\n optimizers,\n epoch,\n extensions=extensions,\n out_dir=str(out_dir),\n)\n# Save predictor.pt every epoch\nmanager.extend(E.snapshot_object(predictor, \"predictor.pt\"),\n trigger=(flags.snapshot_freq, \"iteration\"))\n# Check & Save best validation predictor.pt every epoch\n# manager.extend(E.snapshot_object(predictor, \"best_predictor.pt\"),\n# trigger=MinValueTrigger(\"validation/main/nll\", trigger=(flags.snapshot_freq, \"iteration\")))\n# --- lr scheduler ---\n# scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n# optimizer, mode='min', factor=0.7, patience=5, min_lr=1e-10)\nscheduler = torch.optim.lr_scheduler.ExponentialLR(\n optimizer, gamma=0.99999)\nmanager.extend(lambda manager: scheduler.step(), trigger=(1, \"iteration\"))\n# Show \"lr\" column in log\nmanager.extend(E.observe_lr(optimizer=optimizer), trigger=log_trigger)\n\n# --- HACKING to fix ProgressBarNotebook bug ---\nmanager.iteration = 0\nmanager._iters_per_epoch = len(train_loader)\n\ntrainer.run(train_loader, max_epochs=epoch)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So what is happening in above training abstraction? Let's understand what each extension did.\n\n**Extensions** - Each role:\n - **`ProgressBar` (`ProgressBarNotebook`)**: Shows training progress in formatted style.\n - **`LogReport`**: Logging metrics reported by `ppe.reporter.report` (see `LyftMultiRegressor` for reporting point) method and save to **log** file. It automatically collects reported value in each iteration and saves the \"mean\" of reported value for regular frequency (for example every 1 epoch).\n - **`PrintReport` (`PrintReportNotebook`)**: Prints the value which `LogReport` collected in formatted style.\n - **`Evaluator`**: Evaluate on validation dataset.\n - **`snapshot_object`**: Saves the object. Here the `model` is saved in regular interval `flags.snapshot_freq`. Even you quit training using Ctrl+C without finishing all the epoch, the intermediate trained model is saved and you can use it for inference.\n - **lambda function with `scheduler.step()`**: You can invoke any function in regular interval specified by `trigger`. Here exponential decay of learning rate is applied by calling `scheduler.step()` every iteration.\n - **`observe_lr`**: `LogReport` will check optimizer's learning rate using this extension. So you can follow how the learning rate changed through the training.\n\n\nSuch many functionalities can be \"added\" easily using extensions!"},{"metadata":{},"cell_type":"markdown","source":"You can obtrain training history results really easily by just accessing `LogReport` class, which is useful for managing a lot of experiments during kaggle competitions."},{"metadata":{"trusted":true},"cell_type":"code","source":"df = log_report.to_dataframe()\ndf.to_csv(out_dir/\"log.csv\", index=False)\ndf[[\"epoch\", \"iteration\", \"main/loss\", \"main/nll\", \"validation/main/loss\", \"validation/main/nll\", \"lr\", \"elapsed_time\"]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The history log and model's weight are saved by \"extensions\" (`LogReport` and `E.snapshot_object` respectively) easily, which is a benefit of using training abstration."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's see training results directory\n\n!ls results/multi_train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Items to try\n\nThis kernel shows demonstration run of the training (`debug=True`). You can try these things to see how the score changes at first\n - set debug=False to train with actual training dataset\n - change training hyperparameters (training epoch, change optimizer, scheduler learning rate etc...)\n - Especially, just training much longer time may improve the score.\n \nTo go further, these items may be nice to try:\n - Change the cnn model (now simple resnet18 is used as baseline modeling)\n - Training the model using full dataset: [lyft-full-training-set](https://www.kaggle.com/philculliton/lyft-full-training-set)\n - Write your own rasterizer to prepare input image as motivation explained in previous kernel.\n - Consider much better scheme to predict multi-trajectory\n - The model just predicts multiple trajectory at the same time in this kernel, but it is possible to collapse \"trivial\" solution where all trajectory converges to same. How to avoid this?"},{"metadata":{},"cell_type":"markdown","source":"# Next to go\n\n[Update 2020/9/6]
\nPublished prediction kernel: [Lyft: Prediction with multi-mode confidence](https://www.kaggle.com/corochann/lyft-prediction-with-multi-mode-confidence)
\nTry yourself how good score you can get using only single model without ensemble! :)\n\nTo understand the competition in more detail, please refer my other kernels too.\n - [Lyft: Comprehensive guide to start competition](https://www.kaggle.com/corochann/lyft-comprehensive-guide-to-start-competition)\n - [Lyft: Deep into the l5kit library](https://www.kaggle.com/corochann/lyft-deep-into-the-l5kit-library)\n - [Save your time, submit without kernel inference](https://www.kaggle.com/corochann/save-your-time-submit-without-kernel-inference)\n - [Lyft: pytorch implementation of evaluation metric](https://www.kaggle.com/corochann/lyft-pytorch-implementation-of-evaluation-metric)"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Further reference\n\n - Paper of this Lyft Level 5 prediction dataset: [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://arxiv.org/abs/2006.14480)\n - [jpbremer/lyft-scene-visualisations](https://www.kaggle.com/jpbremer/lyft-scene-visualisations)"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"If this kernel helps you, please upvote to keep me motivated :)
Thanks!
"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}