{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Lyft: pytorch implementation of evaluation metric\n\n\nThe image from L5Kit official document: http://www.l5kit.org/README.html\n\nIn this kernel, I will show pytorch implementaiton of [Evaluation metric](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/overview/evaluation).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## import","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import gc\nimport os\nfrom pathlib import Path\nimport random\nimport sys\n\nfrom tqdm.notebook import tqdm\nimport numpy as np\nimport pandas as pd\nimport scipy as sp\n\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom IPython.core.display import display, HTML\n\n# --- plotly ---\nfrom plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.express as px\nimport plotly.figure_factory as ff\nimport plotly.io as pio\npio.templates.default = \"plotly_dark\"\n\n# --- models ---\nfrom sklearn import preprocessing\nfrom sklearn.model_selection import KFold\nimport lightgbm as lgb\nimport xgboost as xgb\nimport catboost as cb\n\n# --- setup ---\npd.set_option('max_columns', 50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"import l5kit\nfrom l5kit.evaluation.metrics import neg_multi_log_likelihood\nprint(\"l5kit version:\", l5kit.__version__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluation metrics implementation\n\nI will be implementing `neg_multi_log_likelihood` function in pytorch, numpy version of the code is provided by l5kit:\n - [l5kit/evaluation/metrics.py](https://github.com/lyft/l5kit/blob/20ab033c01610d711c3d36e1963ecec86e8b85b6/l5kit/l5kit/evaluation/metrics.py)\n \nOf course pytorch implementation is differentiable, **you use it for loss function to directly optimize evaluation metric!**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"future_len = 50\nn_coords = 2\nn_modes = 3\n\n\ngt = np.random.uniform(-1.0, 1.0, (future_len, n_coords))\npred = np.broadcast_to(gt[None, :, :], (n_modes, future_len, n_coords)) + np.random.uniform(-0.2, 0.2, (n_modes, future_len, n_coords))\nconfidences = np.random.uniform(0.0, 1.0, (n_modes,))\nconfidences /= np.sum(confidences)\navails = (np.random.uniform(0.0, 1.0, (future_len,)) > 0.3).astype(np.float64)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Original code from https://github.com/lyft/l5kit/blob/20ab033c01610d711c3d36e1963ecec86e8b85b6/l5kit/l5kit/evaluation/metrics.py\nimport torch\nfrom torch import Tensor\n\n\ndef pytorch_neg_multi_log_likelihood(\n gt: Tensor, pred: Tensor, confidences: Tensor, avails: Tensor\n) -> Tensor:\n \"\"\"\n Compute a negative log-likelihood for the multi-modal scenario.\n log-sum-exp trick is used here to avoid underflow and overflow, For more information about it see:\n https://en.wikipedia.org/wiki/LogSumExp#log-sum-exp_trick_for_log-domain_calculations\n https://timvieira.github.io/blog/post/2014/02/11/exp-normalize-trick/\n https://leimao.github.io/blog/LogSumExp/\n Args:\n gt (Tensor): array of shape (time)x(2D coords)\n pred (Tensor): array of shape (modes)x(time)x(2D coords)\n confidences (Tensor): array of shape (modes) with a confidence for each mode in each sample\n avails (Tensor): array of shape (time) with the availability for each gt timestep\n Returns:\n Tensor: negative log-likelihood for this example, a single float number\n \"\"\"\n assert len(pred.shape) == 3, f\"expected 3D (MxTxC) array for pred, got {pred.shape}\"\n num_modes, future_len, num_coords = pred.shape\n\n assert gt.shape == (future_len, num_coords), f\"expected 2D (Time x Coords) array for gt, got {gt.shape}\"\n assert confidences.shape == (num_modes,), f\"expected 1D (Modes) array for gt, got {confidences.shape}\"\n assert abs(torch.sum(confidences).item() - 1.0) < 1e-6, \"confidences should sum to 1\"\n assert avails.shape == (future_len,), f\"expected 1D (Time) array for gt, got {avails.shape}\"\n # assert all data are valid\n assert torch.isfinite(pred).all(), \"invalid value found in pred\"\n assert torch.isfinite(gt).all(), \"invalid value found in gt\"\n assert torch.isfinite(confidences).all(), \"invalid value found in confidences\"\n assert torch.isfinite(avails).all(), \"invalid value found in avails\"\n\n gt = torch.unsqueeze(gt, 0) # add modes\n avails = avails[None, :, None] # add modes and cords\n\n error = torch.sum(((gt - pred) * avails) ** 2, dim=-1) # reduce coords and use availability\n\n with np.errstate(divide=\"ignore\"): # when confidence is 0 log goes to -inf, but we're fine with it\n error = torch.log(confidences) - 0.5 * torch.sum(error, dim=-1) # reduce time\n\n # use max aggregator on modes for numerical stability\n max_value = error.max() # error are negative at this point, so max() gives the minimum one\n error = -torch.log(torch.sum(torch.exp(error - max_value), dim=-1)) - max_value # reduce modes\n return error","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"value_numpy = neg_multi_log_likelihood(gt, pred, confidences, avails)\n\nvalue_torch = pytorch_neg_multi_log_likelihood(\n torch.tensor(gt),\n torch.tensor(pred),\n torch.tensor(confidences),\n torch.tensor(avails)\n)\n\nprint(\"value_numpy: \", value_numpy)\nprint(\"value_torch: \", value_torch)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Next to go\n\nPlease refer my other kernels for understanding the competition!\n\n - [Lyft: Comprehensive guide to start competition](https://www.kaggle.com/corochann/lyft-comprehensive-guide-to-start-competition) \n - [Lyft: Deep into the l5kit library](https://www.kaggle.com/corochann/lyft-deep-into-the-l5kit-library)\n - [Save your time, submit without kernel inference](https://www.kaggle.com/corochann/save-your-time-submit-without-kernel-inference)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"markdown","source":"If this kernel helps you, please upvote to keep me motivated :)
Thanks!
","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}