{
  "id": 194025,
  "title": "Validation procedure",
  "url": "/competitions/lyft-motion-prediction-autonomous-vehicles/discussion/194025",
  "author_name": "",
  "post_date": "2020-10-30T09:16:23.274260500Z",
  "votes": 4,
  "comment_count": 6,
  "views": 0,
  "content": "<p>I am using the <a href=\"https://www.kaggle.com/louis925/lyft-complete-train-and-prediction-pipeline\" target=\"_blank\">Lyft: Complete train and prediction pipeline</a>. The loss function and most of the code is the same, and after the training, I got 28.31 loss, and after the submission, I get around 27, which means for me that loss calculation is ok. I tried to add the validation procedure to it. In the as they suggested <a href=\"https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/185762\" target=\"_blank\">this post</a>. I generated a dataset with this code, which is based on <a href=\"https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb\" target=\"_blank\">agent_motion_prediction.ipynb</a></p>\n<pre><code>'validate_data_loader': {\n    'key': 'scenes/validate.zarr',\n    'batch_size': 16,\n    'shuffle': False,\n    'num_workers': 4,\n}\n\n...\n\nnum_frames_to_chop = 100\neval_cfg = cfg[\"validate_data_loader\"]\neval_base_path = create_chopped_dataset(dm.require(eval_cfg[\"key\"]), cfg[\"raster_params\"][\"filter_agents_threshold\"], num_frames_to_chop, \ncfg[\"model_params\"][\"future_num_frames\"], MIN_FUTURE_STEPS)\n\neval_zarr_path = str(Path(eval_base_path) / Path(dm.require(eval_cfg[\"key\"])).name)\neval_mask_path = str(Path(eval_base_path) / \"mask.npz\")\neval_gt_path = str(Path(eval_base_path) / \"gt.csv\")\n\neval_zarr = ChunkedDataset(eval_zarr_path).open()\neval_mask = np.load(eval_mask_path)[\"arr_0\"]\n# ===== INIT DATASET AND LOAD MASK\neval_dataset = AgentDataset(cfg, eval_zarr, rasterizer, agents_mask=eval_mask)\neval_dataloader = DataLoader(eval_dataset, shuffle=eval_cfg[\"shuffle\"], batch_size=eval_cfg[\"batch_size\"], \n                             num_workers=eval_cfg[\"num_workers\"])\n</code></pre>\n<p>After that, I updated the <code>cfg[\"validate_data_loader\"][\"key\"]</code> to <code>scenes/validate_chopped_100/validate.zarr</code> to load the chopped dataset. As I understand, the <code>print(eval_dataset)</code> gives correct output, every scene, has 100 frames.</p>\n<pre><code>+------------+------------+------------+---------------+-----------------+----------------------+----------------------+----------------------+---------------------+\n| Num Scenes | Num Frames | Num Agents | Num TR lights | Total Time (hr) | Avg Frames per Scene | Avg Agents per Frame | Avg Scene Time (sec) | Avg Frame frequency |\n+------------+------------+------------+---------------+-----------------+----------------------+----------------------+----------------------+---------------------+\n|   16220    |  1622000   | 125423254  |    11733321   |      45.06      |        100.00        |        77.33         |        10.00         |        10.00        |\n+------------+------------+------------+---------------+-----------------+----------------------+----------------------+----------------------+---------------------+\n</code></pre>\n<p>I tried to evaluate my model whit this code, and give a incorrect result, which is less than one: <code>loss: 1.292e-08 loss(avg): 2.922e-07</code></p>\n<pre><code># ==== EVAL LOOP\nmodel.eval()\ntorch.set_grad_enabled(False)\nlosses_valid = []\nprogress_bar = tqdm(eval_dataloader)\n\nfor data in progress_bar:\n    loss, _, _ = forward(data, model, device)\n    losses_valid.append(loss.item())   \n    progress_bar.set_description(f\"loss: {loss.item()} loss(avg): {np.mean(losses_valid)}\")\n</code></pre>\n<p>After that, based on the <a href=\"https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb\" target=\"_blank\">agent_motion_prediction.ipynb</a> suggestion:</p>\n<blockquote>\n  <p>The result is that <strong>each scene has been reduced to only 100 frames</strong>, and <strong>only valid agents in the 100th frame will be used to compute the metrics</strong>. </p>\n</blockquote>\n<p>I tried to evaluate every 100th of frame, but I received the same incorrect result.</p>\n<p>What I am doing wrong?</p>\n<p>[Edit]<br>\nI create a notebook, where you can verify your LB score<br>\n<a href=\"https://www.kaggle.com/bessenyeiszilrd/get-lb-score-under-10-min\" target=\"_blank\">https://www.kaggle.com/bessenyeiszilrd/get-lb-score-under-10-min</a></p>",
  "messages": [
    {
      "id": "1064574",
      "postDate": "10/30/2020 09:16:23",
      "content": "<p>I am using the <a href=\"https://www.kaggle.com/louis925/lyft-complete-train-and-prediction-pipeline\" target=\"_blank\">Lyft: Complete train and prediction pipeline</a>. The loss function and most of the code is the same, and after the training, I got 28.31 loss, and after the submission, I get around 27, which means for me that loss calculation is ok. I tried to add the validation procedure to it. In the as they suggested <a href=\"https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/185762\" target=\"_blank\">this post</a>. I generated a dataset with this code, which is based on <a href=\"https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb\" target=\"_blank\">agent_motion_prediction.ipynb</a></p>\n<pre><code>'validate_data_loader': {\n    'key': 'scenes/validate.zarr',\n    'batch_size': 16,\n    'shuffle': False,\n    'num_workers': 4,\n}\n\n...\n\nnum_frames_to_chop = 100\neval_cfg = cfg[\"validate_data_loader\"]\neval_base_path = create_chopped_dataset(dm.require(eval_cfg[\"key\"]), cfg[\"raster_params\"][\"filter_agents_threshold\"], num_frames_to_chop, \ncfg[\"model_params\"][\"future_num_frames\"], MIN_FUTURE_STEPS)\n\neval_zarr_path = str(Path(eval_base_path) / Path(dm.require(eval_cfg[\"key\"])).name)\neval_mask_path = str(Path(eval_base_path) / \"mask.npz\")\neval_gt_path = str(Path(eval_base_path) / \"gt.csv\")\n\neval_zarr = ChunkedDataset(eval_zarr_path).open()\neval_mask = np.load(eval_mask_path)[\"arr_0\"]\n# ===== INIT DATASET AND LOAD MASK\neval_dataset = AgentDataset(cfg, eval_zarr, rasterizer, agents_mask=eval_mask)\neval_dataloader = DataLoader(eval_dataset, shuffle=eval_cfg[\"shuffle\"], batch_size=eval_cfg[\"batch_size\"], \n                             num_workers=eval_cfg[\"num_workers\"])\n</code></pre>\n<p>After that, I updated the <code>cfg[\"validate_data_loader\"][\"key\"]</code> to <code>scenes/validate_chopped_100/validate.zarr</code> to load the chopped dataset. As I understand, the <code>print(eval_dataset)</code> gives correct output, every scene, has 100 frames.</p>\n<pre><code>+------------+------------+------------+---------------+-----------------+----------------------+----------------------+----------------------+---------------------+\n| Num Scenes | Num Frames | Num Agents | Num TR lights | Total Time (hr) | Avg Frames per Scene | Avg Agents per Frame | Avg Scene Time (sec) | Avg Frame frequency |\n+------------+------------+------------+---------------+-----------------+----------------------+----------------------+----------------------+---------------------+\n|   16220    |  1622000   | 125423254  |    11733321   |      45.06      |        100.00        |        77.33         |        10.00         |        10.00        |\n+------------+------------+------------+---------------+-----------------+----------------------+----------------------+----------------------+---------------------+\n</code></pre>\n<p>I tried to evaluate my model whit this code, and give a incorrect result, which is less than one: <code>loss: 1.292e-08 loss(avg): 2.922e-07</code></p>\n<pre><code># ==== EVAL LOOP\nmodel.eval()\ntorch.set_grad_enabled(False)\nlosses_valid = []\nprogress_bar = tqdm(eval_dataloader)\n\nfor data in progress_bar:\n    loss, _, _ = forward(data, model, device)\n    losses_valid.append(loss.item())   \n    progress_bar.set_description(f\"loss: {loss.item()} loss(avg): {np.mean(losses_valid)}\")\n</code></pre>\n<p>After that, based on the <a href=\"https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb\" target=\"_blank\">agent_motion_prediction.ipynb</a> suggestion:</p>\n<blockquote>\n  <p>The result is that <strong>each scene has been reduced to only 100 frames</strong>, and <strong>only valid agents in the 100th frame will be used to compute the metrics</strong>. </p>\n</blockquote>\n<p>I tried to evaluate every 100th of frame, but I received the same incorrect result.</p>\n<p>What I am doing wrong?</p>\n<p>[Edit]<br>\nI create a notebook, where you can verify your LB score<br>\n<a href=\"https://www.kaggle.com/bessenyeiszilrd/get-lb-score-under-10-min\" target=\"_blank\">https://www.kaggle.com/bessenyeiszilrd/get-lb-score-under-10-min</a></p>",
      "rawMarkdown": "I am using the [Lyft: Complete train and prediction pipeline](https://www.kaggle.com/louis925/lyft-complete-train-and-prediction-pipeline). The loss function and most of the code is the same, and after the training, I got 28.31 loss, and after the submission, I get around 27, which means for me that loss calculation is ok. I tried to add the validation procedure to it. In the as they suggested [this post](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/185762). I generated a dataset with this code, which is based on [agent_motion_prediction.ipynb](https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb)\n\n```\n'validate_data_loader': {\n    'key': 'scenes/validate.zarr',\n    'batch_size': 16,\n    'shuffle': False,\n    'num_workers': 4,\n}\n\n...\n\nnum_frames_to_chop = 100\neval_cfg = cfg[\"validate_data_loader\"]\neval_base_path = create_chopped_dataset(dm.require(eval_cfg[\"key\"]), cfg[\"raster_params\"][\"filter_agents_threshold\"], num_frames_to_chop, \ncfg[\"model_params\"][\"future_num_frames\"], MIN_FUTURE_STEPS)\n\neval_zarr_path = str(Path(eval_base_path) / Path(dm.require(eval_cfg[\"key\"])).name)\neval_mask_path = str(Path(eval_base_path) / \"mask.npz\")\neval_gt_path = str(Path(eval_base_path) / \"gt.csv\")\n\neval_zarr = ChunkedDataset(eval_zarr_path).open()\neval_mask = np.load(eval_mask_path)[\"arr_0\"]\n# ===== INIT DATASET AND LOAD MASK\neval_dataset = AgentDataset(cfg, eval_zarr, rasterizer, agents_mask=eval_mask)\neval_dataloader = DataLoader(eval_dataset, shuffle=eval_cfg[\"shuffle\"], batch_size=eval_cfg[\"batch_size\"], \n                             num_workers=eval_cfg[\"num_workers\"])\n```\n\nAfter that, I updated the `cfg[\"validate_data_loader\"][\"key\"]` to `scenes/validate_chopped_100/validate.zarr` to load the chopped dataset. As I understand, the `print(eval_dataset)` gives correct output, every scene, has 100 frames.\n\n```\n+------------+------------+------------+---------------+-----------------+----------------------+----------------------+----------------------+---------------------+\n| Num Scenes | Num Frames | Num Agents | Num TR lights | Total Time (hr) | Avg Frames per Scene | Avg Agents per Frame | Avg Scene Time (sec) | Avg Frame frequency |\n+------------+------------+------------+---------------+-----------------+----------------------+----------------------+----------------------+---------------------+\n|   16220    |  1622000   | 125423254  |    11733321   |      45.06      |        100.00        |        77.33         |        10.00         |        10.00        |\n+------------+------------+------------+---------------+-----------------+----------------------+----------------------+----------------------+---------------------+\n```\n\nI tried to evaluate my model whit this code, and give a incorrect result, which is less than one: `loss: 1.292e-08 loss(avg): 2.922e-07`\n\n```python\n# ==== EVAL LOOP\nmodel.eval()\ntorch.set_grad_enabled(False)\nlosses_valid = []\nprogress_bar = tqdm(eval_dataloader)\n\nfor data in progress_bar:\n    loss, _, _ = forward(data, model, device)\n    losses_valid.append(loss.item())   \n    progress_bar.set_description(f\"loss: {loss.item()} loss(avg): {np.mean(losses_valid)}\")\n```\n\nAfter that, based on the [agent_motion_prediction.ipynb](https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb) suggestion:\n\n> The result is that **each scene has been reduced to only 100 frames**, and **only valid agents in the 100th frame will be used to compute the metrics**. \n\nI tried to evaluate every 100th of frame, but I received the same incorrect result.\n\nWhat I am doing wrong?\n\n\n[Edit]\nI create a notebook, where you can verify your LB score\nhttps://www.kaggle.com/bessenyeiszilrd/get-lb-score-under-10-min",
      "votes": null
    },
    {
      "id": "1064582",
      "postDate": "10/30/2020 09:30:13",
      "content": "<p>Maybe <code>summer</code> in <code>losses_valid.append(summer)</code> should something else? I don't see it anywhere else.</p>",
      "rawMarkdown": "Maybe `summer` in `losses_valid.append(summer)` should something else? I don't see it anywhere else.",
      "votes": null
    },
    {
      "id": "1064594",
      "postDate": "10/30/2020 09:44:54",
      "content": "<p>Sorry, I forgot to update that part of the code. Now I replaced the <code>summer</code> with the <code>loss.item()</code></p>",
      "rawMarkdown": "Sorry, I forgot to update that part of the code. Now I replaced the `summer` with the `loss.item()`",
      "votes": null
    },
    {
      "id": "1064934",
      "postDate": "10/30/2020 17:16:15",
      "content": "<p>If you look in the agent_motion_prediction notebook you link, you'll see that while they train in agent space they have to convert to world space for the validation (create_chopped_dataset still produces gt.csv in world space, even after 1.1.0). This is the same step that you had to take when you provided predictions for submission. I think this is your issue here.</p>",
      "rawMarkdown": "If you look in the agent_motion_prediction notebook you link, you'll see that while they train in agent space they have to convert to world space for the validation (create_chopped_dataset still produces gt.csv in world space, even after 1.1.0). This is the same step that you had to take when you provided predictions for submission. I think this is your issue here.",
      "votes": null
    },
    {
      "id": "1066479",
      "postDate": "11/01/2020 20:14:37",
      "content": "<p>Thank you <a href=\"https://www.kaggle.com/taindow\" target=\"_blank\">@taindow</a>, I think you are right. I tried to update the code based on your suggestion, and to compare the world coordinates with the ground truth. Here is the code. Mostly I copied the eval part from the <code>agent_motion_prediction</code> notebook:</p>\n<pre><code>def forward_valid(data, preds, confidences, device, criterion = pytorch_neg_multi_log_likelihood_batch):\n    try:\n        target_availabilities = data[\"target_availabilities\"].to(device)\n        targets = data[\"target_positions\"].to(device)\n    except:\n        target_availabilities = torch.Tensor([data[\"avail\"]]).to(device)\n        targets = torch.Tensor([data[\"coord\"]]).to(device)\n    # Forward pass\n    loss = criterion(targets, preds, confidences, target_availabilities)\n    return loss\n\nlosses_val_ids = []\ngt_path = '/home/szilard/kaggle/lyft/lyft-motion-prediction-autonomous-vehicles/scenes/validate_chopped_100/gt.csv'\ngt_csv = read_gt_csv(gt_path)\n\nif cfg[\"model_params\"][\"predict\"]:\n    model.eval()\n    torch.set_grad_enabled(False)\n\n    # store information for evaluation\n    future_coords_offsets_pd = []\n    timestamps = []\n    confidences_list = []\n    agent_ids = []\n\n    progress_bar = tqdm(eval_dataloader)\n    losses_val = []\n\n    gt_it = iter(gt_csv)\n    for i, data in enumerate(progress_bar):\n        try:\n            gt_data = next(gt_it)\n        except StopIteration:\n            gt_data = iter(gt_csv)\n            gt_data = next(gt_it)\n\n        loss, preds, confidences = forward(data, model, device)\n\n        preds = preds.cpu().numpy()\n        world_from_agents = data[\"world_from_agent\"].numpy()\n        centroids = data[\"centroid\"].numpy()\n        coords_offset = []\n\n        # convert into world coordinates and compute offsets\n        for idx in range(len(preds)):\n            for mode in range(3):\n                preds[idx, mode, :, :] = transform_points(preds[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]\n\n        future_coords_offsets_pd.append(preds.copy())\n        confidences_list.append(confidences.cpu().numpy().copy())\n        timestamps.append(data[\"timestamp\"].numpy().copy())\n        agent_ids.append(data[\"track_id\"].numpy().copy())         \n\n        preds = torch.Tensor(preds).to(device)\n        valid_loss = forward_valid(gt_data, preds, confidences, device)\n\n        losses_val_ids.append((i, valid_loss.item()))\n        losses_val.append(valid_loss.item())\n\n        progress_bar.set_description(f\"loss: {valid_loss.item()} loss(avg): {np.mean(losses_val)}\")\n</code></pre>\n<p>I this code gives a really changing result:</p>\n<pre><code>(0, 69.87364959716797)\n(1, 0.6804960370063782)\n(2, 3366.071044921875)\n(3, 29930.384765625)\n(4, 62318.41796875)\n(5, 221.79736328125)\n(6, 1562.949951171875)\n(7, 1.0121519565582275)\n(8, 608.3523559570312)\n(9, 0.7305832505226135)\n(10, 200.2021026611328)\n(11, 0.8827717304229736)\n(12, 1.5172334909439087)\n(13, 8092.22509765625)\n...\n</code></pre>\n<p>What I am doing wrong?</p>",
      "rawMarkdown": "Thank you @taindow, I think you are right. I tried to update the code based on your suggestion, and to compare the world coordinates with the ground truth. Here is the code. Mostly I copied the eval part from the `agent_motion_prediction` notebook:\n\n```python\n\ndef forward_valid(data, preds, confidences, device, criterion = pytorch_neg_multi_log_likelihood_batch):\n    try:\n        target_availabilities = data[\"target_availabilities\"].to(device)\n        targets = data[\"target_positions\"].to(device)\n    except:\n        target_availabilities = torch.Tensor([data[\"avail\"]]).to(device)\n        targets = torch.Tensor([data[\"coord\"]]).to(device)\n    # Forward pass\n    loss = criterion(targets, preds, confidences, target_availabilities)\n    return loss\n\nlosses_val_ids = []\ngt_path = '/home/szilard/kaggle/lyft/lyft-motion-prediction-autonomous-vehicles/scenes/validate_chopped_100/gt.csv'\ngt_csv = read_gt_csv(gt_path)\n\nif cfg[\"model_params\"][\"predict\"]:\n    model.eval()\n    torch.set_grad_enabled(False)\n\n    # store information for evaluation\n    future_coords_offsets_pd = []\n    timestamps = []\n    confidences_list = []\n    agent_ids = []\n\n    progress_bar = tqdm(eval_dataloader)\n    losses_val = []\n    \n    gt_it = iter(gt_csv)\n    for i, data in enumerate(progress_bar):\n        try:\n            gt_data = next(gt_it)\n        except StopIteration:\n            gt_data = iter(gt_csv)\n            gt_data = next(gt_it)\n            \n        loss, preds, confidences = forward(data, model, device)\n\n        preds = preds.cpu().numpy()\n        world_from_agents = data[\"world_from_agent\"].numpy()\n        centroids = data[\"centroid\"].numpy()\n        coords_offset = []\n        \n        # convert into world coordinates and compute offsets\n        for idx in range(len(preds)):\n            for mode in range(3):\n                preds[idx, mode, :, :] = transform_points(preds[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]\n                \n        future_coords_offsets_pd.append(preds.copy())\n        confidences_list.append(confidences.cpu().numpy().copy())\n        timestamps.append(data[\"timestamp\"].numpy().copy())\n        agent_ids.append(data[\"track_id\"].numpy().copy())         \n\n        preds = torch.Tensor(preds).to(device)\n        valid_loss = forward_valid(gt_data, preds, confidences, device)\n\n        losses_val_ids.append((i, valid_loss.item()))\n        losses_val.append(valid_loss.item())\n\n        progress_bar.set_description(f\"loss: {valid_loss.item()} loss(avg): {np.mean(losses_val)}\")\n\n```\n\nI this code gives a really changing result:\n```\n(0, 69.87364959716797)\n(1, 0.6804960370063782)\n(2, 3366.071044921875)\n(3, 29930.384765625)\n(4, 62318.41796875)\n(5, 221.79736328125)\n(6, 1562.949951171875)\n(7, 1.0121519565582275)\n(8, 608.3523559570312)\n(9, 0.7305832505226135)\n(10, 200.2021026611328)\n(11, 0.8827717304229736)\n(12, 1.5172334909439087)\n(13, 8092.22509765625)\n...\n```\n\nWhat I am doing wrong?",
      "votes": null
    },
    {
      "id": "1066907",
      "postDate": "11/02/2020 07:43:48",
      "content": "<p>Where I can learn about this problem(Autonomous) As a beginner… </p>",
      "rawMarkdown": "Where I can learn about this problem(Autonomous) As a beginner...",
      "votes": null
    },
    {
      "id": "1066947",
      "postDate": "11/02/2020 08:46:16",
      "content": "<p>I think you can start here:</p>\n<ul>\n<li><a href=\"https://github.com/lyft/l5kit\" target=\"_blank\">https://github.com/lyft/l5kit</a></li>\n<li><a href=\"https://medium.com/lyftlevel5/how-to-build-a-motion-prediction-model-for-autonomous-vehicles-29f7f81f1580\" target=\"_blank\">https://medium.com/lyftlevel5/how-to-build-a-motion-prediction-model-for-autonomous-vehicles-29f7f81f1580</a></li>\n<li><a href=\"https://self-driving.lyft.com/level5/prediction/\" target=\"_blank\">https://self-driving.lyft.com/level5/prediction/</a></li>\n</ul>",
      "rawMarkdown": "I think you can start here:\n* https://github.com/lyft/l5kit\n* https://medium.com/lyftlevel5/how-to-build-a-motion-prediction-model-for-autonomous-vehicles-29f7f81f1580\n* https://self-driving.lyft.com/level5/prediction/",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1064582,
      "author_name": "pestipeti",
      "author_url": "",
      "post_date": "10/30/2020 09:30:13",
      "content": "<p>Maybe <code>summer</code> in <code>losses_valid.append(summer)</code> should something else? I don't see it anywhere else.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1064594,
          "author_name": "bessenyeiszilrd",
          "author_url": "",
          "post_date": "10/30/2020 09:44:54",
          "content": "<p>Sorry, I forgot to update that part of the code. Now I replaced the <code>summer</code> with the <code>loss.item()</code></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1064934,
      "author_name": "taindow",
      "author_url": "",
      "post_date": "10/30/2020 17:16:15",
      "content": "<p>If you look in the agent_motion_prediction notebook you link, you'll see that while they train in agent space they have to convert to world space for the validation (create_chopped_dataset still produces gt.csv in world space, even after 1.1.0). This is the same step that you had to take when you provided predictions for submission. I think this is your issue here.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1066479,
          "author_name": "bessenyeiszilrd",
          "author_url": "",
          "post_date": "11/01/2020 20:14:37",
          "content": "<p>Thank you <a href=\"https://www.kaggle.com/taindow\" target=\"_blank\">@taindow</a>, I think you are right. I tried to update the code based on your suggestion, and to compare the world coordinates with the ground truth. Here is the code. Mostly I copied the eval part from the <code>agent_motion_prediction</code> notebook:</p>\n<pre><code>def forward_valid(data, preds, confidences, device, criterion = pytorch_neg_multi_log_likelihood_batch):\n    try:\n        target_availabilities = data[\"target_availabilities\"].to(device)\n        targets = data[\"target_positions\"].to(device)\n    except:\n        target_availabilities = torch.Tensor([data[\"avail\"]]).to(device)\n        targets = torch.Tensor([data[\"coord\"]]).to(device)\n    # Forward pass\n    loss = criterion(targets, preds, confidences, target_availabilities)\n    return loss\n\nlosses_val_ids = []\ngt_path = '/home/szilard/kaggle/lyft/lyft-motion-prediction-autonomous-vehicles/scenes/validate_chopped_100/gt.csv'\ngt_csv = read_gt_csv(gt_path)\n\nif cfg[\"model_params\"][\"predict\"]:\n    model.eval()\n    torch.set_grad_enabled(False)\n\n    # store information for evaluation\n    future_coords_offsets_pd = []\n    timestamps = []\n    confidences_list = []\n    agent_ids = []\n\n    progress_bar = tqdm(eval_dataloader)\n    losses_val = []\n\n    gt_it = iter(gt_csv)\n    for i, data in enumerate(progress_bar):\n        try:\n            gt_data = next(gt_it)\n        except StopIteration:\n            gt_data = iter(gt_csv)\n            gt_data = next(gt_it)\n\n        loss, preds, confidences = forward(data, model, device)\n\n        preds = preds.cpu().numpy()\n        world_from_agents = data[\"world_from_agent\"].numpy()\n        centroids = data[\"centroid\"].numpy()\n        coords_offset = []\n\n        # convert into world coordinates and compute offsets\n        for idx in range(len(preds)):\n            for mode in range(3):\n                preds[idx, mode, :, :] = transform_points(preds[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]\n\n        future_coords_offsets_pd.append(preds.copy())\n        confidences_list.append(confidences.cpu().numpy().copy())\n        timestamps.append(data[\"timestamp\"].numpy().copy())\n        agent_ids.append(data[\"track_id\"].numpy().copy())         \n\n        preds = torch.Tensor(preds).to(device)\n        valid_loss = forward_valid(gt_data, preds, confidences, device)\n\n        losses_val_ids.append((i, valid_loss.item()))\n        losses_val.append(valid_loss.item())\n\n        progress_bar.set_description(f\"loss: {valid_loss.item()} loss(avg): {np.mean(losses_val)}\")\n</code></pre>\n<p>I this code gives a really changing result:</p>\n<pre><code>(0, 69.87364959716797)\n(1, 0.6804960370063782)\n(2, 3366.071044921875)\n(3, 29930.384765625)\n(4, 62318.41796875)\n(5, 221.79736328125)\n(6, 1562.949951171875)\n(7, 1.0121519565582275)\n(8, 608.3523559570312)\n(9, 0.7305832505226135)\n(10, 200.2021026611328)\n(11, 0.8827717304229736)\n(12, 1.5172334909439087)\n(13, 8092.22509765625)\n...\n</code></pre>\n<p>What I am doing wrong?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1066907,
      "author_name": "realc9der",
      "author_url": "",
      "post_date": "11/02/2020 07:43:48",
      "content": "<p>Where I can learn about this problem(Autonomous) As a beginner… </p>",
      "votes": null,
      "replies": [
        {
          "id": 1066947,
          "author_name": "bessenyeiszilrd",
          "author_url": "",
          "post_date": "11/02/2020 08:46:16",
          "content": "<p>I think you can start here:</p>\n<ul>\n<li><a href=\"https://github.com/lyft/l5kit\" target=\"_blank\">https://github.com/lyft/l5kit</a></li>\n<li><a href=\"https://medium.com/lyftlevel5/how-to-build-a-motion-prediction-model-for-autonomous-vehicles-29f7f81f1580\" target=\"_blank\">https://medium.com/lyftlevel5/how-to-build-a-motion-prediction-model-for-autonomous-vehicles-29f7f81f1580</a></li>\n<li><a href=\"https://self-driving.lyft.com/level5/prediction/\" target=\"_blank\">https://self-driving.lyft.com/level5/prediction/</a></li>\n</ul>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1064574": "I am using the [Lyft: Complete train and prediction pipeline](https://www.kaggle.com/louis925/lyft-complete-train-and-prediction-pipeline). The loss function and most of the code is the same, and after the training, I got 28.31 loss, and after the submission, I get around 27, which means for me that loss calculation is ok. I tried to add the validation procedure to it. In the as they suggested [this post](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/185762). I generated a dataset with this code, which is based on [agent_motion_prediction.ipynb](https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb)\n\n```\n'validate_data_loader': {\n    'key': 'scenes/validate.zarr',\n    'batch_size': 16,\n    'shuffle': False,\n    'num_workers': 4,\n}\n\n...\n\nnum_frames_to_chop = 100\neval_cfg = cfg[\"validate_data_loader\"]\neval_base_path = create_chopped_dataset(dm.require(eval_cfg[\"key\"]), cfg[\"raster_params\"][\"filter_agents_threshold\"], num_frames_to_chop, \ncfg[\"model_params\"][\"future_num_frames\"], MIN_FUTURE_STEPS)\n\neval_zarr_path = str(Path(eval_base_path) / Path(dm.require(eval_cfg[\"key\"])).name)\neval_mask_path = str(Path(eval_base_path) / \"mask.npz\")\neval_gt_path = str(Path(eval_base_path) / \"gt.csv\")\n\neval_zarr = ChunkedDataset(eval_zarr_path).open()\neval_mask = np.load(eval_mask_path)[\"arr_0\"]\n# ===== INIT DATASET AND LOAD MASK\neval_dataset = AgentDataset(cfg, eval_zarr, rasterizer, agents_mask=eval_mask)\neval_dataloader = DataLoader(eval_dataset, shuffle=eval_cfg[\"shuffle\"], batch_size=eval_cfg[\"batch_size\"], \n                             num_workers=eval_cfg[\"num_workers\"])\n```\n\nAfter that, I updated the `cfg[\"validate_data_loader\"][\"key\"]` to `scenes/validate_chopped_100/validate.zarr` to load the chopped dataset. As I understand, the `print(eval_dataset)` gives correct output, every scene, has 100 frames.\n\n```\n+------------+------------+------------+---------------+-----------------+----------------------+----------------------+----------------------+---------------------+\n| Num Scenes | Num Frames | Num Agents | Num TR lights | Total Time (hr) | Avg Frames per Scene | Avg Agents per Frame | Avg Scene Time (sec) | Avg Frame frequency |\n+------------+------------+------------+---------------+-----------------+----------------------+----------------------+----------------------+---------------------+\n|   16220    |  1622000   | 125423254  |    11733321   |      45.06      |        100.00        |        77.33         |        10.00         |        10.00        |\n+------------+------------+------------+---------------+-----------------+----------------------+----------------------+----------------------+---------------------+\n```\n\nI tried to evaluate my model whit this code, and give a incorrect result, which is less than one: `loss: 1.292e-08 loss(avg): 2.922e-07`\n\n```python\n# ==== EVAL LOOP\nmodel.eval()\ntorch.set_grad_enabled(False)\nlosses_valid = []\nprogress_bar = tqdm(eval_dataloader)\n\nfor data in progress_bar:\n    loss, _, _ = forward(data, model, device)\n    losses_valid.append(loss.item())   \n    progress_bar.set_description(f\"loss: {loss.item()} loss(avg): {np.mean(losses_valid)}\")\n```\n\nAfter that, based on the [agent_motion_prediction.ipynb](https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb) suggestion:\n\n> The result is that **each scene has been reduced to only 100 frames**, and **only valid agents in the 100th frame will be used to compute the metrics**. \n\nI tried to evaluate every 100th of frame, but I received the same incorrect result.\n\nWhat I am doing wrong?\n\n\n[Edit]\nI create a notebook, where you can verify your LB score\nhttps://www.kaggle.com/bessenyeiszilrd/get-lb-score-under-10-min",
    "1064582": "Maybe `summer` in `losses_valid.append(summer)` should something else? I don't see it anywhere else.",
    "1064594": "Sorry, I forgot to update that part of the code. Now I replaced the `summer` with the `loss.item()`",
    "1064934": "If you look in the agent_motion_prediction notebook you link, you'll see that while they train in agent space they have to convert to world space for the validation (create_chopped_dataset still produces gt.csv in world space, even after 1.1.0). This is the same step that you had to take when you provided predictions for submission. I think this is your issue here.",
    "1066479": "Thank you @taindow, I think you are right. I tried to update the code based on your suggestion, and to compare the world coordinates with the ground truth. Here is the code. Mostly I copied the eval part from the `agent_motion_prediction` notebook:\n\n```python\n\ndef forward_valid(data, preds, confidences, device, criterion = pytorch_neg_multi_log_likelihood_batch):\n    try:\n        target_availabilities = data[\"target_availabilities\"].to(device)\n        targets = data[\"target_positions\"].to(device)\n    except:\n        target_availabilities = torch.Tensor([data[\"avail\"]]).to(device)\n        targets = torch.Tensor([data[\"coord\"]]).to(device)\n    # Forward pass\n    loss = criterion(targets, preds, confidences, target_availabilities)\n    return loss\n\nlosses_val_ids = []\ngt_path = '/home/szilard/kaggle/lyft/lyft-motion-prediction-autonomous-vehicles/scenes/validate_chopped_100/gt.csv'\ngt_csv = read_gt_csv(gt_path)\n\nif cfg[\"model_params\"][\"predict\"]:\n    model.eval()\n    torch.set_grad_enabled(False)\n\n    # store information for evaluation\n    future_coords_offsets_pd = []\n    timestamps = []\n    confidences_list = []\n    agent_ids = []\n\n    progress_bar = tqdm(eval_dataloader)\n    losses_val = []\n    \n    gt_it = iter(gt_csv)\n    for i, data in enumerate(progress_bar):\n        try:\n            gt_data = next(gt_it)\n        except StopIteration:\n            gt_data = iter(gt_csv)\n            gt_data = next(gt_it)\n            \n        loss, preds, confidences = forward(data, model, device)\n\n        preds = preds.cpu().numpy()\n        world_from_agents = data[\"world_from_agent\"].numpy()\n        centroids = data[\"centroid\"].numpy()\n        coords_offset = []\n        \n        # convert into world coordinates and compute offsets\n        for idx in range(len(preds)):\n            for mode in range(3):\n                preds[idx, mode, :, :] = transform_points(preds[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]\n                \n        future_coords_offsets_pd.append(preds.copy())\n        confidences_list.append(confidences.cpu().numpy().copy())\n        timestamps.append(data[\"timestamp\"].numpy().copy())\n        agent_ids.append(data[\"track_id\"].numpy().copy())         \n\n        preds = torch.Tensor(preds).to(device)\n        valid_loss = forward_valid(gt_data, preds, confidences, device)\n\n        losses_val_ids.append((i, valid_loss.item()))\n        losses_val.append(valid_loss.item())\n\n        progress_bar.set_description(f\"loss: {valid_loss.item()} loss(avg): {np.mean(losses_val)}\")\n\n```\n\nI this code gives a really changing result:\n```\n(0, 69.87364959716797)\n(1, 0.6804960370063782)\n(2, 3366.071044921875)\n(3, 29930.384765625)\n(4, 62318.41796875)\n(5, 221.79736328125)\n(6, 1562.949951171875)\n(7, 1.0121519565582275)\n(8, 608.3523559570312)\n(9, 0.7305832505226135)\n(10, 200.2021026611328)\n(11, 0.8827717304229736)\n(12, 1.5172334909439087)\n(13, 8092.22509765625)\n...\n```\n\nWhat I am doing wrong?",
    "1066907": "Where I can learn about this problem(Autonomous) As a beginner...",
    "1066947": "I think you can start here:\n* https://github.com/lyft/l5kit\n* https://medium.com/lyftlevel5/how-to-build-a-motion-prediction-model-for-autonomous-vehicles-29f7f81f1580\n* https://self-driving.lyft.com/level5/prediction/"
  },
  "source": "meta"
}