filter_agents_threshold used for? When I load the training dataset, I got the following consequence, but I can not figure out the meaning of this.![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3722352%2Fb7b3b6d7dbec3aed4e877025e583b05e%2F269931601990885_.pic_hd.jpg?generation=1601990897546131&alt=media\")
I had a look at the LYFT5 metrics package and tried to integrate the evaluations into my code, but still could not entirely managed. I have two issues and any advice is welcome of course.
\nSince the module \"accuracy\" in pytorch lighting kept giving me ValueError \"cannot infer num_classes when target is all zero\", I wrote my own script
\ndef forward(data, model, device, criterion = pytorch_neg_multi_log_likelihood_batch):
\n test_accuracy =[]
\n inputs = data[\"image\"].to(device)
\n target_availabilities = data[\"target_availabilities\"].to(device)
\n targets = data[\"target_positions\"].to(device)
\n # Forward pass
\n preds, confidences = model(inputs)
\n _, predicted = torch.max(preds.data, 1) #highest probable
\n loss = criterion(targets, preds, confidences, target_availabilities)
\n # not all the output steps are valid, but we can filter them out from the loss using availabilities
\n loss = loss * target_availabilities
\n loss = loss.mean()
#Accuracy\npredicted = (predicted>0.1).float()\ncorrect = (predicted == targets).float().sum()\n\nacc = correct / predicted.size(0)#Accuracy(preds, targets)\nacc = acc * target_availabilities\nacc = acc.mean()\nreturn acc, loss, preds\nand calculated an average value in my training loop, as I did for the loss. Unfortunately the results are not as expected, since I get values over 1, ideal case for all correct == predicted.
\nerror = compute_error_csv(eval_gt_path, pred_path)
\npred_path is my submitted file \"submission.csv\"
\neval_gt_path should be the ground truth \"gt.csv\"
to create the latter, there is a package
\nl5kit.evaluation.export_zarr_to_csv(zarr_dataset: l5kit.data.zarr_dataset.ChunkedDataset, csv_file_path: str, future_num_frames: int, filter_agents_threshold: float, history_step_size: int = 1, future_step_size: int = 1, agents_mask: Optional[numpy.array] = None) → None
\nbut the command
\nexport_zarr_to_csv(test_zarr, eval_gt_path, 50, 0.5)
\nreturns the following KeyError: 'agents_mask/.zgroup'
after creating an agents mask. If I specify the mask myself
\ntest_mask = np.load(f\"{PATH_TO_DATA}/scenes/mask.npz\")[\"arr_0\"]
\nagents_mask=test_mask
I get a value error for filter_agents_threshold and the agents mask is still created, leading to the above KeyError
", "messages": [ { "id": 1039577, "postDate": "2020-10-06T16:43:16.460Z", "content": "it should be \"threshold over agents probabilities used in select_agents function\", from https://lyft.github.io/l5kit/API/l5kit.evaluation.html
\nSo the minimum probability that identifies an agent as car, bicycle, truck etc. It is specified in the configuration, but and as far as I understood should be used to create the agent mask. The message that it does not have a match and mask is going to be created appeared also to me. Even when I tried to change the threshold. After creation though I got also the KeyError: 'agents_mask/.zgroup'
", "rawMarkdown": "it should be \"threshold over agents probabilities used in select_agents function\", from https://lyft.github.io/l5kit/API/l5kit.evaluation.html\n\nSo the minimum probability that identifies an agent as car, bicycle, truck etc. It is specified in the configuration, but and as far as I understood should be used to create the agent mask. The message that it does not have a match and mask is going to be created appeared also to me. Even when I tried to change the threshold. After creation though I got also the KeyError: 'agents_mask/.zgroup'", "replies": [ { "id": 1042813, "postDate": "2020-10-08T13:41:20.163Z", "content": "Did you download the training dataset from https://self-driving.lyft.com/level5/download/? Is it possible that the cfg file is not matching the official dataset?
", "rawMarkdown": "Did you download the training dataset from [https://self-driving.lyft.com/level5/download/](url)? Is it possible that the cfg file is not matching the official dataset?", "replies": [ { "id": 1043929, "postDate": "2020-10-09T10:44:23.330Z", "content": "unlikely. Plus when I checked other kernels, all had the threshold set to 0.5.
\nI may try again to see if that works with validation dataset as in this example https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb
I also got a question about the mask. What the parameter filter_agents_threshold used for? When I load the training dataset, I got the following consequence, but I can not figure out the meaning of this.
I had a look at the LYFT5 metrics package and tried to integrate the evaluations into my code, but still could not entirely managed. I have two issues and any advice is welcome of course.
\nSince the module \"accuracy\" in pytorch lighting kept giving me ValueError \"cannot infer num_classes when target is all zero\", I wrote my own script
\ndef forward(data, model, device, criterion = pytorch_neg_multi_log_likelihood_batch):
\n test_accuracy =[]
\n inputs = data[\"image\"].to(device)
\n target_availabilities = data[\"target_availabilities\"].to(device)
\n targets = data[\"target_positions\"].to(device)
\n # Forward pass
\n preds, confidences = model(inputs)
\n _, predicted = torch.max(preds.data, 1) #highest probable
\n loss = criterion(targets, preds, confidences, target_availabilities)
\n # not all the output steps are valid, but we can filter them out from the loss using availabilities
\n loss = loss * target_availabilities
\n loss = loss.mean()
#Accuracy\npredicted = (predicted>0.1).float()\ncorrect = (predicted == targets).float().sum()\n\nacc = correct / predicted.size(0)#Accuracy(preds, targets)\nacc = acc * target_availabilities\nacc = acc.mean()\nreturn acc, loss, preds\nand calculated an average value in my training loop, as I did for the loss. Unfortunately the results are not as expected, since I get values over 1, ideal case for all correct == predicted.
\nerror = compute_error_csv(eval_gt_path, pred_path)
\npred_path is my submitted file \"submission.csv\"
\neval_gt_path should be the ground truth \"gt.csv\"
to create the latter, there is a package
\nl5kit.evaluation.export_zarr_to_csv(zarr_dataset: l5kit.data.zarr_dataset.ChunkedDataset, csv_file_path: str, future_num_frames: int, filter_agents_threshold: float, history_step_size: int = 1, future_step_size: int = 1, agents_mask: Optional[numpy.array] = None) → None
\nbut the command
\nexport_zarr_to_csv(test_zarr, eval_gt_path, 50, 0.5)
\nreturns the following KeyError: 'agents_mask/.zgroup'
after creating an agents mask. If I specify the mask myself
\ntest_mask = np.load(f\"{PATH_TO_DATA}/scenes/mask.npz\")[\"arr_0\"]
\nagents_mask=test_mask
I get a value error for filter_agents_threshold and the agents mask is still created, leading to the above KeyError
", "rawMarkdown": "I had a look at the LYFT5 metrics package and tried to integrate the evaluations into my code, but still could not entirely managed. I have two issues and any advice is welcome of course.\n\n- Accuracy for predictions during the training loop\n\nSince the module \"accuracy\" in pytorch lighting kept giving me ValueError \"cannot infer num_classes when target is all zero\", I wrote my own script\n\ndef forward(data, model, device, criterion = pytorch_neg_multi_log_likelihood_batch):\n test_accuracy =[]\n inputs = data[\"image\"].to(device)\n target_availabilities = data[\"target_availabilities\"].to(device)\n targets = data[\"target_positions\"].to(device)\n # Forward pass\n preds, confidences = model(inputs)\n _, predicted = torch.max(preds.data, 1) #highest probable\n loss = criterion(targets, preds, confidences, target_availabilities)\n # not all the output steps are valid, but we can filter them out from the loss using availabilities\n loss = loss * target_availabilities\n loss = loss.mean()\n \n #Accuracy\n predicted = (predicted>0.1).float()\n correct = (predicted == targets).float().sum()\n \n acc = correct / predicted.size(0)#Accuracy(preds, targets)\n acc = acc * target_availabilities\n acc = acc.mean()\n return acc, loss, preds\n\nand calculated an average value in my training loop, as I did for the loss. Unfortunately the results are not as expected, since I get values over 1, ideal case for all correct == predicted.\n\n\n- Error metrics for the submission\n\nerror = compute_error_csv(eval_gt_path, pred_path)\n\npred_path is my submitted file \"submission.csv\"\neval_gt_path should be the ground truth \"gt.csv\"\n\nto create the latter, there is a package \n\nl5kit.evaluation.export_zarr_to_csv(zarr_dataset: l5kit.data.zarr_dataset.ChunkedDataset, csv_file_path: str, future_num_frames: int, filter_agents_threshold: float, history_step_size: int = 1, future_step_size: int = 1, agents_mask: Optional[numpy.array] = None) → None\n\nbut the command \nexport_zarr_to_csv(test_zarr, eval_gt_path, 50, 0.5)\nreturns the following KeyError: 'agents_mask/.zgroup'\n\nafter creating an agents mask. If I specify the mask myself \n\ntest_mask = np.load(f\"{PATH_TO_DATA}/scenes/mask.npz\")[\"arr_0\"]\nagents_mask=test_mask\n\nI get a value error for filter_agents_threshold and the agents mask is still created, leading to the above KeyError \n" }, { "id": 1042811, "postDate": "2020-10-08T13:39:59.763Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1039577, "author_name": "Marco B", "author_url": "", "post_date": "2020-10-06T16:43:16.460000", "content": "it should be \"threshold over agents probabilities used in select_agents function\", from https://lyft.github.io/l5kit/API/l5kit.evaluation.html
\nSo the minimum probability that identifies an agent as car, bicycle, truck etc. It is specified in the configuration, but and as far as I understood should be used to create the agent mask. The message that it does not have a match and mask is going to be created appeared also to me. Even when I tried to change the threshold. After creation though I got also the KeyError: 'agents_mask/.zgroup'
", "votes": 0, "replies": [ { "id": 1042813, "author_name": "Yannik", "author_url": "", "post_date": "2020-10-08T13:41:20.163000", "content": "Did you download the training dataset from https://self-driving.lyft.com/level5/download/? Is it possible that the cfg file is not matching the official dataset?
", "votes": 0, "replies": [ { "id": 1043929, "author_name": "Marco B", "author_url": "", "post_date": "2020-10-09T10:44:23.330000", "content": "unlikely. Plus when I checked other kernels, all had the threshold set to 0.5.
\nI may try again to see if that works with validation dataset as in this example https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb
I also got a question about the mask. What the parameter filter_agents_threshold used for? When I load the training dataset, I got the following consequence, but I can not figure out the meaning of this.