![\"typical](\"https://i.imgur.com/uQYCT7u.png\")
Hi Kagglers, I came across something important that I am excited to share. Despite my LB score improving from 32.210 to 25.524 after that fix with using 3x less training epochs to get there, I am still not 100% sure that I do not miss something simple and obvious to everyone but me. But here it is for your judgement.
We have input images of the following kind, aligned for our agent vehicle to look right and at 'ego_center': [0.25, 0.5]. It is the only input to our simple NNs that everyone uses.
But then we predict targets which are not aligned, at least if you follow the example notebook code. But the public notebooks that I saw use the same approach, if I draw several hundreds of target trajectories that we try to predict then it looks like this:
\n![\"targets](\"https://i.imgur.com/gxPjOAn.png\")
It is in all different directions, as expected. Then I spent a day trying to understand how is that possible to predict correct directions from direction-less input. To no avail, I still do not understand it. But the predictions are so accurate, they are actually very good! The direction is somehow predicted correctly from a direction-less input.
\nAfter failing to understand why everyone does it wrong and why it even works, I tried to implement it correctly, that is, transform the targets to image coordinates before training. I attach the pytorch code with the transformations below so you can try it yourself fast. If you try and get an improvement please report it here.
\nTransformed target trajectories look like this, this is 160 randomly sampled targets
\n![\"targets](\"https://i.imgur.com/NZVGGkT.png\")
And it worked like a charm. With only 15 epochs, 10k batches each, batch size 16, which is about 5h training on my machine, I improved my training score from 30.87 to 23.79 and public LB 32.210 to 25.524.
So how is the old approach able to predict the direction? I have two theories but no prove:
\nWhat do you think? I have a lingering feeling that I am missing something obvious!
\nBelow boolean flag cfg['train_params']['image_coords'] controls if I use the new approach. I apply the needed transformations on targets and then do inverse transformation on predictions in reverse order. Note that my predictions are multi-scenario there, you need to get rid of this additional dimensions if you use single scenario.
def forward(data, model, device, criterion):\n inputs = data[\"image\"].to(device)\n target_availabilities = data[\"target_availabilities\"].to(device)\n targets = data[\"target_positions\"].to(device)\n matrix = data[\"world_to_image\"].to(device)\n centroid = data[\"centroid\"].to(device)[:,None,:].to(torch.float)\n\n # Forward pass\n outputs = model(inputs)\n\n bs,tl,_ = targets.shape\n assert tl == cfg[\"model_params\"][\"future_num_frames\"]\n\n if cfg['train_params']['image_coords']:\n targets = targets + centroid\n targets = torch.cat([targets,torch.ones((bs,tl,1)).to(device)], dim=2)\n targets = torch.matmul(matrix.to(torch.float), targets.transpose(1,2))\n targets = targets.transpose(1,2)[:,:,:2]\n bias = torch.tensor([56.25, 112.5])[None,None,:].to(device)\n targets = targets - bias\n\n confidences, pred = outputs[:,:3], outputs[:,3:]\n pred = pred.view(bs, 3, tl, 2)\n assert confidences.shape == (bs, 3)\n confidences = torch.softmax(confidences, dim=1)\n\n loss = criterion(targets, pred, confidences, target_availabilities)\n loss = torch.mean(loss)\n\n if cfg['train_params']['image_coords']:\n matrix_inv = torch.inverse(matrix)\n pred = pred + bias[:,None,:,:]\n pred = torch.cat([pred,torch.ones((bs,3,tl,1)).to(device)], dim=3)\n pred = torch.stack([torch.matmul(matrix_inv.to(torch.float), pred[:,i].transpose(1,2)) \n for i in range(3)], dim=1)\n pred = pred.transpose(2,3)[:,:,:,:2]\n pred = pred - centroid[:,None,:,:]\n\n return loss, pred, confidences\nand in your loss function you may want to divide the coordinates by 2:
\n ...\n error = torch.sum(((gt - pred) * avails) ** 2, dim=-1) # reduce coords and use availability\n if cfg['train_params']['image_coords']:\n error = error / 4\n ...\nHi Kagglers, I came across something important that I am excited to share. Despite my LB score improving from 32.210 to 25.524 after that fix with using 3x less training epochs to get there, I am still not 100% sure that I do not miss something simple and obvious to everyone but me. But here it is for your judgement.
We have input images of the following kind, aligned for our agent vehicle to look right and at 'ego_center': [0.25, 0.5]. It is the only input to our simple NNs that everyone uses.
But then we predict targets which are not aligned, at least if you follow the example notebook code. But the public notebooks that I saw use the same approach, if I draw several hundreds of target trajectories that we try to predict then it looks like this:
\n
It is in all different directions, as expected. Then I spent a day trying to understand how is that possible to predict correct directions from direction-less input. To no avail, I still do not understand it. But the predictions are so accurate, they are actually very good! The direction is somehow predicted correctly from a direction-less input.
\nAfter failing to understand why everyone does it wrong and why it even works, I tried to implement it correctly, that is, transform the targets to image coordinates before training. I attach the pytorch code with the transformations below so you can try it yourself fast. If you try and get an improvement please report it here.
\nTransformed target trajectories look like this, this is 160 randomly sampled targets
\n
And it worked like a charm. With only 15 epochs, 10k batches each, batch size 16, which is about 5h training on my machine, I improved my training score from 30.87 to 23.79 and public LB 32.210 to 25.524.
So how is the old approach able to predict the direction? I have two theories but no prove:
\nWhat do you think? I have a lingering feeling that I am missing something obvious!
\nBelow boolean flag cfg['train_params']['image_coords'] controls if I use the new approach. I apply the needed transformations on targets and then do inverse transformation on predictions in reverse order. Note that my predictions are multi-scenario there, you need to get rid of this additional dimensions if you use single scenario.
def forward(data, model, device, criterion):\n inputs = data[\"image\"].to(device)\n target_availabilities = data[\"target_availabilities\"].to(device)\n targets = data[\"target_positions\"].to(device)\n matrix = data[\"world_to_image\"].to(device)\n centroid = data[\"centroid\"].to(device)[:,None,:].to(torch.float)\n\n # Forward pass\n outputs = model(inputs)\n\n bs,tl,_ = targets.shape\n assert tl == cfg[\"model_params\"][\"future_num_frames\"]\n\n if cfg['train_params']['image_coords']:\n targets = targets + centroid\n targets = torch.cat([targets,torch.ones((bs,tl,1)).to(device)], dim=2)\n targets = torch.matmul(matrix.to(torch.float), targets.transpose(1,2))\n targets = targets.transpose(1,2)[:,:,:2]\n bias = torch.tensor([56.25, 112.5])[None,None,:].to(device)\n targets = targets - bias\n\n confidences, pred = outputs[:,:3], outputs[:,3:]\n pred = pred.view(bs, 3, tl, 2)\n assert confidences.shape == (bs, 3)\n confidences = torch.softmax(confidences, dim=1)\n\n loss = criterion(targets, pred, confidences, target_availabilities)\n loss = torch.mean(loss)\n\n if cfg['train_params']['image_coords']:\n matrix_inv = torch.inverse(matrix)\n pred = pred + bias[:,None,:,:]\n pred = torch.cat([pred,torch.ones((bs,3,tl,1)).to(device)], dim=3)\n pred = torch.stack([torch.matmul(matrix_inv.to(torch.float), pred[:,i].transpose(1,2)) \n for i in range(3)], dim=1)\n pred = pred.transpose(2,3)[:,:,:,:2]\n pred = pred - centroid[:,None,:,:]\n\n return loss, pred, confidences\nand in your loss function you may want to divide the coordinates by 2:
\n ...\n error = torch.sum(((gt - pred) * avails) ** 2, dim=-1) # reduce coords and use availability\n if cfg['train_params']['image_coords']:\n error = error / 4\n ...\nHey Everyone,
\n\n\nWhat do you think? I have a lingering feeling that I am missing something obvious!
\n
On the contrary, you're spot on :) This is something that slipped through our tests and that we only recently became aware of.
\nUsing pixel coordinates is absolutely legit here, but we're also working on a long term fix in L5Kit we hope we can ship next week. Once that is thoroughly checked we will sync with Kaggle to update the default package.
\nI'll start a post with more info about this issue and how we plan to fix it to increase visibility and awareness.
", "rawMarkdown": "Hey Everyone,\n> What do you think? I have a lingering feeling that I am missing something obvious!\n\nOn the contrary, you're spot on :) This is something that slipped through our tests and that we only recently became aware of.\n\nUsing pixel coordinates is absolutely legit here, but **we're also working on a long term fix in L5Kit** we hope we can ship next week. Once that is thoroughly checked we will sync with Kaggle to update the default package.\n\nI'll start a post with more info about this issue and how we plan to fix it to increase visibility and awareness.", "votes": 10, "replies": [ { "id": 1030060, "postDate": "2020-09-28T11:39:01.493Z", "content": "any updates?
", "rawMarkdown": "any updates?" }, { "id": 1030368, "postDate": "2020-09-28T15:58:08.520Z", "content": "almost ready to merge https://github.com/lyft/l5kit/pull/150 :) After that we will cut a new release and update Kaggle, doc and baseline
", "rawMarkdown": "almost ready to merge https://github.com/lyft/l5kit/pull/150 :) After that we will cut a new release and update Kaggle, doc and baseline", "votes": 2 }, { "id": 1030406, "postDate": "2020-09-28T16:41:34.090Z", "content": "I compared that PR to the fix posted here and looks like you're keeping the target positions in metres whereas above code converts to pixels, otherwise they match (barring slight inaccuracy likely due to floating point). That's the intended final result correct? (No issue, I think metres is better, just checking)
\nI gather also that you'll be updating the Kaggle ground truths so predictions will be in the new coordinates. Is that correct?
", "rawMarkdown": "I compared that PR to the fix posted here and looks like you're keeping the target positions in metres whereas above code converts to pixels, otherwise they match (barring slight inaccuracy likely due to floating point). That's the intended final result correct? (No issue, I think metres is better, just checking)\n\nI gather also that you'll be updating the Kaggle ground truths so predictions will be in the new coordinates. Is that correct?", "votes": 2 }, { "id": 1030418, "postDate": "2020-09-28T16:52:25.093Z", "content": "I think pixels are better, - in pixels they can be directly connected in creative ways to the images.
", "rawMarkdown": "I think pixels are better, - in pixels they can be directly connected in creative ways to the images.", "votes": 1 }, { "id": 1030428, "postDate": "2020-09-28T16:58:47.383Z", "content": "\n\nThat's the intended final result correct?
\n
yes, we're keeping metres (basically the only difference is a 2D rotation) :)
\n\n\nI gather also that you'll be updating the Kaggle ground truths so predictions will be in the new coordinates. Is that correct?
\n
We're not, this means you will have to convert them back into offsets in world coordinates (but we'll update the baseline notebook to show how to do that and we're adding a lot of matrices to make it easier). Updating the ground truth would have required a lot of additional testing, this was the best compromise :(
", "rawMarkdown": "> That's the intended final result correct?\n\nyes, we're keeping metres (basically the only difference is a 2D rotation) :)\n\n> I gather also that you'll be updating the Kaggle ground truths so predictions will be in the new coordinates. Is that correct?\n\nWe're not, this means you will have to convert them back into offsets in world coordinates (but we'll update the baseline notebook to show how to do that and we're adding a lot of matrices to make it easier). Updating the ground truth would have required a lot of additional testing, this was the best compromise :( ", "votes": 1 }, { "id": 1030433, "postDate": "2020-09-28T17:02:57.873Z", "content": "\n\nI think pixels are better
\n
How so? Either way you can just multiply/divide by pixel size right? I think if you want that you probably want to do it within a single model and then convert back to metres at the end. Otherwise if doing something like combining outputs from multiple models with different pixel sizes you'd have to be tracking the pixel sizes of all models.
\nWhat if you aren't even using a pixel based model, why convert to some arbitrary pizel size?
\n\nWe're not, this means you will have to convert them back into offsets in world coordinates (but we'll update the baseline notebook to show how to do that and we're adding a lot of matrices to make it easier). Updating the ground truth would have required a lot of additional testing, this was the best compromise :(
\n
Ah OK, I misunderstood the changes to the GT.csv code as meaning they were being updated to new coords when I guess they're converting new coords back to the old ones. Presumably they will match expected outputs on Kaggle.
\nI understand the compromise, obviously tricky either way.
\n\nI guess they're converting new coords back to the old ones. Presumably they will match expected outputs on Kaggle
\n
That's is exactly what's happening there :) I've checked and they match the old ones (also I've submitted a new baseline trained using the latest master and results are much better so I'm positive we didn't screw everything up completely)
", "rawMarkdown": "> I guess they're converting new coords back to the old ones. Presumably they will match expected outputs on Kaggle\n\nThat's is exactly what's happening there :) I've checked and they match the old ones (also I've submitted a new baseline trained using the latest master and results are much better so I'm positive we didn't screw everything up completely)", "votes": 2 }, { "id": 1030458, "postDate": "2020-09-28T17:20:15.487Z", "content": "Probably belongs elsewhere but in relation to the GCP credit prize for those beating the baseline does that mean they need to beat new baseline? Think the first deadline for that was tomorrow so not a lot of time to update models.
", "rawMarkdown": "Probably belongs elsewhere but in relation to the GCP credit prize for those beating the baseline does that mean they need to beat new baseline? Think the first deadline for that was tomorrow so not a lot of time to update models.", "votes": 2 }, { "id": 1030480, "postDate": "2020-09-28T17:29:02.487Z", "content": "Where is the new baseline result? I could not find it. Or if someone can just report the new score here?
", "rawMarkdown": "Where is the new baseline result? I could not find it. Or if someone can just report the new score here? " }, { "id": 1030498, "postDate": "2020-09-28T17:35:05.717Z", "content": "82.246
\nhttps://www.kaggle.com/lucabergamini/lyft-baseline-09-02
\n\nHow so? Either way you can just multiply/divide by pixel size right? I think if you want that you probably want to do it within a single model and then convert back to metres at the end. Otherwise if doing something like combining outputs from multiple models with different pixel sizes you'd have to be tracking the pixel sizes of all models.
\n
\n What if you aren't even using a pixel based model, why convert to some arbitrary pizel size?
@thomasbrandon , you have good points, hard to argue with what you say. In defense of pixels, - we get data object in which I would naturally expect everything to be in the same coordinate system. There is the image object, with its own coordinate system, so everything needs to match. My second argument is that if everything is in pixels then we can easily draw on the image without any additional transformations, rasterize those pixels into something etc.
Did you calculate bias = torch.tensor([56.25, 112.5]) from the ego_center config? So the input size is 225x225px?
That was my question as well. I assume this will not be the correct alignment at other resolutions.
", "rawMarkdown": "That was my question as well. I assume this will not be the correct alignment at other resolutions. ", "votes": 1 }, { "id": 1025694, "postDate": "2020-09-24T17:58:03.900Z", "content": "Maybe this..
\nrs = cfg[\"raster_params\"][\"raster_size\"]\nec = cfg[\"raster_params\"][\"ego_center\"]\n\nbias = torch.tensor([rs[0] * ec[0], rs[1] * ec[1]])[None, None, :].to(device)\nExactly right, thank you! The point of this is of course to have non-moving target at zero, as it was before.
", "rawMarkdown": "Exactly right, thank you! The point of this is of course to have non-moving target at zero, as it was before. ", "votes": 2 } ] }, { "id": 1029906, "postDate": "2020-09-28T08:37:46.400Z", "content": "Thank you for sharing this. We were doing it completely wrong, and most people would have been oblivious to that fact had you not so generously highlighted it. The epitome of community - kudos.
", "rawMarkdown": "Thank you for sharing this. We were doing it completely wrong, and most people would have been oblivious to that fact had you not so generously highlighted it. The epitome of community - kudos.", "votes": 6 }, { "id": 1027976, "postDate": "2020-09-26T13:47:09.237Z", "content": "Thank you for sharing.
\nWith this approach, in my case, the model(45.0) trained on all of the datasets was improved to the model(37.7) trained on 1/4 of the dataset.
\nHowever, there is a gap between validation metric (13.1) and public LB.
Your validation metric is too small, this can be because you divide the errors by 2 after the predictions are transformed. Basically check that the errors are divided by 2 only when needed.
", "rawMarkdown": "Your validation metric is too small, this can be because you divide the errors by 2 **after** the predictions are transformed. Basically check that the errors are divided by 2 only when needed.", "votes": 1 }, { "id": 1028488, "postDate": "2020-09-26T22:29:26.957Z", "content": "Sorry. When I checked it, I used the square error, so I divided it by 4.
", "rawMarkdown": "Sorry. When I checked it, I used the square error, so I divided it by 4." } ] }, { "id": 1026441, "postDate": "2020-09-25T10:22:50.063Z", "content": "Good approach @zaharch . Congrats !
\nCan I ask you why do you choose to divide the error by 4 ?
In case one uses default configuration there is a scaling factor 'pixel_size': [0.5, 0.5]. So there are two pixels (new coordinates) per meter (old coordinates). To get the error back into the old scale I need to divide it by 2, or by 4 in case of squared error.
Depending on your model it may not be necessary to scale, you will just have 4x bigger loss. But I train a multi-scenario output, and there is a subtle inter-play between the errors and the confidences. Therefore I want to keep it at the old scale, which is the correct one for the loss.
", "rawMarkdown": "In case one uses default configuration there is a scaling factor `'pixel_size': [0.5, 0.5]`. So there are two pixels (new coordinates) per meter (old coordinates). To get the error back into the old scale I need to divide it by 2, or by 4 in case of squared error. \n\nDepending on your model it may not be necessary to scale, you will just have 4x bigger loss. But I train a multi-scenario output, and there is a subtle inter-play between the errors and the confidences. Therefore I want to keep it at the old scale, which is the correct one for the loss.", "votes": 5 } ] }, { "id": 1025888, "postDate": "2020-09-24T21:06:16.230Z", "content": "We really did it all wrong! Thanks for sharing and pointing it out.
", "rawMarkdown": "We really did it all wrong! Thanks for sharing and pointing it out. ", "votes": 3 }, { "id": 1031260, "postDate": "2020-09-29T10:54:47.223Z", "content": "I'm really new to this community and this helped me a lot. Thanks!
", "rawMarkdown": "I'm really new to this community and this helped me a lot. Thanks!", "votes": 1 }, { "id": 1028934, "postDate": "2020-09-27T10:36:53.357Z", "content": "Hi, Great insight! Can i know how you obtained those graphs?
", "rawMarkdown": "Hi, Great insight! Can i know how you obtained those graphs?", "votes": 1, "replies": [ { "id": 1028945, "postDate": "2020-09-27T10:44:40.177Z", "content": "Sure, the first graph is this code:
\nfig = plt.figure(figsize = (12,12))\nfor k in range(500):\n data = valid_dataset[4001+1000*k]\n tgt = data[\"target_positions\"]\n plt.plot(tgt[:,0],tgt[:,1])\nthe second is similar but with the transformation, depends on how you shape your code.
", "rawMarkdown": "Sure, the first graph is this code:\n\n```\nfig = plt.figure(figsize = (12,12))\nfor k in range(500):\n data = valid_dataset[4001+1000*k]\n tgt = data[\"target_positions\"]\n plt.plot(tgt[:,0],tgt[:,1])\n```\n\nthe second is similar but with the transformation, depends on how you shape your code.", "votes": 5 }, { "id": 1028966, "postDate": "2020-09-27T11:04:52.797Z", "content": "Thanks a Lot! By the looks of it… the valid_dataset suggests you are using an evaluation dataset… why? just curious…
", "rawMarkdown": "Thanks a Lot! By the looks of it... the valid_dataset suggests you are using an evaluation dataset... why? just curious..." }, { "id": 1029308, "postDate": "2020-09-27T16:34:59.013Z", "content": "No reason, it doesn't really matter because it is just targets from the data. I don't think there is any meaningful difference between the train and the validation.
", "rawMarkdown": "No reason, it doesn't really matter because it is just targets from the data. I don't think there is any meaningful difference between the train and the validation.", "votes": 1 }, { "id": 1029763, "postDate": "2020-09-28T05:48:28.323Z", "content": "Understood! thanks again😀
", "rawMarkdown": "Understood! thanks again😀\n", "votes": 1 } ] }, { "id": 1027575, "postDate": "2020-09-26T08:43:53.657Z", "content": "@zaharch does this mean while inference we have to transform the predicted trajectories using the code block you shared
\n if cfg['train_params']['image_coords']:\n matrix_inv = torch.inverse(matrix)\n pred = pred + bias[:,None,:,:]\n pred = torch.cat([pred,torch.ones((bs,3,tl,1)).to(device)], dim=3)\n pred = torch.stack([torch.matmul(matrix_inv.to(torch.float), pred[:,i].transpose(1,2)) \n for i in range(3)], dim=1)\n pred = pred.transpose(2,3)[:,:,:,:2]\n pred = pred - centroid[:,None,:,:]\nTrying to understand if I am interpreting the targets correctly
", "rawMarkdown": "@zaharch does this mean while inference we have to transform the predicted trajectories using the code block you shared\n```\n if cfg['train_params']['image_coords']:\n matrix_inv = torch.inverse(matrix)\n pred = pred + bias[:,None,:,:]\n pred = torch.cat([pred,torch.ones((bs,3,tl,1)).to(device)], dim=3)\n pred = torch.stack([torch.matmul(matrix_inv.to(torch.float), pred[:,i].transpose(1,2)) \n for i in range(3)], dim=1)\n pred = pred.transpose(2,3)[:,:,:,:2]\n pred = pred - centroid[:,None,:,:]\n```\nTrying to understand if I am interpreting the targets correctly", "votes": 1, "replies": [ { "id": 1027577, "postDate": "2020-09-26T08:44:51.447Z", "content": "Yes, in inference you will have to use the same code block.
", "rawMarkdown": "Yes, in inference you will have to use the same code block.", "votes": 2 }, { "id": 1027613, "postDate": "2020-09-26T08:56:53.187Z", "content": "Yes, that is right, same for inference.
", "rawMarkdown": "Yes, that is right, same for inference.", "votes": 1 }, { "id": 1027628, "postDate": "2020-09-26T09:06:41.810Z", "content": "Interesting, I trained model for 6 epochs 10k steps each my train loss was around ~60.x but the inference kernel with transformation gave really bad result. Anything obvious I might be missing?
\nNote that I am using raster rize 224x224 so I changed the bias value.
\nInference code for reference
\n inputs = data[\"image\"].to(device)\n #target_availabilities = data[\"target_availabilities\"].unsqueeze(-1).to(device)\n target_availabilities = data[\"target_availabilities\"].to(device)\n targets = data[\"target_positions\"].to(device)\n matrix = data[\"world_to_image\"].to(device)\n centroid = data[\"centroid\"].to(device)[:,None,:].to(torch.float)\n\n bs,tl,_ = targets.shape\n rs = cfg[\"raster_params\"][\"raster_size\"]\n ec = cfg[\"raster_params\"][\"ego_center\"]\n bias = torch.tensor([rs[0] * ec[0], rs[1] * ec[1]])[None, None, :].to(device)\n\n\n #outputs = model(inputs).reshape(targets.shape)\n pred, confidences = model(inputs)\n\n if cfg['test_params']['image_coords']:\n matrix_inv = torch.inverse(matrix)\n pred = pred + bias[:,None,:,:]\n pred = torch.cat([pred,torch.ones((bs,3,tl,1)).to(device)], dim=3)\n pred = torch.stack([torch.matmul(matrix_inv.to(torch.float), pred[:,i].transpose(1,2)) \n for i in range(3)], dim=1)\n pred = pred.transpose(2,3)[:,:,:,:2]\n pred = pred - centroid[:,None,:,:]\nHave you changed the bias to 56,112? How do you judge that the result is bad? Do prediction start from close to zero? I guess it is some kind of a technical error.
\nDo you do softmax for the confidences and pred shape transformations inside the model?
", "rawMarkdown": "Have you changed the bias to 56,112? How do you judge that the result is bad? Do prediction start from close to zero? I guess it is some kind of a technical error.\n\nDo you do softmax for the confidences and pred shape transformations inside the model?" }, { "id": 1027654, "postDate": "2020-09-26T09:31:26.883Z", "content": "Have you changed the bias to 56,112?
\nYes,
\nDo you do softmax for the confidences and pred shape transformations inside the model?
\nYes
\nDo prediction start from close to zero?
\nYes
\nHow do you judge that the result is bad?
\nI am comparing model performance against my old model without transformation after n epochs. Model without transformations seemed to have converged faster in my case. I guess I'll dig deeper into this by comparing my results
Awesome point. Thank you for sharing this
", "rawMarkdown": "Awesome point. Thank you for sharing this", "votes": 1 }, { "id": 1064052, "postDate": "2020-10-29T16:30:18.540Z", "content": "important!!!!
\nSo I tried to modify the forward function above for the new l5kit package (version 1.1.0) following the notebook Lyft-baseline-09-02
\ndef forward(data, model, device, criterion = pytorch_neg_multi_log_likelihood_batch):\n inputs = data[\"image\"].to(device)\n target_availabilities = data[\"target_availabilities\"].to(device)\n targets = data[\"target_positions\"].to(device)\n\n # Forward pass\n preds, confidences = model(inputs) \n\n\n loss = criterion(targets, preds, confidences, target_availabilities)\n\n\n world_from_agents = data[\"world_from_agent\"].numpy()\n centroids = data[\"centroid\"].numpy()\n\n #convert into world coordinates and compute offsets\n for idx in range(len(preds)):\n preds[idx] = transform_points(preds[idx], world_from_agents[idx]) - centroids[idx]\n\n return loss, preds, confidences\nSo as I understand the target in the train set is fixed, however the test set is not so we still need to convert it? However when I ran the above code I encountered the following error:
\n---------------------------------------------------------------------------\nAssertionError Traceback (most recent call last)\n<ipython-input-24-9205635f7dc5> in <module>\n 12 for data in progress_bar:\n 13 inputs = data['image'].to(device)\n---> 14 _, preds, confidences = forward(data, model, device)\n 15 \n 16 # # convert agent coordinates into world offsets\n\n<ipython-input-22-276920502171> in forward(data, model, device, criterion)\n 40 #convert into world coordinates and compute offsets\n 41 for idx in range(len(preds)):\n---> 42 preds[idx] = transform_points(preds[idx], world_from_agents[idx]) - centroids[idx]\n 43 \n 44 return loss, preds, confidences\n\n/kaggle/usr/lib/lyft_l5kit_unofficial_fix/l5kit/geometry/transform.py in transform_points(points, transf_matrix)\n 87 np.ndarray: array of shape (N,2) for 2D input points, or (N,3) points for 3D input points\n 88 \"\"\"\n---> 89 assert len(points.shape) == len(transf_matrix.shape) == 2\n 90 assert transf_matrix.shape[0] == transf_matrix.shape[1]\n 91 \n\nAssertionError: \nDoes anyone know what is going on? Also I assume that with the fix I do not have to divide the error by 4 anymore? Any insight is appreciated.
\nThanks a lot for your help!
", "rawMarkdown": "So I tried to modify the *forward* function above for the new l5kit package (version 1.1.0) following the notebook [Lyft-baseline-09-02](https://www.kaggle.com/lucabergamini/lyft-baseline-09-02?scriptVersionId=43751473)\n```\ndef forward(data, model, device, criterion = pytorch_neg_multi_log_likelihood_batch):\n inputs = data[\"image\"].to(device)\n target_availabilities = data[\"target_availabilities\"].to(device)\n targets = data[\"target_positions\"].to(device)\n \n # Forward pass\n preds, confidences = model(inputs) \n \n \n loss = criterion(targets, preds, confidences, target_availabilities)\n \n \n world_from_agents = data[\"world_from_agent\"].numpy()\n centroids = data[\"centroid\"].numpy()\n \n #convert into world coordinates and compute offsets\n for idx in range(len(preds)):\n preds[idx] = transform_points(preds[idx], world_from_agents[idx]) - centroids[idx]\n \n return loss, preds, confidences\n```\nSo as I understand the target in the train set is fixed, however the test set is not so we still need to convert it? However when I ran the above code I encountered the following error:\n\n```\n---------------------------------------------------------------------------\nAssertionError Traceback (most recent call last)\nThe official example is only for single-mode prediction. For multi-mode I am doing something like this (when making predictions):
\noutputs = outputs.cpu().numpy()\nworld_from_agents = data[\"world_from_agent\"].numpy()\ncentroids = data[\"centroid\"].numpy()\ncoords_offset = []\n\nfor agent_coords, world_from_agent, centroid in zip(outputs, world_from_agents, centroids):\n for i in range(3):\n agent_coords[i, :, :] = transform_points(agent_coords[i, :, :], world_from_agent) - centroid[:2]\n coords_offset.append(agent_coords)\nYou can use some matrix manipulation to avoid the loops so it could be a liitle bit faster but this seems to work fine for me.
", "rawMarkdown": "The official example is only for single-mode prediction. For multi-mode I am doing something like this (when making predictions):\n\n```\noutputs = outputs.cpu().numpy()\nworld_from_agents = data[\"world_from_agent\"].numpy()\ncentroids = data[\"centroid\"].numpy()\ncoords_offset = []\n\nfor agent_coords, world_from_agent, centroid in zip(outputs, world_from_agents, centroids):\n for i in range(3):\n agent_coords[i, :, :] = transform_points(agent_coords[i, :, :], world_from_agent) - centroid[:2]\n coords_offset.append(agent_coords)\n```\nYou can use some matrix manipulation to avoid the loops so it could be a liitle bit faster but this seems to work fine for me.", "votes": 3 }, { "id": 1037644, "postDate": "2020-10-05T07:40:22.533Z", "content": "We're considering to support batch matrix multiplication for that function in the future release. It's not anything difficult to do (numpy can handle it without any troubles ), we just didn't have enough time to implement and test it :(
", "rawMarkdown": "We're considering to support batch matrix multiplication for that function in the future release. It's not anything difficult to do (numpy can handle it without any troubles ), we just didn't have enough time to implement and test it :( ", "votes": 3, "replies": [ { "id": 1038958, "postDate": "2020-10-06T07:42:41.653Z", "content": "\n\nWe're considering to support batch matrix multiplication for that function in the future release. It's not anything difficult to do (numpy can handle it without any troubles ), we just didn't have enough time to implement and test it :(
\n
worthy of trying https://github.com/lyft/l5kit/pull/166
", "rawMarkdown": "> We're considering to support batch matrix multiplication for that function in the future release. It's not anything difficult to do (numpy can handle it without any troubles ), we just didn't have enough time to implement and test it :(\n\nworthy of trying https://github.com/lyft/l5kit/pull/166" } ] }, { "id": 1037713, "postDate": "2020-10-05T09:01:43.140Z", "content": "@frankpanxj Is future_coords_offsets_pd.append() in your code coords_offset.append(agent_coords) or do you stack it at the end?
EDIT: seems to need future_coords_offsets_pd.append(np.stack(coords_offset))
@huanvo I have tried the same yesterday, it is like mentioned because of single model. They also use different shapes e.g.output = model(inputs).reshape(targets.shape)
I tried to change everything accordingly but I ended up using the mentioned solution from @zaharch
\nI will try a couple of things out and report back.
UPDATE: @frankpanxj Solution works best for me.
", "rawMarkdown": "@frankpanxj Is ` future_coords_offsets_pd.append()` in your code `coords_offset.append(agent_coords)` or do you stack it at the end?\n\n**EDIT**: seems to need `future_coords_offsets_pd.append(np.stack(coords_offset))`\n\n@huanvo I have tried the same yesterday, it is like mentioned because of single model. They also use different shapes e.g.`output = model(inputs).reshape(targets.shape)` \n\nI tried to change everything accordingly but I ended up using the mentioned solution from @zaharch\nI will try a couple of things out and report back.\n\n**UPDATE**: @frankpanxj Solution works best for me.", "votes": 1 }, { "id": 1037772, "postDate": "2020-10-05T10:19:25.590Z", "content": "Thanks for the input @frankpanxj . Is coords_offset the same shape as the original agents_coords ? It can replace it when we pass it to the loss function to get the metric of the training ?
", "rawMarkdown": "Thanks for the input @frankpanxj . Is coords_offset the same shape as the original agents_coords ? It can replace it when we pass it to the loss function to get the metric of the training ?", "votes": 1 }, { "id": 1037822, "postDate": "2020-10-05T11:10:27.450Z", "content": "@aliabdin1 You are right. You need to stack the reults.
", "rawMarkdown": "@aliabdin1 You are right. You need to stack the reults." }, { "id": 1037828, "postDate": "2020-10-05T11:14:48.693Z", "content": "@vladvdv the shape of outputs is (batch size)x(modes)x(time)x(2D coords), which is needed for the loss function by @corochann at https://www.kaggle.com/corochann/lyft-training-with-multi-mode-confidence.
\ncoords_offset here is just a list of the preds for every sample, after stacking it you will get a np array with the same shape as outputs. (I changed some the variable names above to make this clearer.)
Thanks @frankpanxj this seems to resolve my error. However my prediction error got much worse in the new l5kit environment (old env 18.x vs new env 4xxx).
\nMaybe I misunderstood something here. I was using the fix by @zaharch before in the old environment and it seemed to work fine.
@huanvo have similar issues. Did you find the problem ?
\nlater edit: I was doing the transform at training time also. From what I understand now, the transformation should be done only on inference !
The fix posted here is different to the fix in l5kit v1.1.0. In the fix here it not only corrects for the rotation error but also converts from metres to pixels. The fix in v1.1 only corrects the rotation, leaving things in metres. So models trained with the fix here aren't compatible with the fix in v1.1.
\nTo use a model trained with the fix here in v1.1 you'd need to convert the targets from metres to pixels and then transform the predictions from pixels to metres before applying the rotation fix from the l5kit docs.
I used the above @frankpanxj 's code to inference a model trained with L5kit ver1.1.0 but my submission file created only up to coordinates 0-29 i.e. coord_x29, coord_y29.
\nDoes anybody experience the same thing?
\nI could not see what I am doing wrong in my code right now.
", "rawMarkdown": "I used the above @frankpanxj 's code to inference a model trained with L5kit ver1.1.0 but my submission file created only up to coordinates 0-29 i.e. coord_x29, coord_y29. \n\nDoes anybody experience the same thing?\n\n I could not see what I am doing wrong in my code right now." }, { "id": 1038634, "postDate": "2020-10-05T23:20:19.687Z", "content": "Hi @huanvo , I tried zaharch's approach and got similar results as using v1.1. However, my score was never that good (my validation loss was always much higher than my training loss, did you experience the same?) so there certainly is something I am missing.
\n@sheriytm very interesting results. I did not have the same problem. There are coords beyond 29 in my results.
", "rawMarkdown": "Hi @huanvo , I tried zaharch's approach and got similar results as using v1.1. However, my score was never that good (my validation loss was always much higher than my training loss, did you experience the same?) so there certainly is something I am missing.\n\n@sheriytm very interesting results. I did not have the same problem. There are coords beyond 29 in my results.", "votes": 1 }, { "id": 1040128, "postDate": "2020-10-07T01:48:21.617Z", "content": "Thanks @frankpanxj. I just found the bug that caused cuts out the rest of the coordinates during prediction.
", "rawMarkdown": "Thanks @frankpanxj. I just found the bug that caused cuts out the rest of the coordinates during prediction.", "votes": 1 }, { "id": 1041478, "postDate": "2020-10-07T19:04:16.717Z", "content": "I'm a bit confused. My results seem to be completely non-sensical with the new version and the example code provided. The issue seems to be stemming from subtracting the centroid. My centroid is values in the thousands typically and shifting my predictions way too much. Anyone having similar experience or an explanation of what I might be doing wrong?
", "rawMarkdown": "I'm a bit confused. My results seem to be completely non-sensical with the new version and the example code provided. The issue seems to be stemming from subtracting the centroid. My centroid is values in the thousands typically and shifting my predictions way too much. Anyone having similar experience or an explanation of what I might be doing wrong?" }, { "id": 1041480, "postDate": "2020-10-07T19:05:52.430Z", "content": "Did you remove the here mentioned training snippet? Its not needed anymore.
", "rawMarkdown": "Did you remove the here mentioned training snippet? Its not needed anymore." }, { "id": 1041493, "postDate": "2020-10-07T19:17:16.053Z", "content": "Yes I am training on the raw outputs from the dataloader on the newest version of l5kit. No transforms anymore. Training loss looks fine but validation is messed up. in the hundreds of thousands using roughly the code that was shared here. My predictions should not be in the thousands, right?
", "rawMarkdown": "Yes I am training on the raw outputs from the dataloader on the newest version of l5kit. No transforms anymore. Training loss looks fine but validation is messed up. in the hundreds of thousands using roughly the code that was shared here. My predictions should not be in the thousands, right?" }, { "id": 1041494, "postDate": "2020-10-07T19:18:10.640Z", "content": "Hi @ryches , the new version works fine for me, I haven't noticed anything interesting changed after the switch. Please share the code.
", "rawMarkdown": "Hi @ryches , the new version works fine for me, I haven't noticed anything interesting changed after the switch. Please share the code." }, { "id": 1041514, "postDate": "2020-10-07T19:34:51.987Z", "content": "My outputs from my model are roughly what they were prior to the l5kit upgrade. Previously I was doing the transform and then inverse transform and it was working fine.
\nNow with my model trained on the new l5kit targets without any transformations its max output on x, y coordinates is ~77 which seems normal, but centroid is showing up as 785, -2024. Validation predictions used to look like ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1035002%2F38339d77fe0592dbb4b92bed316dfff3%2FScreenshot%20from%202020-10-07%2012-33-02.png?generation=1602099206020101&alt=media\")
but now they look like this and the big difference seems to be these huge centroid outputs ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1035002%2Fb7d99f675f22689d08b7ad4a37d43907%2FScreenshot%20from%202020-10-07%2012-33-48.png?generation=1602099263121087&alt=media\")
Is the l5kit version you are using from pip or directly from github?
", "rawMarkdown": "My outputs from my model are roughly what they were prior to the l5kit upgrade. Previously I was doing the transform and then inverse transform and it was working fine. \n\nNow with my model trained on the new l5kit targets without any transformations its max output on x, y coordinates is ~77 which seems normal, but centroid is showing up as 785, -2024. Validation predictions used to look like \n\nbut now they look like this and the big difference seems to be these huge centroid outputs \n\nIs the l5kit version you are using from pip or directly from github?" }, { "id": 1041527, "postDate": "2020-10-07T19:42:52.290Z", "content": "The code I have been poking at to try to understand. If I dont subtract the huge centroids then I still get questionable results
\ntorch.set_grad_enabled(False)\nwith torch.no_grad():\n model.eval()\n # store information for evaluation\n future_coords_offsets_pd = []\n timestamps = []\n agent_ids = []\n confidences = []\n progress_bar = tqdm(eval_dataloader)\n for data in progress_bar:\n inputs = data\n targets = data[\"target_positions\"]\n timestamps.append(data[\"timestamp\"].cpu().numpy())\n agent_ids.append(data[\"track_id\"].cpu().numpy())\n outputs, confs = model(inputs)\n# outputs = inverse_tranform_predictions(outputs, data[\"world_to_image\"], data[\"centroid\"])\n# future_coords_offsets_pd.append(outputs.cpu().numpy())\n for agent_coords, world_from_agent, centroid in zip(outputs.cpu().numpy(), \n data[\"world_to_image\"].cpu().numpy(),\n data[\"centroid\"].cpu().numpy()):\n for mode in range(3):\n agent_coords[mode, :, :] = transform_points(agent_coords[mode, :, :], world_from_agent) - centroid[:2]\n future_coords_offsets_pd.append(agent_coords)\n confidences.append(confs.cpu().numpy())\n pred_path = f\"{gettempdir()}/validation_preds.csv\"\n write_pred_csv(pred_path,\n timestamps=np.concatenate(timestamps),\n track_ids=np.concatenate(agent_ids),\n coords=np.stack(future_coords_offsets_pd),\n confs = np.concatenate(confidences)\n )\n del confidences, future_coords_offsets_pd\n metrics = compute_metrics_csv(eval_gt_path, pred_path, [neg_multi_log_likelihood, time_displace])\n for metric_name, metric_mean in metrics.items():\n print(metric_name, metric_mean)\nthis is how you need to transform for multimode:
\n # convert agent coordinates into world offsets\n pred = pred.cpu().numpy()\n world_from_agents = data[\"world_from_agent\"].numpy()\n centroids = data[\"centroid\"].numpy()\n coords_offset = []\n\n # convert into world coordinates and compute offsets\n for idx in range(len(pred)):\n for mode in range(3):\n pred[idx, mode, :, :] = transform_points(pred[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]\n\n confidences_list.append(confidences.cpu().numpy().copy())\n pred_coords_list.append(pred.copy())\n timestamps.append(data[\"timestamp\"].numpy().copy())\n agent_ids.append(data[\"track_id\"].numpy().copy())\nI see that you used a batched approach @ryches , that could be the problem. I am not sure if that works for multi-mode. I am actually pretty sure that it doesn't work as the code for transform_points is expecting points in (Nx2) or (Nx3).
", "rawMarkdown": "this is how you need to transform for multimode:\n\n```\n # convert agent coordinates into world offsets\n pred = pred.cpu().numpy()\n world_from_agents = data[\"world_from_agent\"].numpy()\n centroids = data[\"centroid\"].numpy()\n coords_offset = []\n\n # convert into world coordinates and compute offsets\n for idx in range(len(pred)):\n for mode in range(3):\n pred[idx, mode, :, :] = transform_points(pred[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]\n\n confidences_list.append(confidences.cpu().numpy().copy())\n pred_coords_list.append(pred.copy())\n timestamps.append(data[\"timestamp\"].numpy().copy())\n agent_ids.append(data[\"track_id\"].numpy().copy())\n```\n\nI see that you used a batched approach @ryches , that could be the problem. I am not sure if that works for multi-mode. I am actually pretty sure that it doesn't work as the code for transform_points is expecting points in (Nx2) or (Nx3).", "votes": 7 }, { "id": 1041539, "postDate": "2020-10-07T19:50:27.217Z", "content": "Thanks, I will look at that and try it out
", "rawMarkdown": "Thanks, I will look at that and try it out" }, { "id": 1041548, "postDate": "2020-10-07T20:03:06.530Z", "content": "Maybe the problem is that you are using data[\"world_to_image\"], instead of world_from_agent
Looks like that works. Thank you so much. Devil is in the details
", "rawMarkdown": "Looks like that works. Thank you so much. Devil is in the details" }, { "id": 1041559, "postDate": "2020-10-07T20:08:28.657Z", "content": "Yeah on second review I think the key was using the right transformation matrix. I was using the same method of zipping across the three inputs for the batches as shown in the example notebook and then iterating 3 times for the various modes as others have mentioned in this thread. I think my dimensions and operations were correct but I was using the wrong matrix.
", "rawMarkdown": "Yeah on second review I think the key was using the right transformation matrix. I was using the same method of zipping across the three inputs for the batches as shown in the example notebook and then iterating 3 times for the various modes as others have mentioned in this thread. I think my dimensions and operations were correct but I was using the wrong matrix. ", "votes": 2 }, { "id": 1041564, "postDate": "2020-10-07T20:12:05.090Z", "content": "Great to hear, i'll switch to batched transformation, too if that works ;)
", "rawMarkdown": "Great to hear, i'll switch to batched transformation, too if that works ;)" }, { "id": 1041590, "postDate": "2020-10-07T20:36:18.320Z", "content": "Probably not worth switching. I did not see any appreciable difference in speed
", "rawMarkdown": "Probably not worth switching. I did not see any appreciable difference in speed" } ] }, { "id": 1030216, "postDate": "2020-09-28T13:34:14.877Z", "content": "First @corochann helped the heard think & helped us and now you ( @zaharch ) are doing the same. I like to thank you both.
", "rawMarkdown": "First @corochann helped the heard think & helped us and now you ( @zaharch ) are doing the same. I like to thank you both. ", "votes": 2, "replies": [ { "id": 1030748, "postDate": "2020-09-28T23:22:36.097Z", "content": "Thanks for feedback & mention :)
\nI couldn't notice this before @zaharch pointed out, even I go through the EDA in several notebooks…
hey @nosound, thanks for sharing this code and these insights. I'm still noob at ML and DS, but I've been \"studying\" some of the newest codes at current competitions and i feel I can get what you mean by directionless input's. don't know how to approach directly to that matter but If i get something I'll share it =) thanks for it.
", "rawMarkdown": "hey @nosound, thanks for sharing this code and these insights. I'm still noob at ML and DS, but I've been \"studying\" some of the newest codes at current competitions and i feel I can get what you mean by directionless input's. don't know how to approach directly to that matter but If i get something I'll share it =) thanks for it.", "votes": 2 }, { "id": 1028503, "postDate": "2020-09-26T23:18:52.250Z", "content": "The fact the original model performed well without any knowledge about the agent orientation raises the question about how generalisable the models are, even with the orientation fix applied. Would be easy to fix by the position specific train/validation split.
\nWould be great if hosts allocated the test set in the different geographic position (and not available until the models freeze) to check how well solutions perform in more realistic situation of roads model have not seen during training, but I'd expect it's too much to ask at this stage.
", "rawMarkdown": "The fact the original model performed well without any knowledge about the agent orientation raises the question about how generalisable the models are, even with the orientation fix applied. Would be easy to fix by the position specific train/validation split.\n\nWould be great if hosts allocated the test set in the different geographic position (and not available until the models freeze) to check how well solutions perform in more realistic situation of roads model have not seen during training, but I'd expect it's too much to ask at this stage.", "votes": 2, "replies": [ { "id": 1028864, "postDate": "2020-09-27T09:08:52.447Z", "content": "On the other hand if the interesting zone is limited to what we have and AVs do not drive out of this zone anyway, then it is OK to overfit on it. But you have a good point.
\nIf direction is possible to learn then the models potentially can learn average speeds on those roads and common routes taken by cars, like rare vs common turns.
", "rawMarkdown": "On the other hand if the interesting zone is limited to what we have and AVs do not drive out of this zone anyway, then it is OK to overfit on it. But you have a good point.\n\nIf direction is possible to learn then the models potentially can learn average speeds on those roads and common routes taken by cars, like rare vs common turns.", "votes": 1 } ] }, { "id": 1027116, "postDate": "2020-09-25T21:04:05.090Z", "content": "Fantastic post, thanks for sharing! Will implement and report results :)
", "rawMarkdown": "Fantastic post, thanks for sharing! Will implement and report results :)", "votes": 2, "replies": [ { "id": 1029481, "postDate": "2020-09-27T19:16:51.673Z", "content": "Can confirm this provided a great boost to models from 3x.x to 2x.x, and current score 18.x using this approach too. Thanks again to nosound for sharing. Super interested to better understand how we were doing so well before, very odd!
", "rawMarkdown": "Can confirm this provided a great boost to models from 3x.x to 2x.x, and current score 18.x using this approach too. Thanks again to nosound for sharing. Super interested to better understand how we were doing so well before, very odd!", "votes": 2 } ] }, { "id": 1026626, "postDate": "2020-09-25T13:07:50.927Z", "content": "Oh wow thanks for sharing this! It took me forever to bring the error down, and it is not even close to 23. On the other hand, it is still surprising how the model can learn without any directions.
", "rawMarkdown": "Oh wow thanks for sharing this! It took me forever to bring the error down, and it is not even close to 23. On the other hand, it is still surprising how the model can learn without any directions. ", "votes": 2, "replies": [ { "id": 1051298, "postDate": "2020-10-16T11:09:36.493Z", "content": "@huanvo I don't understand about what are you talking about
\nbut do have a question, please
\nRecently you have posted a high scoring kernel.
\nis the forward function which mentioned by nosound on the discussion topic in that kernel
Would it be possible for you to create a plot of the transformed paths like you did for the originals? I'd be interested to see what that looks like
", "rawMarkdown": "Would it be possible for you to create a plot of the transformed paths like you did for the originals? I'd be interested to see what that looks like", "votes": 2, "replies": [ { "id": 1025860, "postDate": "2020-09-24T20:04:23.183Z", "content": "Will do when I get home.
\nUpdate: added
", "rawMarkdown": "Will do when I get home.\n\nUpdate: added", "votes": 4 } ] }, { "id": 1051279, "postDate": "2020-10-16T10:35:24.620Z", "content": "I have a few questions after a long time of discussion of different people around the world
\nQuestion
\nHi, @morizin , I am glad you join the discussion!
\nnot sure what to comment on your other points.
", "rawMarkdown": "Hi, @morizin , I am glad you join the discussion!\n\n1. No, the coordinates have already been rotated in 1.1.0, this code is not needed. And no, no need to divide by 4 in the new version, because they do not work in pixels.\n\nnot sure what to comment on your other points." } ] }, { "id": 1039767, "postDate": "2020-10-06T19:21:11.993Z", "content": "I'm really new to this community and this helped me a lot. Thanks!
", "rawMarkdown": "I'm really new to this community and this helped me a lot. Thanks!" }, { "id": 1037514, "postDate": "2020-10-05T05:00:42.687Z", "content": "Thanks for sharing! This was extremely useful.
\nWondering if the following achieves the same transformation:
\n\ntransform_points(data['target_positions'], data['raster_from_agent'])\n
Why is centroid added to target_positions? (The centroid is the initial position of the agent in world coordinates?)
", "rawMarkdown": "Why is centroid added to target_positions? (The centroid is the initial position of the agent in world coordinates?)", "replies": [ { "id": 1034058, "postDate": "2020-10-01T13:11:20.757Z", "content": "Instead of understanding this I recommend to move to the new l5kit version, it is different there. Go to the github page and read the coordinate system description and see the example notebooks.
", "rawMarkdown": "Instead of understanding this I recommend to move to the new l5kit version, it is different there. Go to the github page and read the coordinate system description and see the example notebooks.", "votes": 2 }, { "id": 1034119, "postDate": "2020-10-01T14:11:38.867Z", "content": "Yeah, the process is different in the l5kit implementation but you do still need to add/subtract the centroid in some places. In both approaches this is because the targets are relative positions from the initial agent position while some of the transformation matrices need absolute positions. So adding the centroid to relative world coordinates converts to absolute world coordinates (and vice-versa subtracting it). Though as @zaharch says the places where it's needed differ from the code here as some of the new transformation matrices include this translation.
\nSo with l5 kit 1.1 you see that:
\n> transform_point(np.array([0,0]), data['world_from_agent'])\narray([ 671.78094482, -2196.83374023])\nis adding the centroid in and converting the rotation. So you then subtract the centroid from the result to get back to the initial relative world targets.
\nTo go back the other way from old world relative target coordinates to agent relative coordinates you add the centroid and then transform. So:
\n>target_in_agent = np.array([0.1, 0.2])\n>target_in_world = transform_point(target_in_agent, data['world_from_agent']) - data['centroid']\n>print(\"Relative World coords: \", target_in_world)\n>print(\"Recovered: \", transform_point(target_in_world + data['centroid'], data['agent_from_world']))\nWorld coords: [-0.11271524 0.19311985]\nRecovered: [0.1 0.2]\nNote the centroid is that same in each case as the change simply involves how targets are rotated around this centroid. Previously they were rotated with respect to the Autonomous Vehicle (/ego), now they are rotated with respect to the actual agent vehicle you're predicting (which may or may not be the ego). In both cases they're relative to centroid.
\n(I think, obviously all a little complex and certainly encourage you to check my work)
", "rawMarkdown": "Yeah, the process is different in the l5kit implementation but you do still need to add/subtract the centroid in some places. In both approaches this is because the targets are relative positions from the initial agent position while some of the transformation matrices need absolute positions. So adding the centroid to relative world coordinates converts to absolute world coordinates (and vice-versa subtracting it). Though as @zaharch says the places where it's needed differ from the code here as some of the new transformation matrices include this translation.\n\nSo with l5 kit 1.1 you see that:\n```\n> transform_point(np.array([0,0]), data['world_from_agent'])\narray([ 671.78094482, -2196.83374023])\n```\nis adding the centroid in and converting the rotation. So you then subtract the centroid from the result to get back to the initial relative world targets.\n\nTo go back the other way from old world relative target coordinates to agent relative coordinates you add the centroid and then transform. So:\n```\n>target_in_agent = np.array([0.1, 0.2])\n>target_in_world = transform_point(target_in_agent, data['world_from_agent']) - data['centroid']\n>print(\"Relative World coords: \", target_in_world)\n>print(\"Recovered: \", transform_point(target_in_world + data['centroid'], data['agent_from_world']))\nWorld coords: [-0.11271524 0.19311985]\nRecovered: [0.1 0.2]\n```\n\nNote the centroid is that same in each case as the change simply involves how targets are rotated around this centroid. Previously they were rotated with respect to the Autonomous Vehicle (/ego), now they are rotated with respect to the actual agent vehicle you're predicting (which may or may not be the ego). In both cases they're relative to centroid.\n\n(I think, obviously all a little complex and certainly encourage you to check my work)", "votes": 3 } ] }, { "id": 1031817, "postDate": "2020-09-29T17:44:04.043Z", "content": "Thank you for sharing, It is very helpful 👍
", "rawMarkdown": "Thank you for sharing, It is very helpful 👍" }, { "id": 1031758, "postDate": "2020-09-29T17:09:28.250Z", "content": "Thank you for sharing your findings with the community! It helps the people who are new to the competition! :)
", "rawMarkdown": "Thank you for sharing your findings with the community! It helps the people who are new to the competition! :)" }, { "id": 1031163, "postDate": "2020-09-29T09:08:12.123Z", "content": "good job ! good
", "rawMarkdown": "good job ! good" }, { "id": 1031127, "postDate": "2020-09-29T09:05:10.613Z", "content": "Cool :)))))))
", "rawMarkdown": "Cool :)))))))" }, { "id": 1030450, "postDate": "2020-09-28T17:16:27.573Z", "content": "Great job I am here to win
", "rawMarkdown": "Great job I am here to win" }, { "id": 1029934, "postDate": "2020-09-28T09:24:30.040Z", "content": "really helpful
", "rawMarkdown": "really helpful" }, { "id": 1029493, "postDate": "2020-09-27T19:26:48.257Z", "content": "Are you using a custom loss function?
", "rawMarkdown": "Are you using a custom loss function?", "replies": [ { "id": 1029545, "postDate": "2020-09-27T20:47:08.440Z", "content": "I am using a loss function from here, take a look.
", "rawMarkdown": "I am using a [loss function from here](https://www.kaggle.com/corochann/lyft-training-with-multi-mode-confidence), take a look. ", "votes": 2 }, { "id": 1030032, "postDate": "2020-09-28T11:15:47.660Z", "content": "Thank you, I was wondering first why you are passing 4 arguments to the loss function :D
", "rawMarkdown": "Thank you, I was wondering first why you are passing 4 arguments to the loss function :D", "votes": 1 }, { "id": 1030642, "postDate": "2020-09-28T19:39:50.927Z", "content": "@zaharch May I ask, why do you calculate the loss first and afterwards you transform the pred?
\nloss = criterion(targets, pred, confidences, target_availabilities)\nloss = torch.mean(loss)\n\n if cfg['train_params']['image_coords']:\n matrix_inv = torch.inverse(matrix)\n pred = pred + bias[:,None,:,:]\n pred = torch.cat([pred,torch.ones((bs,3,tl,1)).to(device)], dim=3)\n pred = torch.stack([torch.matmul(matrix_inv.to(torch.float), pred[:,i].transpose(1,2)) \n for i in range(3)], dim=1)\n pred = pred.transpose(2,3)[:,:,:,:2]\n pred = pred - centroid[:,None,:,:]\nWhat do you do with the returned pred, do you use it for validation?
", "rawMarkdown": "@zaharch May I ask, why do you calculate the loss first and afterwards you transform the pred?\n\n```\nloss = criterion(targets, pred, confidences, target_availabilities)\nloss = torch.mean(loss)\n\n if cfg['train_params']['image_coords']:\n matrix_inv = torch.inverse(matrix)\n pred = pred + bias[:,None,:,:]\n pred = torch.cat([pred,torch.ones((bs,3,tl,1)).to(device)], dim=3)\n pred = torch.stack([torch.matmul(matrix_inv.to(torch.float), pred[:,i].transpose(1,2)) \n for i in range(3)], dim=1)\n pred = pred.transpose(2,3)[:,:,:,:2]\n pred = pred - centroid[:,None,:,:]\n```\n\nWhat do you do with the returned pred, do you use it for validation?", "votes": 1 }, { "id": 1030653, "postDate": "2020-09-28T20:05:39.573Z", "content": "Hi @aliabdin1 , yes you are right, specifically I am using them for independent loss calculation and also calculating other metrics, like rmse. Want to do it in the original coordinates. Afterwards this transformation is actually needed for the test, as we submit in original coordinates.
", "rawMarkdown": "Hi @aliabdin1 , yes you are right, specifically I am using them for independent loss calculation and also calculating other metrics, like rmse. Want to do it in the original coordinates. Afterwards this transformation is actually needed for the test, as we submit in original coordinates.", "votes": 1 }, { "id": 1030657, "postDate": "2020-09-28T20:11:16.337Z", "content": "Thank you :)
", "rawMarkdown": "Thank you :)" } ] }, { "id": 1028755, "postDate": "2020-09-27T07:07:17.407Z", "content": "sure, i hope it will work
", "rawMarkdown": "sure, i hope it will work" }, { "id": 1028499, "postDate": "2020-09-26T22:57:25.237Z", "content": "It might be worth trying to anonymize the vehicles, Maybe draw them all with the same extent?
\nProbably be faster to try on inference, it should still show how much the vehicle size contributes to accuracy
", "rawMarkdown": "It might be worth trying to anonymize the vehicles, Maybe draw them all with the same extent?\n\nProbably be faster to try on inference, it should still show how much the vehicle size contributes to accuracy" }, { "id": 1028438, "postDate": "2020-09-26T21:04:58.093Z", "content": "Big thank you for sharing this huge and relevant information!
", "rawMarkdown": "Big thank you for sharing this huge and relevant information!" }, { "id": 1027963, "postDate": "2020-09-26T13:42:47.897Z", "content": "Good job, continue like this ! BRAVO
", "rawMarkdown": "Good job, continue like this ! BRAVO " }, { "id": 1026606, "postDate": "2020-09-25T12:46:48.603Z", "content": "I am wondering if drawing the images always with x axis pointing in the direction of travel will put too much bias on the predictions.
", "rawMarkdown": "I am wondering if drawing the images always with x axis pointing in the direction of travel will put too much bias on the predictions." }, { "id": 1032440, "postDate": "2020-09-30T07:50:47.030Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1026099, "postDate": "2020-09-25T04:42:40.040Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1039704, "postDate": "2020-10-06T18:26:12.170Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" }, { "id": 1035802, "postDate": "2020-10-03T05:35:21.447Z", "content": "thanks for sharing this information
", "rawMarkdown": "thanks for sharing this information" }, { "id": 1030352, "postDate": "2020-09-28T15:37:31.927Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing" }, { "id": 1029508, "postDate": "2020-09-27T19:39:32.697Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 1028482, "postDate": "2020-09-26T22:21:22.550Z", "content": "Great job guys, thanks for sharing!
", "rawMarkdown": "Great job guys, thanks for sharing!" } ], "comments": [ { "id": 1026361, "author_name": "Luca Bergamini", "author_url": "", "post_date": "2020-09-25T08:49:31.527000", "content": "Hey Everyone,
\n\n\nWhat do you think? I have a lingering feeling that I am missing something obvious!
\n
On the contrary, you're spot on :) This is something that slipped through our tests and that we only recently became aware of.
\nUsing pixel coordinates is absolutely legit here, but we're also working on a long term fix in L5Kit we hope we can ship next week. Once that is thoroughly checked we will sync with Kaggle to update the default package.
\nI'll start a post with more info about this issue and how we plan to fix it to increase visibility and awareness.
", "votes": 10, "replies": [ { "id": 1030060, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-09-28T11:39:01.493000", "content": "any updates?
", "votes": 0, "replies": [] }, { "id": 1030368, "author_name": "Luca Bergamini", "author_url": "", "post_date": "2020-09-28T15:58:08.520000", "content": "almost ready to merge https://github.com/lyft/l5kit/pull/150 :) After that we will cut a new release and update Kaggle, doc and baseline
", "votes": 2, "replies": [] }, { "id": 1030406, "author_name": "Thomas Brandon", "author_url": "", "post_date": "2020-09-28T16:41:34.090000", "content": "I compared that PR to the fix posted here and looks like you're keeping the target positions in metres whereas above code converts to pixels, otherwise they match (barring slight inaccuracy likely due to floating point). That's the intended final result correct? (No issue, I think metres is better, just checking)
\nI gather also that you'll be updating the Kaggle ground truths so predictions will be in the new coordinates. Is that correct?
", "votes": 2, "replies": [] }, { "id": 1030418, "author_name": "nosound", "author_url": "", "post_date": "2020-09-28T16:52:25.093000", "content": "I think pixels are better, - in pixels they can be directly connected in creative ways to the images.
", "votes": 1, "replies": [] }, { "id": 1030428, "author_name": "Luca Bergamini", "author_url": "", "post_date": "2020-09-28T16:58:47.383000", "content": "\n\nThat's the intended final result correct?
\n
yes, we're keeping metres (basically the only difference is a 2D rotation) :)
\n\n\nI gather also that you'll be updating the Kaggle ground truths so predictions will be in the new coordinates. Is that correct?
\n
We're not, this means you will have to convert them back into offsets in world coordinates (but we'll update the baseline notebook to show how to do that and we're adding a lot of matrices to make it easier). Updating the ground truth would have required a lot of additional testing, this was the best compromise :(
", "votes": 1, "replies": [] }, { "id": 1030433, "author_name": "Thomas Brandon", "author_url": "", "post_date": "2020-09-28T17:02:57.873000", "content": "\n\nI think pixels are better
\n
How so? Either way you can just multiply/divide by pixel size right? I think if you want that you probably want to do it within a single model and then convert back to metres at the end. Otherwise if doing something like combining outputs from multiple models with different pixel sizes you'd have to be tracking the pixel sizes of all models.
\nWhat if you aren't even using a pixel based model, why convert to some arbitrary pizel size?
\n\nWe're not, this means you will have to convert them back into offsets in world coordinates (but we'll update the baseline notebook to show how to do that and we're adding a lot of matrices to make it easier). Updating the ground truth would have required a lot of additional testing, this was the best compromise :(
\n
Ah OK, I misunderstood the changes to the GT.csv code as meaning they were being updated to new coords when I guess they're converting new coords back to the old ones. Presumably they will match expected outputs on Kaggle.
\nI understand the compromise, obviously tricky either way.
\n\nI guess they're converting new coords back to the old ones. Presumably they will match expected outputs on Kaggle
\n
That's is exactly what's happening there :) I've checked and they match the old ones (also I've submitted a new baseline trained using the latest master and results are much better so I'm positive we didn't screw everything up completely)
", "votes": 2, "replies": [] }, { "id": 1030458, "author_name": "Thomas Brandon", "author_url": "", "post_date": "2020-09-28T17:20:15.487000", "content": "Probably belongs elsewhere but in relation to the GCP credit prize for those beating the baseline does that mean they need to beat new baseline? Think the first deadline for that was tomorrow so not a lot of time to update models.
", "votes": 2, "replies": [] }, { "id": 1030480, "author_name": "Huan Vo", "author_url": "", "post_date": "2020-09-28T17:29:02.487000", "content": "Where is the new baseline result? I could not find it. Or if someone can just report the new score here?
", "votes": 0, "replies": [] }, { "id": 1030498, "author_name": "Thomas Brandon", "author_url": "", "post_date": "2020-09-28T17:35:05.717000", "content": "82.246
\nhttps://www.kaggle.com/lucabergamini/lyft-baseline-09-02
\n\nHow so? Either way you can just multiply/divide by pixel size right? I think if you want that you probably want to do it within a single model and then convert back to metres at the end. Otherwise if doing something like combining outputs from multiple models with different pixel sizes you'd have to be tracking the pixel sizes of all models.
\n
\n What if you aren't even using a pixel based model, why convert to some arbitrary pizel size?
@thomasbrandon , you have good points, hard to argue with what you say. In defense of pixels, - we get data object in which I would naturally expect everything to be in the same coordinate system. There is the image object, with its own coordinate system, so everything needs to match. My second argument is that if everything is in pixels then we can easily draw on the image without any additional transformations, rasterize those pixels into something etc.
Did you calculate bias = torch.tensor([56.25, 112.5]) from the ego_center config? So the input size is 225x225px?
That was my question as well. I assume this will not be the correct alignment at other resolutions.
", "votes": 1, "replies": [] }, { "id": 1025694, "author_name": "Peter", "author_url": "", "post_date": "2020-09-24T17:58:03.900000", "content": "Maybe this..
\nrs = cfg[\"raster_params\"][\"raster_size\"]\nec = cfg[\"raster_params\"][\"ego_center\"]\n\nbias = torch.tensor([rs[0] * ec[0], rs[1] * ec[1]])[None, None, :].to(device)\nExactly right, thank you! The point of this is of course to have non-moving target at zero, as it was before.
", "votes": 2, "replies": [] } ] }, { "id": 1029906, "author_name": "fergusoci", "author_url": "", "post_date": "2020-09-28T08:37:46.400000", "content": "Thank you for sharing this. We were doing it completely wrong, and most people would have been oblivious to that fact had you not so generously highlighted it. The epitome of community - kudos.
", "votes": 6, "replies": [] }, { "id": 1027976, "author_name": "tattaka", "author_url": "", "post_date": "2020-09-26T13:47:09.237000", "content": "Thank you for sharing.
\nWith this approach, in my case, the model(45.0) trained on all of the datasets was improved to the model(37.7) trained on 1/4 of the dataset.
\nHowever, there is a gap between validation metric (13.1) and public LB.
Your validation metric is too small, this can be because you divide the errors by 2 after the predictions are transformed. Basically check that the errors are divided by 2 only when needed.
", "votes": 1, "replies": [] }, { "id": 1028488, "author_name": "tattaka", "author_url": "", "post_date": "2020-09-26T22:29:26.957000", "content": "Sorry. When I checked it, I used the square error, so I divided it by 4.
", "votes": 0, "replies": [] } ] }, { "id": 1026441, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-09-25T10:22:50.063000", "content": "Good approach @zaharch . Congrats !
\nCan I ask you why do you choose to divide the error by 4 ?
In case one uses default configuration there is a scaling factor 'pixel_size': [0.5, 0.5]. So there are two pixels (new coordinates) per meter (old coordinates). To get the error back into the old scale I need to divide it by 2, or by 4 in case of squared error.
Depending on your model it may not be necessary to scale, you will just have 4x bigger loss. But I train a multi-scenario output, and there is a subtle inter-play between the errors and the confidences. Therefore I want to keep it at the old scale, which is the correct one for the loss.
", "votes": 5, "replies": [] } ] }, { "id": 1025888, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-09-24T21:06:16.230000", "content": "We really did it all wrong! Thanks for sharing and pointing it out.
", "votes": 3, "replies": [] }, { "id": 1031260, "author_name": "M. Esad Tutar", "author_url": "", "post_date": "2020-09-29T10:54:47.223000", "content": "I'm really new to this community and this helped me a lot. Thanks!
", "votes": 1, "replies": [] }, { "id": 1028934, "author_name": "Shashank Madan", "author_url": "", "post_date": "2020-09-27T10:36:53.357000", "content": "Hi, Great insight! Can i know how you obtained those graphs?
", "votes": 1, "replies": [ { "id": 1028945, "author_name": "nosound", "author_url": "", "post_date": "2020-09-27T10:44:40.177000", "content": "Sure, the first graph is this code:
\nfig = plt.figure(figsize = (12,12))\nfor k in range(500):\n data = valid_dataset[4001+1000*k]\n tgt = data[\"target_positions\"]\n plt.plot(tgt[:,0],tgt[:,1])\nthe second is similar but with the transformation, depends on how you shape your code.
", "votes": 5, "replies": [] }, { "id": 1028966, "author_name": "Shashank Madan", "author_url": "", "post_date": "2020-09-27T11:04:52.797000", "content": "Thanks a Lot! By the looks of it… the valid_dataset suggests you are using an evaluation dataset… why? just curious…
", "votes": 0, "replies": [] }, { "id": 1029308, "author_name": "nosound", "author_url": "", "post_date": "2020-09-27T16:34:59.013000", "content": "No reason, it doesn't really matter because it is just targets from the data. I don't think there is any meaningful difference between the train and the validation.
", "votes": 1, "replies": [] }, { "id": 1029763, "author_name": "Shashank Madan", "author_url": "", "post_date": "2020-09-28T05:48:28.323000", "content": "Understood! thanks again😀
", "votes": 1, "replies": [] } ] }, { "id": 1027575, "author_name": "Ram Ramrakhya", "author_url": "", "post_date": "2020-09-26T08:43:53.657000", "content": "@zaharch does this mean while inference we have to transform the predicted trajectories using the code block you shared
\n if cfg['train_params']['image_coords']:\n matrix_inv = torch.inverse(matrix)\n pred = pred + bias[:,None,:,:]\n pred = torch.cat([pred,torch.ones((bs,3,tl,1)).to(device)], dim=3)\n pred = torch.stack([torch.matmul(matrix_inv.to(torch.float), pred[:,i].transpose(1,2)) \n for i in range(3)], dim=1)\n pred = pred.transpose(2,3)[:,:,:,:2]\n pred = pred - centroid[:,None,:,:]\nTrying to understand if I am interpreting the targets correctly
", "votes": 1, "replies": [ { "id": 1027577, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-09-26T08:44:51.447000", "content": "Yes, in inference you will have to use the same code block.
", "votes": 2, "replies": [] }, { "id": 1027613, "author_name": "nosound", "author_url": "", "post_date": "2020-09-26T08:56:53.187000", "content": "Yes, that is right, same for inference.
", "votes": 1, "replies": [] }, { "id": 1027628, "author_name": "Ram Ramrakhya", "author_url": "", "post_date": "2020-09-26T09:06:41.810000", "content": "Interesting, I trained model for 6 epochs 10k steps each my train loss was around ~60.x but the inference kernel with transformation gave really bad result. Anything obvious I might be missing?
\nNote that I am using raster rize 224x224 so I changed the bias value.
\nInference code for reference
\n inputs = data[\"image\"].to(device)\n #target_availabilities = data[\"target_availabilities\"].unsqueeze(-1).to(device)\n target_availabilities = data[\"target_availabilities\"].to(device)\n targets = data[\"target_positions\"].to(device)\n matrix = data[\"world_to_image\"].to(device)\n centroid = data[\"centroid\"].to(device)[:,None,:].to(torch.float)\n\n bs,tl,_ = targets.shape\n rs = cfg[\"raster_params\"][\"raster_size\"]\n ec = cfg[\"raster_params\"][\"ego_center\"]\n bias = torch.tensor([rs[0] * ec[0], rs[1] * ec[1]])[None, None, :].to(device)\n\n\n #outputs = model(inputs).reshape(targets.shape)\n pred, confidences = model(inputs)\n\n if cfg['test_params']['image_coords']:\n matrix_inv = torch.inverse(matrix)\n pred = pred + bias[:,None,:,:]\n pred = torch.cat([pred,torch.ones((bs,3,tl,1)).to(device)], dim=3)\n pred = torch.stack([torch.matmul(matrix_inv.to(torch.float), pred[:,i].transpose(1,2)) \n for i in range(3)], dim=1)\n pred = pred.transpose(2,3)[:,:,:,:2]\n pred = pred - centroid[:,None,:,:]\nHave you changed the bias to 56,112? How do you judge that the result is bad? Do prediction start from close to zero? I guess it is some kind of a technical error.
\nDo you do softmax for the confidences and pred shape transformations inside the model?
", "votes": 0, "replies": [] }, { "id": 1027654, "author_name": "Ram Ramrakhya", "author_url": "", "post_date": "2020-09-26T09:31:26.883000", "content": "Have you changed the bias to 56,112?
\nYes,
\nDo you do softmax for the confidences and pred shape transformations inside the model?
\nYes
\nDo prediction start from close to zero?
\nYes
\nHow do you judge that the result is bad?
\nI am comparing model performance against my old model without transformation after n epochs. Model without transformations seemed to have converged faster in my case. I guess I'll dig deeper into this by comparing my results
Awesome point. Thank you for sharing this
", "votes": 1, "replies": [] }, { "id": 1064052, "author_name": "hengck23", "author_url": "", "post_date": "2020-10-29T16:30:18.540000", "content": "important!!!!
\nSo I tried to modify the forward function above for the new l5kit package (version 1.1.0) following the notebook Lyft-baseline-09-02
\ndef forward(data, model, device, criterion = pytorch_neg_multi_log_likelihood_batch):\n inputs = data[\"image\"].to(device)\n target_availabilities = data[\"target_availabilities\"].to(device)\n targets = data[\"target_positions\"].to(device)\n\n # Forward pass\n preds, confidences = model(inputs) \n\n\n loss = criterion(targets, preds, confidences, target_availabilities)\n\n\n world_from_agents = data[\"world_from_agent\"].numpy()\n centroids = data[\"centroid\"].numpy()\n\n #convert into world coordinates and compute offsets\n for idx in range(len(preds)):\n preds[idx] = transform_points(preds[idx], world_from_agents[idx]) - centroids[idx]\n\n return loss, preds, confidences\nSo as I understand the target in the train set is fixed, however the test set is not so we still need to convert it? However when I ran the above code I encountered the following error:
\n---------------------------------------------------------------------------\nAssertionError Traceback (most recent call last)\n<ipython-input-24-9205635f7dc5> in <module>\n 12 for data in progress_bar:\n 13 inputs = data['image'].to(device)\n---> 14 _, preds, confidences = forward(data, model, device)\n 15 \n 16 # # convert agent coordinates into world offsets\n\n<ipython-input-22-276920502171> in forward(data, model, device, criterion)\n 40 #convert into world coordinates and compute offsets\n 41 for idx in range(len(preds)):\n---> 42 preds[idx] = transform_points(preds[idx], world_from_agents[idx]) - centroids[idx]\n 43 \n 44 return loss, preds, confidences\n\n/kaggle/usr/lib/lyft_l5kit_unofficial_fix/l5kit/geometry/transform.py in transform_points(points, transf_matrix)\n 87 np.ndarray: array of shape (N,2) for 2D input points, or (N,3) points for 3D input points\n 88 \"\"\"\n---> 89 assert len(points.shape) == len(transf_matrix.shape) == 2\n 90 assert transf_matrix.shape[0] == transf_matrix.shape[1]\n 91 \n\nAssertionError: \nDoes anyone know what is going on? Also I assume that with the fix I do not have to divide the error by 4 anymore? Any insight is appreciated.
\nThanks a lot for your help!
", "votes": 2, "replies": [ { "id": 1037423, "author_name": "Frank Pan", "author_url": "", "post_date": "2020-10-05T02:35:19.387000", "content": "The official example is only for single-mode prediction. For multi-mode I am doing something like this (when making predictions):
\noutputs = outputs.cpu().numpy()\nworld_from_agents = data[\"world_from_agent\"].numpy()\ncentroids = data[\"centroid\"].numpy()\ncoords_offset = []\n\nfor agent_coords, world_from_agent, centroid in zip(outputs, world_from_agents, centroids):\n for i in range(3):\n agent_coords[i, :, :] = transform_points(agent_coords[i, :, :], world_from_agent) - centroid[:2]\n coords_offset.append(agent_coords)\nYou can use some matrix manipulation to avoid the loops so it could be a liitle bit faster but this seems to work fine for me.
", "votes": 3, "replies": [] }, { "id": 1037644, "author_name": "Luca Bergamini", "author_url": "", "post_date": "2020-10-05T07:40:22.533000", "content": "We're considering to support batch matrix multiplication for that function in the future release. It's not anything difficult to do (numpy can handle it without any troubles ), we just didn't have enough time to implement and test it :(
", "votes": 3, "replies": [ { "id": 1038958, "author_name": "Jiayu", "author_url": "", "post_date": "2020-10-06T07:42:41.653000", "content": "\n\nWe're considering to support batch matrix multiplication for that function in the future release. It's not anything difficult to do (numpy can handle it without any troubles ), we just didn't have enough time to implement and test it :(
\n
worthy of trying https://github.com/lyft/l5kit/pull/166
", "votes": 0, "replies": [] } ] }, { "id": 1037713, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-10-05T09:01:43.140000", "content": "@frankpanxj Is future_coords_offsets_pd.append() in your code coords_offset.append(agent_coords) or do you stack it at the end?
EDIT: seems to need future_coords_offsets_pd.append(np.stack(coords_offset))
@huanvo I have tried the same yesterday, it is like mentioned because of single model. They also use different shapes e.g.output = model(inputs).reshape(targets.shape)
I tried to change everything accordingly but I ended up using the mentioned solution from @zaharch
\nI will try a couple of things out and report back.
UPDATE: @frankpanxj Solution works best for me.
", "votes": 1, "replies": [] }, { "id": 1037772, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-10-05T10:19:25.590000", "content": "Thanks for the input @frankpanxj . Is coords_offset the same shape as the original agents_coords ? It can replace it when we pass it to the loss function to get the metric of the training ?
", "votes": 1, "replies": [] }, { "id": 1037822, "author_name": "Frank Pan", "author_url": "", "post_date": "2020-10-05T11:10:27.450000", "content": "@aliabdin1 You are right. You need to stack the reults.
", "votes": 0, "replies": [] }, { "id": 1037828, "author_name": "Frank Pan", "author_url": "", "post_date": "2020-10-05T11:14:48.693000", "content": "@vladvdv the shape of outputs is (batch size)x(modes)x(time)x(2D coords), which is needed for the loss function by @corochann at https://www.kaggle.com/corochann/lyft-training-with-multi-mode-confidence.
\ncoords_offset here is just a list of the preds for every sample, after stacking it you will get a np array with the same shape as outputs. (I changed some the variable names above to make this clearer.)
Thanks @frankpanxj this seems to resolve my error. However my prediction error got much worse in the new l5kit environment (old env 18.x vs new env 4xxx).
\nMaybe I misunderstood something here. I was using the fix by @zaharch before in the old environment and it seemed to work fine.
@huanvo have similar issues. Did you find the problem ?
\nlater edit: I was doing the transform at training time also. From what I understand now, the transformation should be done only on inference !
The fix posted here is different to the fix in l5kit v1.1.0. In the fix here it not only corrects for the rotation error but also converts from metres to pixels. The fix in v1.1 only corrects the rotation, leaving things in metres. So models trained with the fix here aren't compatible with the fix in v1.1.
\nTo use a model trained with the fix here in v1.1 you'd need to convert the targets from metres to pixels and then transform the predictions from pixels to metres before applying the rotation fix from the l5kit docs.
I used the above @frankpanxj 's code to inference a model trained with L5kit ver1.1.0 but my submission file created only up to coordinates 0-29 i.e. coord_x29, coord_y29.
\nDoes anybody experience the same thing?
\nI could not see what I am doing wrong in my code right now.
", "votes": 0, "replies": [] }, { "id": 1038634, "author_name": "Frank Pan", "author_url": "", "post_date": "2020-10-05T23:20:19.687000", "content": "Hi @huanvo , I tried zaharch's approach and got similar results as using v1.1. However, my score was never that good (my validation loss was always much higher than my training loss, did you experience the same?) so there certainly is something I am missing.
\n@sheriytm very interesting results. I did not have the same problem. There are coords beyond 29 in my results.
", "votes": 1, "replies": [] }, { "id": 1040128, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2020-10-07T01:48:21.617000", "content": "Thanks @frankpanxj. I just found the bug that caused cuts out the rest of the coordinates during prediction.
", "votes": 1, "replies": [] }, { "id": 1041478, "author_name": "ryches", "author_url": "", "post_date": "2020-10-07T19:04:16.717000", "content": "I'm a bit confused. My results seem to be completely non-sensical with the new version and the example code provided. The issue seems to be stemming from subtracting the centroid. My centroid is values in the thousands typically and shifting my predictions way too much. Anyone having similar experience or an explanation of what I might be doing wrong?
", "votes": 0, "replies": [] }, { "id": 1041480, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-10-07T19:05:52.430000", "content": "Did you remove the here mentioned training snippet? Its not needed anymore.
", "votes": 0, "replies": [] }, { "id": 1041493, "author_name": "ryches", "author_url": "", "post_date": "2020-10-07T19:17:16.053000", "content": "Yes I am training on the raw outputs from the dataloader on the newest version of l5kit. No transforms anymore. Training loss looks fine but validation is messed up. in the hundreds of thousands using roughly the code that was shared here. My predictions should not be in the thousands, right?
", "votes": 0, "replies": [] }, { "id": 1041494, "author_name": "nosound", "author_url": "", "post_date": "2020-10-07T19:18:10.640000", "content": "Hi @ryches , the new version works fine for me, I haven't noticed anything interesting changed after the switch. Please share the code.
", "votes": 0, "replies": [] }, { "id": 1041514, "author_name": "ryches", "author_url": "", "post_date": "2020-10-07T19:34:51.987000", "content": "My outputs from my model are roughly what they were prior to the l5kit upgrade. Previously I was doing the transform and then inverse transform and it was working fine.
\nNow with my model trained on the new l5kit targets without any transformations its max output on x, y coordinates is ~77 which seems normal, but centroid is showing up as 785, -2024. Validation predictions used to look like
but now they look like this and the big difference seems to be these huge centroid outputs
Is the l5kit version you are using from pip or directly from github?
", "votes": 0, "replies": [] }, { "id": 1041527, "author_name": "ryches", "author_url": "", "post_date": "2020-10-07T19:42:52.290000", "content": "The code I have been poking at to try to understand. If I dont subtract the huge centroids then I still get questionable results
\ntorch.set_grad_enabled(False)\nwith torch.no_grad():\n model.eval()\n # store information for evaluation\n future_coords_offsets_pd = []\n timestamps = []\n agent_ids = []\n confidences = []\n progress_bar = tqdm(eval_dataloader)\n for data in progress_bar:\n inputs = data\n targets = data[\"target_positions\"]\n timestamps.append(data[\"timestamp\"].cpu().numpy())\n agent_ids.append(data[\"track_id\"].cpu().numpy())\n outputs, confs = model(inputs)\n# outputs = inverse_tranform_predictions(outputs, data[\"world_to_image\"], data[\"centroid\"])\n# future_coords_offsets_pd.append(outputs.cpu().numpy())\n for agent_coords, world_from_agent, centroid in zip(outputs.cpu().numpy(), \n data[\"world_to_image\"].cpu().numpy(),\n data[\"centroid\"].cpu().numpy()):\n for mode in range(3):\n agent_coords[mode, :, :] = transform_points(agent_coords[mode, :, :], world_from_agent) - centroid[:2]\n future_coords_offsets_pd.append(agent_coords)\n confidences.append(confs.cpu().numpy())\n pred_path = f\"{gettempdir()}/validation_preds.csv\"\n write_pred_csv(pred_path,\n timestamps=np.concatenate(timestamps),\n track_ids=np.concatenate(agent_ids),\n coords=np.stack(future_coords_offsets_pd),\n confs = np.concatenate(confidences)\n )\n del confidences, future_coords_offsets_pd\n metrics = compute_metrics_csv(eval_gt_path, pred_path, [neg_multi_log_likelihood, time_displace])\n for metric_name, metric_mean in metrics.items():\n print(metric_name, metric_mean)\nthis is how you need to transform for multimode:
\n # convert agent coordinates into world offsets\n pred = pred.cpu().numpy()\n world_from_agents = data[\"world_from_agent\"].numpy()\n centroids = data[\"centroid\"].numpy()\n coords_offset = []\n\n # convert into world coordinates and compute offsets\n for idx in range(len(pred)):\n for mode in range(3):\n pred[idx, mode, :, :] = transform_points(pred[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]\n\n confidences_list.append(confidences.cpu().numpy().copy())\n pred_coords_list.append(pred.copy())\n timestamps.append(data[\"timestamp\"].numpy().copy())\n agent_ids.append(data[\"track_id\"].numpy().copy())\nI see that you used a batched approach @ryches , that could be the problem. I am not sure if that works for multi-mode. I am actually pretty sure that it doesn't work as the code for transform_points is expecting points in (Nx2) or (Nx3).
", "votes": 7, "replies": [] }, { "id": 1041539, "author_name": "ryches", "author_url": "", "post_date": "2020-10-07T19:50:27.217000", "content": "Thanks, I will look at that and try it out
", "votes": 0, "replies": [] }, { "id": 1041548, "author_name": "nosound", "author_url": "", "post_date": "2020-10-07T20:03:06.530000", "content": "Maybe the problem is that you are using data[\"world_to_image\"], instead of world_from_agent
Looks like that works. Thank you so much. Devil is in the details
", "votes": 0, "replies": [] }, { "id": 1041559, "author_name": "ryches", "author_url": "", "post_date": "2020-10-07T20:08:28.657000", "content": "Yeah on second review I think the key was using the right transformation matrix. I was using the same method of zipping across the three inputs for the batches as shown in the example notebook and then iterating 3 times for the various modes as others have mentioned in this thread. I think my dimensions and operations were correct but I was using the wrong matrix.
", "votes": 2, "replies": [] }, { "id": 1041564, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-10-07T20:12:05.090000", "content": "Great to hear, i'll switch to batched transformation, too if that works ;)
", "votes": 0, "replies": [] }, { "id": 1041590, "author_name": "ryches", "author_url": "", "post_date": "2020-10-07T20:36:18.320000", "content": "Probably not worth switching. I did not see any appreciable difference in speed
", "votes": 0, "replies": [] } ] }, { "id": 1030216, "author_name": "The Brown Iceman", "author_url": "", "post_date": "2020-09-28T13:34:14.877000", "content": "First @corochann helped the heard think & helped us and now you ( @zaharch ) are doing the same. I like to thank you both.
", "votes": 2, "replies": [ { "id": 1030748, "author_name": "corochann", "author_url": "", "post_date": "2020-09-28T23:22:36.097000", "content": "Thanks for feedback & mention :)
\nI couldn't notice this before @zaharch pointed out, even I go through the EDA in several notebooks…
hey @nosound, thanks for sharing this code and these insights. I'm still noob at ML and DS, but I've been \"studying\" some of the newest codes at current competitions and i feel I can get what you mean by directionless input's. don't know how to approach directly to that matter but If i get something I'll share it =) thanks for it.
", "votes": 2, "replies": [] }, { "id": 1028503, "author_name": "Dmytro Poplavskiy", "author_url": "", "post_date": "2020-09-26T23:18:52.250000", "content": "The fact the original model performed well without any knowledge about the agent orientation raises the question about how generalisable the models are, even with the orientation fix applied. Would be easy to fix by the position specific train/validation split.
\nWould be great if hosts allocated the test set in the different geographic position (and not available until the models freeze) to check how well solutions perform in more realistic situation of roads model have not seen during training, but I'd expect it's too much to ask at this stage.
", "votes": 2, "replies": [ { "id": 1028864, "author_name": "nosound", "author_url": "", "post_date": "2020-09-27T09:08:52.447000", "content": "On the other hand if the interesting zone is limited to what we have and AVs do not drive out of this zone anyway, then it is OK to overfit on it. But you have a good point.
\nIf direction is possible to learn then the models potentially can learn average speeds on those roads and common routes taken by cars, like rare vs common turns.
", "votes": 1, "replies": [] } ] }, { "id": 1027116, "author_name": "Tom Aindow", "author_url": "", "post_date": "2020-09-25T21:04:05.090000", "content": "Fantastic post, thanks for sharing! Will implement and report results :)
", "votes": 2, "replies": [ { "id": 1029481, "author_name": "Tom Aindow", "author_url": "", "post_date": "2020-09-27T19:16:51.673000", "content": "Can confirm this provided a great boost to models from 3x.x to 2x.x, and current score 18.x using this approach too. Thanks again to nosound for sharing. Super interested to better understand how we were doing so well before, very odd!
", "votes": 2, "replies": [] } ] }, { "id": 1026626, "author_name": "Huan Vo", "author_url": "", "post_date": "2020-09-25T13:07:50.927000", "content": "Oh wow thanks for sharing this! It took me forever to bring the error down, and it is not even close to 23. On the other hand, it is still surprising how the model can learn without any directions.
", "votes": 2, "replies": [ { "id": 1051298, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2020-10-16T11:09:36.493000", "content": "@huanvo I don't understand about what are you talking about
\nbut do have a question, please
\nRecently you have posted a high scoring kernel.
\nis the forward function which mentioned by nosound on the discussion topic in that kernel
Would it be possible for you to create a plot of the transformed paths like you did for the originals? I'd be interested to see what that looks like
", "votes": 2, "replies": [ { "id": 1025860, "author_name": "nosound", "author_url": "", "post_date": "2020-09-24T20:04:23.183000", "content": "Will do when I get home.
\nUpdate: added
", "votes": 4, "replies": [] } ] }, { "id": 1051279, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2020-10-16T10:35:24.620000", "content": "I have a few questions after a long time of discussion of different people around the world
\nQuestion
\nHi, @morizin , I am glad you join the discussion!
\nnot sure what to comment on your other points.
", "votes": 0, "replies": [] } ] }, { "id": 1039767, "author_name": "Rahul Kumar", "author_url": "", "post_date": "2020-10-06T19:21:11.993000", "content": "I'm really new to this community and this helped me a lot. Thanks!
", "votes": 0, "replies": [] }, { "id": 1037514, "author_name": "Srihari Radhakrishna", "author_url": "", "post_date": "2020-10-05T05:00:42.687000", "content": "Thanks for sharing! This was extremely useful.
\nWondering if the following achieves the same transformation:
\n\ntransform_points(data['target_positions'], data['raster_from_agent'])\n
Why is centroid added to target_positions? (The centroid is the initial position of the agent in world coordinates?)
", "votes": 0, "replies": [ { "id": 1034058, "author_name": "nosound", "author_url": "", "post_date": "2020-10-01T13:11:20.757000", "content": "Instead of understanding this I recommend to move to the new l5kit version, it is different there. Go to the github page and read the coordinate system description and see the example notebooks.
", "votes": 2, "replies": [] }, { "id": 1034119, "author_name": "Thomas Brandon", "author_url": "", "post_date": "2020-10-01T14:11:38.867000", "content": "Yeah, the process is different in the l5kit implementation but you do still need to add/subtract the centroid in some places. In both approaches this is because the targets are relative positions from the initial agent position while some of the transformation matrices need absolute positions. So adding the centroid to relative world coordinates converts to absolute world coordinates (and vice-versa subtracting it). Though as @zaharch says the places where it's needed differ from the code here as some of the new transformation matrices include this translation.
\nSo with l5 kit 1.1 you see that:
\n> transform_point(np.array([0,0]), data['world_from_agent'])\narray([ 671.78094482, -2196.83374023])\nis adding the centroid in and converting the rotation. So you then subtract the centroid from the result to get back to the initial relative world targets.
\nTo go back the other way from old world relative target coordinates to agent relative coordinates you add the centroid and then transform. So:
\n>target_in_agent = np.array([0.1, 0.2])\n>target_in_world = transform_point(target_in_agent, data['world_from_agent']) - data['centroid']\n>print(\"Relative World coords: \", target_in_world)\n>print(\"Recovered: \", transform_point(target_in_world + data['centroid'], data['agent_from_world']))\nWorld coords: [-0.11271524 0.19311985]\nRecovered: [0.1 0.2]\nNote the centroid is that same in each case as the change simply involves how targets are rotated around this centroid. Previously they were rotated with respect to the Autonomous Vehicle (/ego), now they are rotated with respect to the actual agent vehicle you're predicting (which may or may not be the ego). In both cases they're relative to centroid.
\n(I think, obviously all a little complex and certainly encourage you to check my work)
", "votes": 3, "replies": [] } ] }, { "id": 1031817, "author_name": "sandeep", "author_url": "", "post_date": "2020-09-29T17:44:04.043000", "content": "Thank you for sharing, It is very helpful 👍
", "votes": 0, "replies": [] }, { "id": 1031758, "author_name": "Tarun Paparaju", "author_url": "", "post_date": "2020-09-29T17:09:28.250000", "content": "Thank you for sharing your findings with the community! It helps the people who are new to the competition! :)
", "votes": 0, "replies": [] }, { "id": 1031163, "author_name": "Naim Mhedhbi", "author_url": "", "post_date": "2020-09-29T09:08:12.123000", "content": "good job ! good
", "votes": 0, "replies": [] }, { "id": 1031127, "author_name": "Saeed Zou", "author_url": "", "post_date": "2020-09-29T09:05:10.613000", "content": "Cool :)))))))
", "votes": 0, "replies": [] }, { "id": 1030450, "author_name": "Akhil Bhalerao", "author_url": "", "post_date": "2020-09-28T17:16:27.573000", "content": "Great job I am here to win
", "votes": 0, "replies": [] }, { "id": 1029934, "author_name": "Vinayak M S", "author_url": "", "post_date": "2020-09-28T09:24:30.040000", "content": "really helpful
", "votes": 0, "replies": [] }, { "id": 1029493, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-09-27T19:26:48.257000", "content": "Are you using a custom loss function?
", "votes": 0, "replies": [ { "id": 1029545, "author_name": "nosound", "author_url": "", "post_date": "2020-09-27T20:47:08.440000", "content": "I am using a loss function from here, take a look.
", "votes": 2, "replies": [] }, { "id": 1030032, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-09-28T11:15:47.660000", "content": "Thank you, I was wondering first why you are passing 4 arguments to the loss function :D
", "votes": 1, "replies": [] }, { "id": 1030642, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-09-28T19:39:50.927000", "content": "@zaharch May I ask, why do you calculate the loss first and afterwards you transform the pred?
\nloss = criterion(targets, pred, confidences, target_availabilities)\nloss = torch.mean(loss)\n\n if cfg['train_params']['image_coords']:\n matrix_inv = torch.inverse(matrix)\n pred = pred + bias[:,None,:,:]\n pred = torch.cat([pred,torch.ones((bs,3,tl,1)).to(device)], dim=3)\n pred = torch.stack([torch.matmul(matrix_inv.to(torch.float), pred[:,i].transpose(1,2)) \n for i in range(3)], dim=1)\n pred = pred.transpose(2,3)[:,:,:,:2]\n pred = pred - centroid[:,None,:,:]\nWhat do you do with the returned pred, do you use it for validation?
", "votes": 1, "replies": [] }, { "id": 1030653, "author_name": "nosound", "author_url": "", "post_date": "2020-09-28T20:05:39.573000", "content": "Hi @aliabdin1 , yes you are right, specifically I am using them for independent loss calculation and also calculating other metrics, like rmse. Want to do it in the original coordinates. Afterwards this transformation is actually needed for the test, as we submit in original coordinates.
", "votes": 1, "replies": [] }, { "id": 1030657, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-09-28T20:11:16.337000", "content": "Thank you :)
", "votes": 0, "replies": [] } ] }, { "id": 1028755, "author_name": "", "author_url": "", "post_date": "2020-09-27T07:07:17.407000", "content": "", "votes": 0, "replies": [] }, { "id": 1028499, "author_name": "", "author_url": "", "post_date": "2020-09-26T22:57:25.237000", "content": "", "votes": 0, "replies": [] }, { "id": 1028438, "author_name": "", "author_url": "", "post_date": "2020-09-26T21:04:58.093000", "content": "", "votes": 0, "replies": [] }, { "id": 1027963, "author_name": "", "author_url": "", "post_date": "2020-09-26T13:42:47.897000", "content": "", "votes": 0, "replies": [] }, { "id": 1026606, "author_name": "", "author_url": "", "post_date": "2020-09-25T12:46:48.603000", "content": "", "votes": 0, "replies": [] }, { "id": 1032440, "author_name": "", "author_url": "", "post_date": "2020-09-30T07:50:47.030000", "content": "", "votes": 0, "replies": [] }, { "id": 1026099, "author_name": "", "author_url": "", "post_date": "2020-09-25T04:42:40.040000", "content": "", "votes": 0, "replies": [] }, { "id": 1039704, "author_name": "", "author_url": "", "post_date": "2020-10-06T18:26:12.170000", "content": "", "votes": 0, "replies": [] }, { "id": 1035802, "author_name": "", "author_url": "", "post_date": "2020-10-03T05:35:21.447000", "content": "", "votes": 0, "replies": [] }, { "id": 1030352, "author_name": "", "author_url": "", "post_date": "2020-09-28T15:37:31.927000", "content": "", "votes": 0, "replies": [] }, { "id": 1029508, "author_name": "", "author_url": "", "post_date": "2020-09-27T19:39:32.697000", "content": "", "votes": 0, "replies": [] }, { "id": 1028482, "author_name": "", "author_url": "", "post_date": "2020-09-26T22:21:22.550000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1025600": "Hi Kagglers, I came across something important that I am excited to share. Despite my LB score improving from `32.210` to `25.524` after that fix with using `3x` less training epochs to get there, I am still not 100% sure that I do not miss something simple and obvious to everyone but me. But here it is for your judgement.\n\n# Statement\n\nWe have input images of the following kind, aligned for our agent vehicle to look right and at `'ego_center': [0.25, 0.5]`. It is the only input to our simple NNs that everyone uses.\n\n\n\nBut then we predict targets which are not aligned, at least if you follow [the example notebook](https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb) code. But the public notebooks that I saw use the same approach, if I draw several hundreds of target trajectories that we try to predict then it looks like this:\n\n\n\nIt is in all different directions, as expected. Then I spent a day trying to understand how is that possible to predict correct directions from direction-less input. To no avail, I still do not understand it. But the predictions are so accurate, they are actually very good! The direction is somehow predicted correctly from a direction-less input.\n\nAfter failing to understand why everyone does it wrong and why it even works, I tried to implement it correctly, that is, **transform the targets to image coordinates** before training. I attach the pytorch code with the transformations below so you can try it yourself fast. If you try and get an improvement please report it here.\n\nTransformed target trajectories look like this, this is 160 randomly sampled targets\n\n\n\nAnd it worked like a charm. With only 15 epochs, 10k batches each, batch size 16, which is about 5h training on my machine, I improved my training score from `30.87` to `23.79` and public LB `32.210` to `25.524`. \n\nSo how is the old approach able to predict the direction? I have two theories but no prove:\n1. Number of streets is very limited, and the network is able to learn what street has what orientation and then apply it to predictions.\n2. The image does somehow contain the rotation transformation in itself, maybe some subtle pixel level traces of it are there.\n\nWhat do you think? I have a lingering feeling that I am missing something obvious!\n\n# Code\n\nBelow boolean flag `cfg['train_params']['image_coords']` controls if I use the new approach. I apply the needed transformations on targets and then do inverse transformation on predictions in reverse order. Note that my predictions are multi-scenario there, you need to get rid of this additional dimensions if you use single scenario.\n\n```\ndef forward(data, model, device, criterion):\n inputs = data[\"image\"].to(device)\n target_availabilities = data[\"target_availabilities\"].to(device)\n targets = data[\"target_positions\"].to(device)\n matrix = data[\"world_to_image\"].to(device)\n centroid = data[\"centroid\"].to(device)[:,None,:].to(torch.float)\n \n # Forward pass\n outputs = model(inputs)\n \n bs,tl,_ = targets.shape\n assert tl == cfg[\"model_params\"][\"future_num_frames\"]\n \n if cfg['train_params']['image_coords']:\n targets = targets + centroid\n targets = torch.cat([targets,torch.ones((bs,tl,1)).to(device)], dim=2)\n targets = torch.matmul(matrix.to(torch.float), targets.transpose(1,2))\n targets = targets.transpose(1,2)[:,:,:2]\n bias = torch.tensor([56.25, 112.5])[None,None,:].to(device)\n targets = targets - bias\n \n confidences, pred = outputs[:,:3], outputs[:,3:]\n pred = pred.view(bs, 3, tl, 2)\n assert confidences.shape == (bs, 3)\n confidences = torch.softmax(confidences, dim=1)\n \n loss = criterion(targets, pred, confidences, target_availabilities)\n loss = torch.mean(loss)\n \n if cfg['train_params']['image_coords']:\n matrix_inv = torch.inverse(matrix)\n pred = pred + bias[:,None,:,:]\n pred = torch.cat([pred,torch.ones((bs,3,tl,1)).to(device)], dim=3)\n pred = torch.stack([torch.matmul(matrix_inv.to(torch.float), pred[:,i].transpose(1,2)) \n for i in range(3)], dim=1)\n pred = pred.transpose(2,3)[:,:,:,:2]\n pred = pred - centroid[:,None,:,:]\n \n return loss, pred, confidences\n```\n\nand in your loss function you may want to divide the coordinates by 2:\n\n```\n ...\n error = torch.sum(((gt - pred) * avails) ** 2, dim=-1) # reduce coords and use availability\n if cfg['train_params']['image_coords']:\n error = error / 4\n ...\n```", "1026361": "Hey Everyone,\n> What do you think? I have a lingering feeling that I am missing something obvious!\n\nOn the contrary, you're spot on :) This is something that slipped through our tests and that we only recently became aware of.\n\nUsing pixel coordinates is absolutely legit here, but **we're also working on a long term fix in L5Kit** we hope we can ship next week. Once that is thoroughly checked we will sync with Kaggle to update the default package.\n\nI'll start a post with more info about this issue and how we plan to fix it to increase visibility and awareness.", "1025660": "Did you calculate `bias = torch.tensor([56.25, 112.5])` from the `ego_center` config? So the input size is 225x225px?", "1029906": "Thank you for sharing this. We were doing it completely wrong, and most people would have been oblivious to that fact had you not so generously highlighted it. The epitome of community - kudos.", "1027976": "Thank you for sharing. \nWith this approach, in my case, the model(45.0) trained on all of the datasets was improved to the model(37.7) trained on 1/4 of the dataset.\nHowever, there is a gap between validation metric (13.1) and public LB. ", "1026441": "Good approach @zaharch . Congrats !\nCan I ask you why do you choose to divide the error by 4 ?", "1025888": "We really did it all wrong! Thanks for sharing and pointing it out. ", "1031260": "I'm really new to this community and this helped me a lot. Thanks!", "1028934": "Hi, Great insight! Can i know how you obtained those graphs?", "1027575": "@zaharch does this mean while inference we have to transform the predicted trajectories using the code block you shared\n```\n if cfg['train_params']['image_coords']:\n matrix_inv = torch.inverse(matrix)\n pred = pred + bias[:,None,:,:]\n pred = torch.cat([pred,torch.ones((bs,3,tl,1)).to(device)], dim=3)\n pred = torch.stack([torch.matmul(matrix_inv.to(torch.float), pred[:,i].transpose(1,2)) \n for i in range(3)], dim=1)\n pred = pred.transpose(2,3)[:,:,:,:2]\n pred = pred - centroid[:,None,:,:]\n```\nTrying to understand if I am interpreting the targets correctly", "1025678": "Awesome point. Thank you for sharing this", "1064052": "important!!!! \n1. check if you can get the same values if convert from world to image coordinates, and then back again.\n2. i think you need a 64-bit float (double) to do this", "1037369": "So I tried to modify the *forward* function above for the new l5kit package (version 1.1.0) following the notebook [Lyft-baseline-09-02](https://www.kaggle.com/lucabergamini/lyft-baseline-09-02?scriptVersionId=43751473)\n```\ndef forward(data, model, device, criterion = pytorch_neg_multi_log_likelihood_batch):\n inputs = data[\"image\"].to(device)\n target_availabilities = data[\"target_availabilities\"].to(device)\n targets = data[\"target_positions\"].to(device)\n \n # Forward pass\n preds, confidences = model(inputs) \n \n \n loss = criterion(targets, preds, confidences, target_availabilities)\n \n \n world_from_agents = data[\"world_from_agent\"].numpy()\n centroids = data[\"centroid\"].numpy()\n \n #convert into world coordinates and compute offsets\n for idx in range(len(preds)):\n preds[idx] = transform_points(preds[idx], world_from_agents[idx]) - centroids[idx]\n \n return loss, preds, confidences\n```\nSo as I understand the target in the train set is fixed, however the test set is not so we still need to convert it? However when I ran the above code I encountered the following error:\n\n```\n---------------------------------------------------------------------------\nAssertionError Traceback (most recent call last)\n