I noticed that agents are not always perfectly heading toward the +x direction in the agent coordinate frame at t=0 (with the new l5kit v1.1.0). Rather, it has some slight error in the y direction so even if an agent is travel almost on a straight line, it still shift slightly in the y-axis.
\nLooks like in the l5kit code, it computes the transformation from world to agent space using the observation yaw. Perhaps, there is some precision limit or bias in the way they detect yaw?
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1010129%2Fbbff7631637bdc00f9c5bb933c9e9aec%2Fslight%20error%20in%20the%20agent%20yaw.png?generation=1606075325540714&alt=media\")
\n(Though looks like the error in y is less than 1 m for most of the examples)
\nI am trying to correct this in the l5kit using the average velocity now.
\nHave anyone tried similar correction before?
I noticed that agents are not always perfectly heading toward the +x direction in the agent coordinate frame at t=0 (with the new l5kit v1.1.0). Rather, it has some slight error in the y direction so even if an agent is travel almost on a straight line, it still shift slightly in the y-axis.
\nLooks like in the l5kit code, it computes the transformation from world to agent space using the observation yaw. Perhaps, there is some precision limit or bias in the way they detect yaw?
\n
\n(Though looks like the error in y is less than 1 m for most of the examples)
\nI am trying to correct this in the l5kit using the average velocity now.
\nHave anyone tried similar correction before?
I tried to correct the current frame yaw (the important one) by filtering, but it didn't help. I am reasonably sure that the yaws (and other stats) are already filtered, most probably by Kalman filter.
", "rawMarkdown": "I tried to correct the current frame yaw (the important one) by filtering, but it didn't help. I am reasonably sure that the yaws (and other stats) are already filtered, most probably by Kalman filter.", "votes": null }, { "id": "1087595", "postDate": "11/22/2020 22:12:21", "content": "well, the GT for the future frames are calculated by the \"wrong\" yaw, too which is the origin for all future positions. So, you would need to correct this as well. Otherwise, LB score cannot even theoretically be improved.
", "rawMarkdown": "well, the GT for the future frames are calculated by the \"wrong\" yaw, too which is the origin for all future positions. So, you would need to correct this as well. Otherwise, LB score cannot even theoretically be improved.", "votes": null }, { "id": "1087597", "postDate": "11/22/2020 22:21:05", "content": "not sure, but I think what you are talking about is exactly what I did. I tried to improve the current frame yaw estimation and rotated everything accordingly with that new estimate. Do you use it?
", "rawMarkdown": "not sure, but I think what you are talking about is exactly what I did. I tried to improve the current frame yaw estimation and rotated everything accordingly with that new estimate. Do you use it?", "votes": null }, { "id": "1087600", "postDate": "11/22/2020 22:26:55", "content": "Yeah, it's a little tricky to explain and to discuss.
\nworld_from_agent value is not correct for inference. \n\nDo you use it?
\n
In the current state of the competion I would rather go with the \"I can neither confirm nor deny\"
", "rawMarkdown": "Yeah, it's a little tricky to explain and to discuss. \n1. Did you create a new gt.csv to evaluate the performance or how did you evaluate it? If so, I believe your approach was correct and if it didn't improve the results it may just not work with your model\n2. If not, the `world_from_agent` value is not correct for inference. \n3. If 1. was showing better results but they didn't work for LB, it was probably due to 2.\n\n> Do you use it?\n\nIn the current state of the competion I would rather go with the \"I can neither confirm nor deny\"", "votes": null }, { "id": "1087603", "postDate": "11/22/2020 22:38:54", "content": "Fair enough. Not sure a new gt.csv is needed though, because it is without rotation (world rotation). I am implying a validation on a chopped dataset here. I did correct the matrices, like world_from_agent. But I didn't even try to submit, just noticed that the validation got worse.
That's exactly what i mean. If you corrected world_from_agent by your new yaw, the gt.csv, which was once created when you made your chopped validation dataset was calculated with a different world_from_agent. So, your new preds won't do any good on the old chopped dataset (same problem with the test set from the LB)
You would need to revert your correction to actually predict the future positions from a wrong origin.
", "rawMarkdown": "That's exactly what i mean. If you corrected `world_from_agent` by your new yaw, the gt.csv, which was once created when you made your chopped validation dataset was calculated with a different `world_from_agent`. So, your new preds won't do any good on the old chopped dataset (same problem with the test set from the LB)\n\nYou would need to revert your correction to actually predict the future positions from a wrong origin.", "votes": null }, { "id": "1087609", "postDate": "11/22/2020 22:50:18", "content": "And my point is that world_from_agent was not used to produce gt.csv. It was not rotated to the agent view, it is still the good old world view. I do not need to correct it because it was never rotated to begin with. One of us is mistaken :)
Oh, true, it's me with the mistake ;)
\nIf you correctly calculated world_from_agent for the new yaw, the results are indeed in the correct coordinate frame already and validation score can be trusted. No new gt.csv needed.
Good discussion. Right! I think the truth in chopped dataset (as well as what is evaluated on kaggle) is still in the world orientation but only shifted to the agent centroid so it is not affected by the agent yaw.
", "rawMarkdown": "Good discussion. Right! I think the truth in chopped dataset (as well as what is evaluated on kaggle) is still in the world orientation but only shifted to the agent centroid so it is not affected by the agent yaw.", "votes": null }, { "id": "1089559", "postDate": "11/24/2020 15:51:13", "content": "very interesting discussion and there might be a way to use this to perturb the data (pre-rasterisation) -- no time to explore that!
", "rawMarkdown": "very interesting discussion and there might be a way to use this to perturb the data (pre-rasterisation) -- no time to explore that!", "votes": null } ], "comments": [ { "id": 1087571, "author_name": "zaharch", "author_url": "", "post_date": "11/22/2020 21:32:14", "content": "I tried to correct the current frame yaw (the important one) by filtering, but it didn't help. I am reasonably sure that the yaws (and other stats) are already filtered, most probably by Kalman filter.
", "votes": null, "replies": [ { "id": 1087595, "author_name": "ilu000", "author_url": "", "post_date": "11/22/2020 22:12:21", "content": "well, the GT for the future frames are calculated by the \"wrong\" yaw, too which is the origin for all future positions. So, you would need to correct this as well. Otherwise, LB score cannot even theoretically be improved.
", "votes": null, "replies": [] }, { "id": 1087597, "author_name": "zaharch", "author_url": "", "post_date": "11/22/2020 22:21:05", "content": "not sure, but I think what you are talking about is exactly what I did. I tried to improve the current frame yaw estimation and rotated everything accordingly with that new estimate. Do you use it?
", "votes": null, "replies": [] }, { "id": 1087600, "author_name": "ilu000", "author_url": "", "post_date": "11/22/2020 22:26:55", "content": "Yeah, it's a little tricky to explain and to discuss.
\nworld_from_agent value is not correct for inference. \n\nDo you use it?
\n
In the current state of the competion I would rather go with the \"I can neither confirm nor deny\"
", "votes": null, "replies": [] }, { "id": 1087603, "author_name": "zaharch", "author_url": "", "post_date": "11/22/2020 22:38:54", "content": "Fair enough. Not sure a new gt.csv is needed though, because it is without rotation (world rotation). I am implying a validation on a chopped dataset here. I did correct the matrices, like world_from_agent. But I didn't even try to submit, just noticed that the validation got worse.
That's exactly what i mean. If you corrected world_from_agent by your new yaw, the gt.csv, which was once created when you made your chopped validation dataset was calculated with a different world_from_agent. So, your new preds won't do any good on the old chopped dataset (same problem with the test set from the LB)
You would need to revert your correction to actually predict the future positions from a wrong origin.
", "votes": null, "replies": [] }, { "id": 1087609, "author_name": "zaharch", "author_url": "", "post_date": "11/22/2020 22:50:18", "content": "And my point is that world_from_agent was not used to produce gt.csv. It was not rotated to the agent view, it is still the good old world view. I do not need to correct it because it was never rotated to begin with. One of us is mistaken :)
Oh, true, it's me with the mistake ;)
\nIf you correctly calculated world_from_agent for the new yaw, the results are indeed in the correct coordinate frame already and validation score can be trusted. No new gt.csv needed.
Good discussion. Right! I think the truth in chopped dataset (as well as what is evaluated on kaggle) is still in the world orientation but only shifted to the agent centroid so it is not affected by the agent yaw.
", "votes": null, "replies": [] }, { "id": 1089559, "author_name": "ggrizzly", "author_url": "", "post_date": "11/24/2020 15:51:13", "content": "very interesting discussion and there might be a way to use this to perturb the data (pre-rasterisation) -- no time to explore that!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1087539": "I noticed that agents are not always perfectly heading toward the +x direction in the agent coordinate frame at t=0 (with the new l5kit v1.1.0). Rather, it has some slight error in the y direction so even if an agent is travel almost on a straight line, it still shift slightly in the y-axis. \nLooks like in the l5kit code, it computes the transformation from world to agent space using the observation `yaw`. Perhaps, there is some precision limit or bias in the way they detect `yaw`? \n\n(Though looks like the error in y is less than 1 m for most of the examples)\nI am trying to correct this in the l5kit using the average velocity now. \nHave anyone tried similar correction before?", "1087571": "I tried to correct the current frame yaw (the important one) by filtering, but it didn't help. I am reasonably sure that the yaws (and other stats) are already filtered, most probably by Kalman filter.", "1087595": "well, the GT for the future frames are calculated by the \"wrong\" yaw, too which is the origin for all future positions. So, you would need to correct this as well. Otherwise, LB score cannot even theoretically be improved.", "1087597": "not sure, but I think what you are talking about is exactly what I did. I tried to improve the current frame yaw estimation and rotated everything accordingly with that new estimate. Do you use it?", "1087600": "Yeah, it's a little tricky to explain and to discuss. \n1. Did you create a new gt.csv to evaluate the performance or how did you evaluate it? If so, I believe your approach was correct and if it didn't improve the results it may just not work with your model\n2. If not, the `world_from_agent` value is not correct for inference. \n3. If 1. was showing better results but they didn't work for LB, it was probably due to 2.\n\n> Do you use it?\n\nIn the current state of the competion I would rather go with the \"I can neither confirm nor deny\"", "1087603": "Fair enough. Not sure a new gt.csv is needed though, because it is without rotation (world rotation). I am implying a validation on a chopped dataset here. I did correct the matrices, like `world_from_agent`. But I didn't even try to submit, just noticed that the validation got worse.", "1087607": "That's exactly what i mean. If you corrected `world_from_agent` by your new yaw, the gt.csv, which was once created when you made your chopped validation dataset was calculated with a different `world_from_agent`. So, your new preds won't do any good on the old chopped dataset (same problem with the test set from the LB)\n\nYou would need to revert your correction to actually predict the future positions from a wrong origin.", "1087609": "And my point is that `world_from_agent` was not used to produce `gt.csv`. It was not rotated to the agent view, it is still the good old world view. I do not need to correct it because it was never rotated to begin with. One of us is mistaken :)", "1087629": "Oh, true, it's me with the mistake ;)\nIf you correctly calculated `world_from_agent` for the new yaw, the results are indeed in the correct coordinate frame already and validation score can be trusted. No new gt.csv needed.", "1087654": "Good discussion. Right! I think the truth in chopped dataset (as well as what is evaluated on kaggle) is still in the world orientation but only shifted to the agent centroid so it is not affected by the agent yaw.", "1089559": "very interesting discussion and there might be a way to use this to perturb the data (pre-rasterisation) -- no time to explore that!" }, "source": "meta" }