![\"world](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1010129%2Fff211c6780b02630d460dc0ec38e8787%2Fworld_from_agent.png?generation=1603958261780537&alt=media\")
I am trying to understand what the final transformation in the prediction was doing. From popular notebook like this, we have
\nfor idx in range(len(preds)):\n for mode in range(3):\n preds[idx, mode, :, :] = transform_points(preds[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]\nAnd, when looks into what the transform_points() is doing github link, which can be simplified as (for our 2d points):
def transform_points(points, transf_matrix)\n return (points @ transf_matrix[:2, :2].T) + transf_matrix[:2, 2]\nOne can simplify this entire transformation as simple matrix multiplication:
\npreds[idx, mode, :, :] = np.matmul(preds[idx, mode, :, :], world_from_agents[idx, :2, :2].T)\n + world_from_agents[idx, :2, 2] - centroids[idx, :2]\nThe funny thing is that when you check the world_from_agents matrix[:, :2, 2] matrix and centroids[:, :2] matrix. They are exactly the same….
![\"centroids\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1010129%2F19cd582ada25e21469f524908af1603d%2Fcentroid.png?generation=1603958284550144&alt=media\")
![\"diff\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1010129%2Fcd56524370ed3785d8398685f5ae950d%2Fdiff.png?generation=1603958304404184&alt=media\")
So I guess we can just simplify the transform as
\npreds[idx, mode, :, :] = np.matmul(preds[idx, mode, :, :], world_from_agents[idx, :2, :2].T)\nAnd, I believe with some numpy trick, you can easily do this for all the idx and mode without the for loop.
\nIs my assumption that the last column of world_from _agents is the same as the centroid correct?
\nI also noticed that the centroid is in real world coordinate so the value tend to be very large like ~500 compared to the agents local coordinate. So adding then subtracting such a large number can slightly harm the precision of prediction in general.
Edit:
\nI found that you can simplify the last transformation as
preds = torch.einsum('bmti,bji->bmtj', preds.double(), \n data[\"world_from_agent\"].to(device)[:, :2, :2]).cpu().numpy()\nI verify this transformation give the same score as the original formula on Kaggle in this notebook (v12)
\nhttps://www.kaggle.com/louis925/lyft-complete-train-and-prediction-pipeline/notebook#Prediction
I am trying to understand what the final transformation in the prediction was doing. From popular notebook like this, we have
\nfor idx in range(len(preds)):\n for mode in range(3):\n preds[idx, mode, :, :] = transform_points(preds[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]\nAnd, when looks into what the transform_points() is doing github link, which can be simplified as (for our 2d points):
def transform_points(points, transf_matrix)\n return (points @ transf_matrix[:2, :2].T) + transf_matrix[:2, 2]\nOne can simplify this entire transformation as simple matrix multiplication:
\npreds[idx, mode, :, :] = np.matmul(preds[idx, mode, :, :], world_from_agents[idx, :2, :2].T)\n + world_from_agents[idx, :2, 2] - centroids[idx, :2]\nThe funny thing is that when you check the world_from_agents matrix[:, :2, 2] matrix and centroids[:, :2] matrix. They are exactly the same….
So I guess we can just simplify the transform as
\npreds[idx, mode, :, :] = np.matmul(preds[idx, mode, :, :], world_from_agents[idx, :2, :2].T)\nAnd, I believe with some numpy trick, you can easily do this for all the idx and mode without the for loop.
\nIs my assumption that the last column of world_from _agents is the same as the centroid correct?
\nI also noticed that the centroid is in real world coordinate so the value tend to be very large like ~500 compared to the agents local coordinate. So adding then subtracting such a large number can slightly harm the precision of prediction in general.
Edit:
\nI found that you can simplify the last transformation as
preds = torch.einsum('bmti,bji->bmtj', preds.double(), \n data[\"world_from_agent\"].to(device)[:, :2, :2]).cpu().numpy()\nI verify this transformation give the same score as the original formula on Kaggle in this notebook (v12)
\nhttps://www.kaggle.com/louis925/lyft-complete-train-and-prediction-pipeline/notebook#Prediction
OK, look at the code, https://github.com/lyft/l5kit/blob/master/l5kit/l5kit/geometry/transform.py#L8-L25
\nThe last column is indeed copy from agent_centroid_m.
Look at more than one sample from the dataset. You've chosen a sample where the AV is the agent. These transformations were introduced in l5kit 1.1.0 to fix an issue where the targets for non-AV vehicles were not properly rotated so while properly oriented for the AV they were randomly oriented for other vehicles. The l5kit rasterizer was updated to properly transform targets but they chose not to modify the competition ground truths given the issues this would cause (invalidating all old submissions). This transformation corrects predictions to match the mistaken ground truths. So it has no affect on AVs but does affect other vehicles.
", "rawMarkdown": "Look at more than one sample from the dataset. You've chosen a sample where the AV is the agent. These transformations were introduced in l5kit 1.1.0 to fix an issue where the targets for non-AV vehicles were not properly rotated so while properly oriented for the AV they were randomly oriented for other vehicles. The l5kit rasterizer was updated to properly transform targets but they chose not to modify the competition ground truths given the issues this would cause (invalidating all old submissions). This transformation corrects predictions to match the mistaken ground truths. So it has no affect on AVs but does affect other vehicles.", "votes": null }, { "id": "1065067", "postDate": "10/30/2020 20:06:10", "content": "I have look into the code. The last column of the transform matrix is just copy from centroid. So they are indeed identical. My test code also get the same scores between two approaches.
", "rawMarkdown": "I have look into the code. The last column of the transform matrix is just copy from centroid. So they are indeed identical. My test code also get the same scores between two approaches.", "votes": null }, { "id": "1066064", "postDate": "11/01/2020 09:18:05", "content": "I found that you can simplify the last transformation as
\npreds = torch.einsum('bmti,bji->bmtj', preds.double(), \n data[\"world_from_agent\"].to(device)[:, :2, :2]).cpu().numpy()\nand doing all of them on GPU using pytorch instead of CPU. I verify this transformation give the same score as the original formula on Kaggle.
", "rawMarkdown": "I found that you can simplify the last transformation as\n```\npreds = torch.einsum('bmti,bji->bmtj', preds.double(), \n data[\"world_from_agent\"].to(device)[:, :2, :2]).cpu().numpy()\n```\nand doing all of them on GPU using pytorch instead of CPU. I verify this transformation give the same score as the original formula on Kaggle.", "votes": null }, { "id": "1069370", "postDate": "11/04/2020 11:13:12", "content": "Or, perhaps, one can do everything on CPU. Not sure which one is faster for inference.
", "rawMarkdown": "Or, perhaps, one can do everything on CPU. Not sure which one is faster for inference.", "votes": null }, { "id": "1069475", "postDate": "11/04/2020 13:57:32", "content": "Yes, my mistake. The world_from_agent is doing a post rotation translation. So the last column is just adding the centroid. So your simplification is entirely valid.
Thanks @louis925 for sharing this. Looking at your kernel, I am suprised this did not improve the inference time by more than a few seconds. Is that right?
", "rawMarkdown": "Thanks @louis925 for sharing this. Looking at your kernel, I am suprised this did not improve the inference time by more than a few seconds. Is that right?", "votes": null }, { "id": "1071354", "postDate": "11/06/2020 19:04:12", "content": "Right, it doesn't seem to improve on Kaggle notebook. However, it seem to improve on my local machine. Maybe someone can double check that.
", "rawMarkdown": "Right, it doesn't seem to improve on Kaggle notebook. However, it seem to improve on my local machine. Maybe someone can double check that.", "votes": null } ], "comments": [ { "id": 1063717, "author_name": "louis925", "author_url": "", "post_date": "10/29/2020 08:23:17", "content": "OK, look at the code, https://github.com/lyft/l5kit/blob/master/l5kit/l5kit/geometry/transform.py#L8-L25
\nThe last column is indeed copy from agent_centroid_m.
Look at more than one sample from the dataset. You've chosen a sample where the AV is the agent. These transformations were introduced in l5kit 1.1.0 to fix an issue where the targets for non-AV vehicles were not properly rotated so while properly oriented for the AV they were randomly oriented for other vehicles. The l5kit rasterizer was updated to properly transform targets but they chose not to modify the competition ground truths given the issues this would cause (invalidating all old submissions). This transformation corrects predictions to match the mistaken ground truths. So it has no affect on AVs but does affect other vehicles.
", "votes": null, "replies": [ { "id": 1065067, "author_name": "louis925", "author_url": "", "post_date": "10/30/2020 20:06:10", "content": "I have look into the code. The last column of the transform matrix is just copy from centroid. So they are indeed identical. My test code also get the same scores between two approaches.
", "votes": null, "replies": [] }, { "id": 1069475, "author_name": "thomasbrandon", "author_url": "", "post_date": "11/04/2020 13:57:32", "content": "Yes, my mistake. The world_from_agent is doing a post rotation translation. So the last column is just adding the centroid. So your simplification is entirely valid.
I found that you can simplify the last transformation as
\npreds = torch.einsum('bmti,bji->bmtj', preds.double(), \n data[\"world_from_agent\"].to(device)[:, :2, :2]).cpu().numpy()\nand doing all of them on GPU using pytorch instead of CPU. I verify this transformation give the same score as the original formula on Kaggle.
", "votes": null, "replies": [ { "id": 1069370, "author_name": "louis925", "author_url": "", "post_date": "11/04/2020 11:13:12", "content": "Or, perhaps, one can do everything on CPU. Not sure which one is faster for inference.
", "votes": null, "replies": [] } ] }, { "id": 1071150, "author_name": "sheriytm", "author_url": "", "post_date": "11/06/2020 15:01:11", "content": "Thanks @louis925 for sharing this. Looking at your kernel, I am suprised this did not improve the inference time by more than a few seconds. Is that right?
", "votes": null, "replies": [ { "id": 1071354, "author_name": "louis925", "author_url": "", "post_date": "11/06/2020 19:04:12", "content": "Right, it doesn't seem to improve on Kaggle notebook. However, it seem to improve on my local machine. Maybe someone can double check that.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1063691": "I am trying to understand what the final transformation in the prediction was doing. From popular notebook like [this](https://www.kaggle.com/huanvo/lyft-complete-train-and-prediction-pipeline), we have\n```python\nfor idx in range(len(preds)):\n for mode in range(3):\n preds[idx, mode, :, :] = transform_points(preds[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]\n```\nAnd, when looks into what the `transform_points()` is doing [github link](https://github.com/lyft/l5kit/blob/master/l5kit/l5kit/geometry/transform.py#L73-L99), which can be simplified as (for our 2d points):\n```\ndef transform_points(points, transf_matrix)\n return (points @ transf_matrix[:2, :2].T) + transf_matrix[:2, 2]\n```\nOne can simplify this entire transformation as simple matrix multiplication:\n```\npreds[idx, mode, :, :] = np.matmul(preds[idx, mode, :, :], world_from_agents[idx, :2, :2].T)\n + world_from_agents[idx, :2, 2] - centroids[idx, :2]\n```\nThe funny thing is that when you check the `world_from_agents matrix[:, :2, 2]` matrix and `centroids[:, :2]` matrix. They are exactly the same....\n\n\n\n\n\n\n\nSo I guess we can just simplify the transform as \n```\npreds[idx, mode, :, :] = np.matmul(preds[idx, mode, :, :], world_from_agents[idx, :2, :2].T)\n```\nAnd, I believe with some numpy trick, you can easily do this for all the idx and mode without the for loop. \nIs my assumption that the last column of `world_from _agents` is the same as the `centroid` correct?\nI also noticed that the centroid is in real world coordinate so the value tend to be very large like ~500 compared to the agents local coordinate. So adding then subtracting such a large number can slightly harm the precision of prediction in general.\n\nEdit:\nI found that you can simplify the last transformation as\n```\npreds = torch.einsum('bmti,bji->bmtj', preds.double(), \n data[\"world_from_agent\"].to(device)[:, :2, :2]).cpu().numpy()\n```\nI verify this transformation give the same score as the original formula on Kaggle in this notebook (v12)\nhttps://www.kaggle.com/louis925/lyft-complete-train-and-prediction-pipeline/notebook#Prediction", "1063717": "OK, look at the code, https://github.com/lyft/l5kit/blob/master/l5kit/l5kit/geometry/transform.py#L8-L25\nThe last column is indeed copy from `agent_centroid_m`.", "1063724": "Look at more than one sample from the dataset. You've chosen a sample where the AV is the agent. These transformations were introduced in l5kit 1.1.0 to fix an issue where the targets for non-AV vehicles were not properly rotated so while properly oriented for the AV they were randomly oriented for other vehicles. The l5kit rasterizer was updated to properly transform targets but they chose not to modify the competition ground truths given the issues this would cause (invalidating all old submissions). This transformation corrects predictions to match the mistaken ground truths. So it has no affect on AVs but does affect other vehicles.", "1065067": "I have look into the code. The last column of the transform matrix is just copy from centroid. So they are indeed identical. My test code also get the same scores between two approaches.", "1066064": "I found that you can simplify the last transformation as\n```\npreds = torch.einsum('bmti,bji->bmtj', preds.double(), \n data[\"world_from_agent\"].to(device)[:, :2, :2]).cpu().numpy()\n```\nand doing all of them on GPU using pytorch instead of CPU. I verify this transformation give the same score as the original formula on Kaggle.", "1069370": "Or, perhaps, one can do everything on CPU. Not sure which one is faster for inference.", "1069475": "Yes, my mistake. The `world_from_agent` is doing a post rotation translation. So the last column is just adding the centroid. So your simplification is entirely valid.", "1071150": "Thanks @louis925 for sharing this. Looking at your kernel, I am suprised this did not improve the inference time by more than a few seconds. Is that right?", "1071354": "Right, it doesn't seem to improve on Kaggle notebook. However, it seem to improve on my local machine. Maybe someone can double check that." }, "source": "meta" }