I'm exploring ego dataset and agent dataset, which are supposed to iterate over AV annotations and agent annotations respectively. I've checked existing data exploration notebooks and they don't cover this. Here are some doubts regarding that:
\nHow to interpret consecutive images of EgoDataset? Is there any relation at all? I've plotted two such images with annotations and they look almost the same:
\nThis is image at 3rd index (0 based index) in Ego dataset:![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1905996%2F264ba27acb199d48469e523f767a3eda%2FScreenshot%202020-08-25%20at%2011.14.42%20PM.png?generation=1598377557649495&alt=media\")
And this is image at 4th index: ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1905996%2Fc99762ece56d67c19bb82e3f48f739a7%2FScreenshot%202020-08-25%20at%2011.16.07%20PM.png?generation=1598377617015873&alt=media\")
These looks almost same except that blue annotation (I think it's an agent). I observed similar behaviour in AgentDataset (different points with similar images) also. So what is this iterating over? Are these frames of a single scene (in case of EgoDataset)?
\nAlso, when I print the data point dict returned by this class, there are lots of attributes, which might be useful. So is there any description about them at a single place? Here is an example of these attributes returned from EgoDataset class:
\n\n\ndict_keys(['image', 'target_positions', 'target_yaws', 'target_availabilities', 'history_positions', 'history_yaws', 'history_availabilities', 'world_to_image', 'track_id', 'timestamp', 'centroid', 'yaw', 'extent'])
\n
Some of these are self-explanatory or explained somewhere in different docs, but others are unknown (for me), for example: history_availabilities, world_to_image etc.
Thanks!
", "messages": [ { "id": "985418", "postDate": "08/25/2020 18:00:53", "content": "I'm exploring ego dataset and agent dataset, which are supposed to iterate over AV annotations and agent annotations respectively. I've checked existing data exploration notebooks and they don't cover this. Here are some doubts regarding that:
\nHow to interpret consecutive images of EgoDataset? Is there any relation at all? I've plotted two such images with annotations and they look almost the same:
\nThis is image at 3rd index (0 based index) in Ego dataset:
And this is image at 4th index:
These looks almost same except that blue annotation (I think it's an agent). I observed similar behaviour in AgentDataset (different points with similar images) also. So what is this iterating over? Are these frames of a single scene (in case of EgoDataset)?
\nAlso, when I print the data point dict returned by this class, there are lots of attributes, which might be useful. So is there any description about them at a single place? Here is an example of these attributes returned from EgoDataset class:
\n\n\ndict_keys(['image', 'target_positions', 'target_yaws', 'target_availabilities', 'history_positions', 'history_yaws', 'history_availabilities', 'world_to_image', 'track_id', 'timestamp', 'centroid', 'yaw', 'extent'])
\n
Some of these are self-explanatory or explained somewhere in different docs, but others are unknown (for me), for example: history_availabilities, world_to_image etc.
Thanks!
", "rawMarkdown": "I'm exploring ego dataset and agent dataset, which are supposed to iterate over AV annotations and agent annotations respectively. I've checked existing data exploration notebooks and they don't cover this. Here are some doubts regarding that:\n\n1. How to interpret consecutive images of EgoDataset? Is there any relation at all? I've plotted two such images with annotations and they look almost the same:\n\n This is image at 3rd index (0 based index) in Ego dataset:\n\n And this is image at 4th index: \n \nThese looks almost same except that blue annotation (I think it's an agent). I observed similar behaviour in AgentDataset (different points with similar images) also. So what is this iterating over? Are these frames of a single scene (in case of EgoDataset)?\n\nAlso, when I print the data point dict returned by this class, there are lots of attributes, which might be useful. So is there any description about them at a single place? Here is an example of these attributes returned from EgoDataset class:\n\n> dict_keys(['image', 'target_positions', 'target_yaws', 'target_availabilities', 'history_positions', 'history_yaws', 'history_availabilities', 'world_to_image', 'track_id', 'timestamp', 'centroid', 'yaw', 'extent'])\n\nSome of these are self-explanatory or explained somewhere in different docs, but others are unknown (for me), for example: `history_availabilities`, `world_to_image` etc.\n\nThanks!", "votes": null }, { "id": "986272", "postDate": "08/26/2020 10:32:25", "content": "Hi @kaushal2896 ,
\nThe EgoDataset iterates on the AV positions as it moves through the world. If you're not performing any shuffling, frames 3 and 4 will be highly correlated (0.1 seconds between them), so it makes sense that the two rasters appears very similar. Agents around come and go based on perception output.
I agree with you on the lack of details, I'll start a PR in L5Kit to include a description of the dict returned by the EgoDataset in the visualisation notebook :)
@lucabergamini Thank you very much!
", "rawMarkdown": "lucabergamini Thank you very much!", "votes": null }, { "id": "987064", "postDate": "08/27/2020 00:39:39", "content": "Hi @lucabergamini,
\nCould you provide the definitions of ego_translation vector and ego_rotation matrix? I'm also wondering the units of the times in the dataset. The durations yield something on the order of 1e+9 or bigger.
Why all history_positions contain [0,0] as the first entry in AgentDataset?
\nWhen history_num_frames=2, history_positions of AgentDataset contains :
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1272379%2Fca58c9e1e389b229bbc2ba10e03b8824%2Fhistory%20positions.png?generation=1599388048371879&alt=media\")
first row is the current position, which is always in the origin (0,0).
", "rawMarkdown": "first row is the current position, which is always in the origin (0,0).", "votes": null }, { "id": "1000777", "postDate": "09/06/2020 19:32:53", "content": "May I ask what target_availabilities / history_availabilities mean? I believe target_availabilities is also used in the example prediction notebook.
hi Frank, great to see you here.
\nthe target_availabilities are masks that are used when calculating the loss. Not all targets (ground truths) are available, e.g. an agent was no longer visible for the AV.
\nhistory_availabilities is the same thing for the past frames. Not all frames are available.
Hi @ilu000, great to see you here too! That makes a lot of senses - agents can enter and leave a frame from time to time. Thank you for your clarification!
", "rawMarkdown": "Hi @ilu000, great to see you here too! That makes a lot of senses - agents can enter and leave a frame from time to time. Thank you for your clarification!", "votes": null } ], "comments": [ { "id": 986272, "author_name": "lucabergamini", "author_url": "", "post_date": "08/26/2020 10:32:25", "content": "Hi @kaushal2896 ,
\nThe EgoDataset iterates on the AV positions as it moves through the world. If you're not performing any shuffling, frames 3 and 4 will be highly correlated (0.1 seconds between them), so it makes sense that the two rasters appears very similar. Agents around come and go based on perception output.
I agree with you on the lack of details, I'll start a PR in L5Kit to include a description of the dict returned by the EgoDataset in the visualisation notebook :)
@lucabergamini Thank you very much!
", "votes": null, "replies": [] }, { "id": 987064, "author_name": "tolgadincer", "author_url": "", "post_date": "08/27/2020 00:39:39", "content": "Hi @lucabergamini,
\nCould you provide the definitions of ego_translation vector and ego_rotation matrix? I'm also wondering the units of the times in the dataset. The durations yield something on the order of 1e+9 or bigger.
Why all history_positions contain [0,0] as the first entry in AgentDataset?
\nWhen history_num_frames=2, history_positions of AgentDataset contains :
\nfirst row is the current position, which is always in the origin (0,0).
", "votes": null, "replies": [] } ] }, { "id": 1000777, "author_name": "pukkinming", "author_url": "", "post_date": "09/06/2020 19:32:53", "content": "May I ask what target_availabilities / history_availabilities mean? I believe target_availabilities is also used in the example prediction notebook.
hi Frank, great to see you here.
\nthe target_availabilities are masks that are used when calculating the loss. Not all targets (ground truths) are available, e.g. an agent was no longer visible for the AV.
\nhistory_availabilities is the same thing for the past frames. Not all frames are available.
Hi @ilu000, great to see you here too! That makes a lot of senses - agents can enter and leave a frame from time to time. Thank you for your clarification!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "985418": "I'm exploring ego dataset and agent dataset, which are supposed to iterate over AV annotations and agent annotations respectively. I've checked existing data exploration notebooks and they don't cover this. Here are some doubts regarding that:\n\n1. How to interpret consecutive images of EgoDataset? Is there any relation at all? I've plotted two such images with annotations and they look almost the same:\n\n This is image at 3rd index (0 based index) in Ego dataset:\n\n And this is image at 4th index: \n \nThese looks almost same except that blue annotation (I think it's an agent). I observed similar behaviour in AgentDataset (different points with similar images) also. So what is this iterating over? Are these frames of a single scene (in case of EgoDataset)?\n\nAlso, when I print the data point dict returned by this class, there are lots of attributes, which might be useful. So is there any description about them at a single place? Here is an example of these attributes returned from EgoDataset class:\n\n> dict_keys(['image', 'target_positions', 'target_yaws', 'target_availabilities', 'history_positions', 'history_yaws', 'history_availabilities', 'world_to_image', 'track_id', 'timestamp', 'centroid', 'yaw', 'extent'])\n\nSome of these are self-explanatory or explained somewhere in different docs, but others are unknown (for me), for example: `history_availabilities`, `world_to_image` etc.\n\nThanks!", "986272": "Hi @kaushal2896 , \nThe EgoDataset iterates on the AV positions as it moves through the world. If you're not performing any shuffling, frames 3 and 4 will be highly correlated (0.1 seconds between them), so it makes sense that the two rasters appears very similar. Agents around come and go based on perception output.\n\nI agree with you on the lack of details, I'll start a PR in L5Kit to include a description of the dict returned by the `EgoDataset` in the visualisation notebook :)", "986319": "lucabergamini Thank you very much!", "987064": "Hi @lucabergamini,\nCould you provide the definitions of ego_translation vector and ego_rotation matrix? I'm also wondering the units of the times in the dataset. The durations yield something on the order of 1e+9 or bigger.", "1000144": "Why all history_positions contain [0,0] as the first entry in AgentDataset? \n\nWhen history_num_frames=2, history_positions of AgentDataset contains :\n1. 3 coordinates instead of 2 why? It outputs 3x2 array instead of 2x2. Is it including the current position in that?\n2. The first row is always [0,0] across all data samples. What does this signify? ", "1000751": "first row is the current position, which is always in the origin (0,0).", "1000777": "May I ask what `target_availabilities` / `history_availabilities` mean? I believe `target_availabilities` is also used in the [example prediction notebook](https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb).", "1000830": "hi Frank, great to see you here.\nthe `target_availabilities` are masks that are used when calculating the loss. Not all targets (ground truths) are available, e.g. an agent was no longer visible for the AV. \n`history_availabilities` is the same thing for the past frames. Not all frames are available.", "1000946": "Hi @ilu000, great to see you here too! That makes a lot of senses - agents can enter and leave a frame from time to time. Thank you for your clarification!" }, "source": "meta" }