![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F19390%2Ffb1fe60c324bbe6fa781a144208052fb%2FScreenshot%202019-09-21%20at%2010.31.00.png?generation=1569051275412497&alt=media\")
Based on provided ground truth, objects that are completely occluded and would not be visible on either lidar or images, are often annotated, but sometimes they do disappear and re-appear when occluded, and it's not clear when should an occluded object be included in the annotations? Example image of complete occlusion from the end of scene 0, front camera:
\n\n
And another example from a bit earlier in the same scene, when an object suddenly appears while being completely occluded, as far as I can see:
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F19390%2Fccb55facb9db9fee1e6808dd5ba0dde5%2FScreenshot%202019-09-21%20at%2010.32.47.png?generation=1569051328923430&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F19390%2Fd7ab9f507f4247b6537b56e011561a93%2FScreenshot%202019-09-21%20at%2010.31.55.png?generation=1569051349413941&alt=media\")
Second question is when do parked cars get annotated? E.g. on scene 0 in the front left cam, we see that the red car is not annotated, while much less clear visible white cars are annotated:
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F19390%2Fd7b3ae91c2ae63c2b5c58d1bf53cd74c%2FScreenshot%202019-09-21%20at%2010.36.53.png?generation=1569051552553745&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F19390%2Ff61e32ab5aca1fcbc62fcafd3e11aefa%2FScreenshot%202019-09-21%20at%2010.37.07.png?generation=1569051596344350&alt=media\")
Such examples are almost in every frame, and they don't look like some annotation errors, more like a consequence of some process, and would be great to know some details about this process so that we could reproduce it better :)
", "messages": [ { "id": "631028", "postDate": "09/21/2019 07:40:55", "content": "Based on provided ground truth, objects that are completely occluded and would not be visible on either lidar or images, are often annotated, but sometimes they do disappear and re-appear when occluded, and it's not clear when should an occluded object be included in the annotations? Example image of complete occlusion from the end of scene 0, front camera:
\n\n
And another example from a bit earlier in the same scene, when an object suddenly appears while being completely occluded, as far as I can see:
\n\n
Second question is when do parked cars get annotated? E.g. on scene 0 in the front left cam, we see that the red car is not annotated, while much less clear visible white cars are annotated:
\n\n
Such examples are almost in every frame, and they don't look like some annotation errors, more like a consequence of some process, and would be great to know some details about this process so that we could reproduce it better :)
", "rawMarkdown": "Based on provided ground truth, objects that are completely occluded and would not be visible on either lidar or images, are often annotated, but sometimes they do disappear and re-appear when occluded, and it's not clear when should an occluded object be included in the annotations? Example image of complete occlusion from the end of scene 0, front camera:\n\n\n\nAnd another example from a bit earlier in the same scene, when an object suddenly appears while being completely occluded, as far as I can see:\n\n\n\n\n\nSecond question is when do parked cars get annotated? E.g. on scene 0 in the front left cam, we see that the red car is not annotated, while much less clear visible white cars are annotated:\n\n\n\n\n\nSuch examples are almost in every frame, and they don't look like some annotation errors, more like a consequence of some process, and would be great to know some details about this process so that we could reproduce it better :)", "votes": null }, { "id": "631036", "postDate": "09/21/2019 08:07:02", "content": "Hey can you provide sample token for these cases? \nAlso, I think the ground truth bounding boxes are annotated using the LIDAR data only (?) and the annotations which have to be shown on an image is decided by box_vis_level argument.
Why I think bounding boxes are annotated using LIDAR only: \n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F761152%2F04d8da250f818d45ab751b9709e9f345%2FWhatsApp%20Image%202019-09-21%20at%201.40.39%20PM.jpeg?generation=1569053510783388&alt=media\")
Your first image shows the annotation of the above red marked car which got occluded by that FedEx Van in the Front camera view. The question is why this annotation is plotted using default render_sample_data when none of this car's corners are visible? 🤔
> The question is why this annotation is plotted using default render_sample_data when none of this car's corners are visible?
\n\nI think it's nice that they plot all annotations we need to predict, even if they are occluded.
\n\n> I think the ground truth bounding boxes are annotated using the LIDAR data only (?)
\n\nyes, that could be the case, looking forward to clarification from the organizers. At least if something is not visible not lidar, it's not annotated it seems (at least if it's not occluded - maybe they take several sweeps when annotating data?).
\n\nRe LIDAR images for the second example, I'm not sure, maybe there are some readings from that cars even when they are occluded by the bus, here are my visualizations (don't have the tokens)
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F19390%2Ff8c113275b2e79f258b4c38c2fc42ce1%2FScreenshot%202019-09-21%20at%2011.33.51.png?generation=1569055172824834&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F19390%2F1ebb04bdad9b06a1ca058037fac74465%2FScreenshot%202019-09-21%20at%2011.33.59.png?generation=1569055191850422&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F19390%2F7304255e2c9fc9f88b5fe0d394564b88%2FScreenshot%202019-09-21%20at%2011.34.14.png?generation=1569055210408277&alt=media\")
And re the first example with FedEx (it's the last frame in the first scene), this is how it looks like on lidar:
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F19390%2Fe4caca641f728688c955f135d3110c01%2FScreenshot%202019-09-21%20at%2011.45.11.png?generation=1569055653252425&alt=media\")
but here again it's just one frame where it's occluded, so maybe we need to always take multiple sweeps.
", "rawMarkdown": "And re the first example with FedEx (it's the last frame in the first scene), this is how it looks like on lidar:\n\n\n\nbut here again it's just one frame where it's occluded, so maybe we need to always take multiple sweeps.", "votes": null }, { "id": "631059", "postDate": "09/21/2019 09:00:12", "content": "", "rawMarkdown": "", "votes": null }, { "id": "631060", "postDate": "09/21/2019 09:05:40", "content": "Alright, so the annotation of that huge bus is missing because all the corners of that bus are not at least 0.1 meters of the camera. See this
", "rawMarkdown": "Alright, so the annotation of that huge bus is missing because all the corners of that bus are not at least 0.1 meters of the camera. See [this](https://github.com/lyft/nuscenes-devkit/blob/master/lyft_dataset_sdk/utils/geometry_utils.py#L83)", "votes": null }, { "id": "631062", "postDate": "09/21/2019 09:07:30", "content": "box_in_image docstring says:
\"\"\"Check if a box is visible inside an image without accounting for occlusions.\"\"\"\nThis clears it up :)
\n\nOpened an issue in the lyft sdk repo.
", "rawMarkdown": "`box_in_image` [docstring](https://github.com/lyft/nuscenes-devkit/blob/master/lyft_dataset_sdk/utils/geometry_utils.py#L63) says:\n\n \"\"\"Check if a box is visible inside an image without accounting for occlusions.\"\"\"\n\nThis clears it up :)\n\nOpened an [issue](https://github.com/lyft/nuscenes-devkit/issues/40) in the lyft sdk repo.", "votes": null }, { "id": "631243", "postDate": "09/21/2019 16:52:09", "content": "thanks for sharing
", "rawMarkdown": "thanks for sharing", "votes": null }, { "id": "631783", "postDate": "09/22/2019 16:50:58", "content": "Another point on annotations process, is that bounding boxes seem to be based on LIDAR readings. On many frames we can see that ground truth boxes moves slightly ahead of the car if we view only images. But if we overlay LIDAR point cloud, we see that LIDAR point cloud of the object also moves a bit ahead of it, because LIDAR readings come a bit later than the image, since LIDAR needs some time to do the sweep.\nOn the other hand, knowing velocity of the object and parameters of the LIDAR (sweep speed and direction), in theory it should be possible to transform boxes so that they correspond to the image, and given that we know which points correspond to the object, it should be even possible to do the same for LIDAR points - but all of this looks not easy to do.
", "rawMarkdown": "Another point on annotations process, is that bounding boxes seem to be based on LIDAR readings. On many frames we can see that ground truth boxes moves slightly ahead of the car if we view only images. But if we overlay LIDAR point cloud, we see that LIDAR point cloud of the object also moves a bit ahead of it, because LIDAR readings come a bit later than the image, since LIDAR needs some time to do the sweep.\nOn the other hand, knowing velocity of the object and parameters of the LIDAR (sweep speed and direction), in theory it should be possible to transform boxes so that they correspond to the image, and given that we know which points correspond to the object, it should be even possible to do the same for LIDAR points - but all of this looks not easy to do.", "votes": null }, { "id": "632162", "postDate": "09/23/2019 09:55:53", "content": "Hello guys~I also confuse about the annotation. It seems that Lidar is enough to picture a 3D vision. And why we still need camera? If the bounding box is for both camera and lidar. Do we need to predict both model for the final score?
", "rawMarkdown": "Hello guys~I also confuse about the annotation. It seems that Lidar is enough to picture a 3D vision. And why we still need camera? If the bounding box is for both camera and lidar. Do we need to predict both model for the final score?", "votes": null }, { "id": "632197", "postDate": "09/23/2019 10:56:36", "content": "Hey Sonya, we have to predict 3d bounding boxes for a given lidar point cloud and corresponding images, there are many approaches to do that, some use only LIDAR and some use both the Lidar and the images.
", "rawMarkdown": "Hey Sonya, we have to predict 3d bounding boxes for a given lidar point cloud and corresponding images, there are many approaches to do that, some use only LIDAR and some use both the Lidar and the images.", "votes": null }, { "id": "634108", "postDate": "09/25/2019 21:12:17", "content": "well shared
", "rawMarkdown": "well shared", "votes": null }, { "id": "637291", "postDate": "09/30/2019 21:32:45", "content": "It's also a bit confusing for me. For example, in the following sample, the car behind the bushes is labeled, but from the lidar and camera data, it's really difficult to tell.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1479654%2F8503d32dc9e8073524950d789f0ce676%2Fcam.JPG?generation=1569879109107868&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1479654%2F97c6d15da8bb003e184b6405625559cd%2Flidar_top2.JPG?generation=1569879143652963&alt=media\")
Hey can you provide sample token for these cases? \nAlso, I think the ground truth bounding boxes are annotated using the LIDAR data only (?) and the annotations which have to be shown on an image is decided by box_vis_level argument.
Why I think bounding boxes are annotated using LIDAR only: \n
Your first image shows the annotation of the above red marked car which got occluded by that FedEx Van in the Front camera view. The question is why this annotation is plotted using default render_sample_data when none of this car's corners are visible? 🤔
> The question is why this annotation is plotted using default render_sample_data when none of this car's corners are visible?
\n\nI think it's nice that they plot all annotations we need to predict, even if they are occluded.
\n\n> I think the ground truth bounding boxes are annotated using the LIDAR data only (?)
\n\nyes, that could be the case, looking forward to clarification from the organizers. At least if something is not visible not lidar, it's not annotated it seems (at least if it's not occluded - maybe they take several sweeps when annotating data?).
\n\nRe LIDAR images for the second example, I'm not sure, maybe there are some readings from that cars even when they are occluded by the bus, here are my visualizations (don't have the tokens)
\n\n
And re the first example with FedEx (it's the last frame in the first scene), this is how it looks like on lidar:
\n\n
but here again it's just one frame where it's occluded, so maybe we need to always take multiple sweeps.
", "votes": null, "replies": [] }, { "id": 631059, "author_name": "rishabhiitbhu", "author_url": "", "post_date": "09/21/2019 09:00:12", "content": "", "votes": null, "replies": [] }, { "id": 631060, "author_name": "rishabhiitbhu", "author_url": "", "post_date": "09/21/2019 09:05:40", "content": "Alright, so the annotation of that huge bus is missing because all the corners of that bus are not at least 0.1 meters of the camera. See this
", "votes": null, "replies": [] }, { "id": 631062, "author_name": "rishabhiitbhu", "author_url": "", "post_date": "09/21/2019 09:07:30", "content": "box_in_image docstring says:
\"\"\"Check if a box is visible inside an image without accounting for occlusions.\"\"\"\nThis clears it up :)
\n\nOpened an issue in the lyft sdk repo.
", "votes": null, "replies": [] }, { "id": 632162, "author_name": "sunyuan0099", "author_url": "", "post_date": "09/23/2019 09:55:53", "content": "Hello guys~I also confuse about the annotation. It seems that Lidar is enough to picture a 3D vision. And why we still need camera? If the bounding box is for both camera and lidar. Do we need to predict both model for the final score?
", "votes": null, "replies": [] }, { "id": 632197, "author_name": "rishabhiitbhu", "author_url": "", "post_date": "09/23/2019 10:56:36", "content": "Hey Sonya, we have to predict 3d bounding boxes for a given lidar point cloud and corresponding images, there are many approaches to do that, some use only LIDAR and some use both the Lidar and the images.
", "votes": null, "replies": [] } ] }, { "id": 631243, "author_name": "ilkerayti", "author_url": "", "post_date": "09/21/2019 16:52:09", "content": "thanks for sharing
", "votes": null, "replies": [] }, { "id": 631783, "author_name": "lopuhin", "author_url": "", "post_date": "09/22/2019 16:50:58", "content": "Another point on annotations process, is that bounding boxes seem to be based on LIDAR readings. On many frames we can see that ground truth boxes moves slightly ahead of the car if we view only images. But if we overlay LIDAR point cloud, we see that LIDAR point cloud of the object also moves a bit ahead of it, because LIDAR readings come a bit later than the image, since LIDAR needs some time to do the sweep.\nOn the other hand, knowing velocity of the object and parameters of the LIDAR (sweep speed and direction), in theory it should be possible to transform boxes so that they correspond to the image, and given that we know which points correspond to the object, it should be even possible to do the same for LIDAR points - but all of this looks not easy to do.
", "votes": null, "replies": [] }, { "id": 634108, "author_name": "zaidkk", "author_url": "", "post_date": "09/25/2019 21:12:17", "content": "well shared
", "votes": null, "replies": [] }, { "id": 637291, "author_name": "yi233333", "author_url": "", "post_date": "09/30/2019 21:32:45", "content": "It's also a bit confusing for me. For example, in the following sample, the car behind the bushes is labeled, but from the lidar and camera data, it's really difficult to tell.\n\n