![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1634816%2Fbed12b74e449a26d2f730ebbd2229961%2FUntitled.png?generation=1575455317465481&alt=media\")
Currently the distance threshold is set to have a minimum of 0.1m, which is too strict in my opinion, and that explains why the score in the leaderboard is so bad.
\n\nFor the comparison I'll make below, let me first introduce another metric that is also commonly used for 3d detection: the 3d iou(https://en.wikipedia.org/wiki/Jaccard_index). Normally, a standard passenger car has length of about 4m, so the 0.1m distance threshold in this competition translates into about (4-0.1)/(4+0.1)=0.95 3d iou.
\n\nHaving that number in mind, let me introduce some well-known 3d detection task as example:\nLyft: requires 3d iou of 0.5(https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/overview/evaluation)\nKITTI: requires 3d iou of 0.7 (http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d)\nnuscenes: requires distance threshold of min 4m (https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any)\nand finally, in the original apolloscape challenge: requires distance threshold of min 2.8m (http://apolloscape.auto/car_instance.html#to_metric_href)
\n\nAs the above arguments suggest, the threshold in this competition is far more than difficult...\nIn my opinion, currently to get \"good\" score you just need to focus on near cars and forget about everything that's 10m away since nobody would get it right...
", "messages": [ { "id": "679813", "postDate": "11/23/2019 12:48:29", "content": "Currently the distance threshold is set to have a minimum of 0.1m, which is too strict in my opinion, and that explains why the score in the leaderboard is so bad.
\n\nFor the comparison I'll make below, let me first introduce another metric that is also commonly used for 3d detection: the 3d iou(https://en.wikipedia.org/wiki/Jaccard_index). Normally, a standard passenger car has length of about 4m, so the 0.1m distance threshold in this competition translates into about (4-0.1)/(4+0.1)=0.95 3d iou.
\n\nHaving that number in mind, let me introduce some well-known 3d detection task as example:\nLyft: requires 3d iou of 0.5(https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/overview/evaluation)\nKITTI: requires 3d iou of 0.7 (http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d)\nnuscenes: requires distance threshold of min 4m (https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any)\nand finally, in the original apolloscape challenge: requires distance threshold of min 2.8m (http://apolloscape.auto/car_instance.html#to_metric_href)
\n\nAs the above arguments suggest, the threshold in this competition is far more than difficult...\nIn my opinion, currently to get \"good\" score you just need to focus on near cars and forget about everything that's 10m away since nobody would get it right...
", "rawMarkdown": "Currently the distance threshold is set to have a minimum of 0.1m, which is too strict in my opinion, and that explains why the score in the leaderboard is so bad.\n\nFor the comparison I'll make below, let me first introduce another metric that is also commonly used for 3d detection: the 3d iou(https://en.wikipedia.org/wiki/Jaccard_index). Normally, a standard passenger car has length of about 4m, so the 0.1m distance threshold in this competition translates into about (4-0.1)/(4+0.1)=0.95 3d iou.\n\nHaving that number in mind, let me introduce some well-known 3d detection task as example:\nLyft: requires 3d iou of 0.5(https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/overview/evaluation)\nKITTI: requires 3d iou of 0.7 (http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d)\nnuscenes: requires distance threshold of min 4m (https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any)\nand finally, in the original apolloscape challenge: requires distance threshold of min 2.8m (http://apolloscape.auto/car_instance.html#to_metric_href)\n\nAs the above arguments suggest, the threshold in this competition is far more than difficult...\nIn my opinion, currently to get \"good\" score you just need to focus on near cars and forget about everything that's 10m away since nobody would get it right...", "votes": null }, { "id": "687412", "postDate": "12/04/2019 10:29:47", "content": "I agree.\nOne can say that this is what the competitions hosts want and we cannot argue with that. If the company really needs <10cm precision, than this is a valid goal of the competition.\nBUT, it also requires angle to be <50 degrees. And that is huge, comparing to 10cm. If a car is 4m long, than turning it by 50 degrees will result in ~1.5 meters shift in some parts!
\n\n
I doubt this is what the competition hosts want. Also, the definition of \"what is the center of car\" is not very precise itself.
", "rawMarkdown": "I agree.\nOne can say that this is what the competitions hosts want and we cannot argue with that. If the company really needs <10cm precision, than this is a valid goal of the competition.\nBUT, it also requires angle to be <50 degrees. And that is huge, comparing to 10cm. If a car is 4m long, than turning it by 50 degrees will result in ~1.5 meters shift in some parts!\n\n\n\nI doubt this is what the competition hosts want. Also, the definition of \"what is the center of car\" is not very precise itself.", "votes": null }, { "id": "687944", "postDate": "12/05/2019 02:44:11", "content": "Maybe the distance threshold should tune with the true distance.
", "rawMarkdown": "Maybe the distance threshold should tune with the true distance.", "votes": null }, { "id": "691855", "postDate": "12/10/2019 15:09:58", "content": "I agree with you.
", "rawMarkdown": "I agree with you.", "votes": null }, { "id": "692313", "postDate": "12/11/2019 06:10:24", "content": "I doubt, that the actual metric compares the cm threshold. In local validation I am only able to reach about 1E-4 with a metric that strict. \nApparently, here something else is implemented and sadly nobody clarifies it ...
", "rawMarkdown": "I doubt, that the actual metric compares the cm threshold. In local validation I am only able to reach about 1E-4 with a metric that strict. \nApparently, here something else is implemented and sadly nobody clarifies it ...", "votes": null }, { "id": "692559", "postDate": "12/11/2019 13:06:51", "content": "Yeah with the current threshold scores like 1e-4 is totally expected...
", "rawMarkdown": "Yeah with the current threshold scores like 1e-4 is totally expected...", "votes": null } ], "comments": [ { "id": 687412, "author_name": "hocop1", "author_url": "", "post_date": "12/04/2019 10:29:47", "content": "I agree.\nOne can say that this is what the competitions hosts want and we cannot argue with that. If the company really needs <10cm precision, than this is a valid goal of the competition.\nBUT, it also requires angle to be <50 degrees. And that is huge, comparing to 10cm. If a car is 4m long, than turning it by 50 degrees will result in ~1.5 meters shift in some parts!
\n\n
I doubt this is what the competition hosts want. Also, the definition of \"what is the center of car\" is not very precise itself.
", "votes": null, "replies": [] }, { "id": 687944, "author_name": "dragonflying", "author_url": "", "post_date": "12/05/2019 02:44:11", "content": "Maybe the distance threshold should tune with the true distance.
", "votes": null, "replies": [] }, { "id": 691855, "author_name": "lisosia", "author_url": "", "post_date": "12/10/2019 15:09:58", "content": "I agree with you.
", "votes": null, "replies": [] }, { "id": 692313, "author_name": "ilu000", "author_url": "", "post_date": "12/11/2019 06:10:24", "content": "I doubt, that the actual metric compares the cm threshold. In local validation I am only able to reach about 1E-4 with a metric that strict. \nApparently, here something else is implemented and sadly nobody clarifies it ...
", "votes": null, "replies": [ { "id": 692559, "author_name": "kwea123", "author_url": "", "post_date": "12/11/2019 13:06:51", "content": "Yeah with the current threshold scores like 1e-4 is totally expected...
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "679813": "Currently the distance threshold is set to have a minimum of 0.1m, which is too strict in my opinion, and that explains why the score in the leaderboard is so bad.\n\nFor the comparison I'll make below, let me first introduce another metric that is also commonly used for 3d detection: the 3d iou(https://en.wikipedia.org/wiki/Jaccard_index). Normally, a standard passenger car has length of about 4m, so the 0.1m distance threshold in this competition translates into about (4-0.1)/(4+0.1)=0.95 3d iou.\n\nHaving that number in mind, let me introduce some well-known 3d detection task as example:\nLyft: requires 3d iou of 0.5(https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/overview/evaluation)\nKITTI: requires 3d iou of 0.7 (http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d)\nnuscenes: requires distance threshold of min 4m (https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any)\nand finally, in the original apolloscape challenge: requires distance threshold of min 2.8m (http://apolloscape.auto/car_instance.html#to_metric_href)\n\nAs the above arguments suggest, the threshold in this competition is far more than difficult...\nIn my opinion, currently to get \"good\" score you just need to focus on near cars and forget about everything that's 10m away since nobody would get it right...", "687412": "I agree.\nOne can say that this is what the competitions hosts want and we cannot argue with that. If the company really needs <10cm precision, than this is a valid goal of the competition.\nBUT, it also requires angle to be <50 degrees. And that is huge, comparing to 10cm. If a car is 4m long, than turning it by 50 degrees will result in ~1.5 meters shift in some parts!\n\n\n\nI doubt this is what the competition hosts want. Also, the definition of \"what is the center of car\" is not very precise itself.", "687944": "Maybe the distance threshold should tune with the true distance.", "691855": "I agree with you.", "692313": "I doubt, that the actual metric compares the cm threshold. In local validation I am only able to reach about 1E-4 with a metric that strict. \nApparently, here something else is implemented and sadly nobody clarifies it ...", "692559": "Yeah with the current threshold scores like 1e-4 is totally expected..." }, "source": "meta" }