Hi all - as noted earlier, we've updated the LB metric to include false negatives. You'll notice that scores might have dropped markedly; this is working as intended. Please let us know if you run into any trouble. Thanks again for all of the feedback!
", "messages": [ { "id": "686821", "postDate": "12/03/2019 15:28:49", "content": "Hi all - as noted earlier, we've updated the LB metric to include false negatives. You'll notice that scores might have dropped markedly; this is working as intended. Please let us know if you run into any trouble. Thanks again for all of the feedback!
", "rawMarkdown": "Hi all - as noted earlier, we've updated the LB metric to include false negatives. You'll notice that scores might have dropped markedly; this is working as intended. Please let us know if you run into any trouble. Thanks again for all of the feedback!", "votes": null }, { "id": "686828", "postDate": "12/03/2019 15:42:08", "content": "Good job！👍
", "rawMarkdown": "Good job！👍", "votes": null }, { "id": "686842", "postDate": "12/03/2019 15:56:01", "content": "Thanks @philculliton, the scores are much more inline with kitti and nuscenes monocular benchmarks now.
", "rawMarkdown": "Thanks @philculliton, the scores are much more inline with kitti and nuscenes monocular benchmarks now.", "votes": null }, { "id": "687101", "postDate": "12/03/2019 23:35:16", "content": "Great. Many thanks. You changed some unknown formula to another unknown formula. \nWe still have no idea how it works. No, the \"explanation\" on the evaluation page does not help. Does not help even you, as we can see.\nPlease, provide the evaluation code.
", "rawMarkdown": "Great. Many thanks. You changed some unknown formula to another unknown formula. \nWe still have no idea how it works. No, the \"explanation\" on the evaluation page does not help. Does not help even you, as we can see.\n**Please, provide the evaluation code**.", "votes": null }, { "id": "687112", "postDate": "12/04/2019 00:02:47", "content": "The distance threshold is too strict... This score is more and more meaningless now... It's just ranking randomness??? Please consider my discussion https://www.kaggle.com/c/pku-autonomous-driving/discussion/118677
", "rawMarkdown": "The distance threshold is too strict... This score is more and more meaningless now... It's just ranking randomness??? Please consider my discussion https://www.kaggle.com/c/pku-autonomous-driving/discussion/118677", "votes": null }, { "id": "687139", "postDate": "12/04/2019 01:07:16", "content": "Thanks for update, but it's more important to know the metric, or what we need to optimize. Please disclose it with code. Do we need to submit every time to know the progress??
", "rawMarkdown": "Thanks for update, but it's more important to know the metric, or what we need to optimize. Please disclose it with code. Do we need to submit every time to know the progress??", "votes": null }, { "id": "687163", "postDate": "12/04/2019 02:06:52", "content": "Please provide the evaluation code. Many thanks
", "rawMarkdown": "Please provide the evaluation code. Many thanks", "votes": null }, { "id": "687258", "postDate": "12/04/2019 06:00:43", "content": "Thanks @philculliton \nNow the scoring makes sense :)
", "rawMarkdown": "Thanks @philculliton \nNow the scoring makes sense :)", "votes": null }, { "id": "688096", "postDate": "12/05/2019 08:22:09", "content": "Please update the overview evaluation page, Thanks!
", "rawMarkdown": "Please update the overview evaluation page, Thanks!", "votes": null }, { "id": "689421", "postDate": "12/06/2019 23:03:37", "content": "@philculliton could you confirm that the angle threshold are in degrees but not in radians?
\n\nIn the description of the evaluation page, it says the angle are in radian, but in the java code snippet, we can see a rad2deg transformation as last step. Also, 50 radians does not make sense. My guess is that the description is a typo and the angle threshold is in degrees, but it would be great if you can confirm it.
", "rawMarkdown": "philculliton could you confirm that the angle threshold are in degrees but not in radians?\n\nIn the description of the evaluation page, it says the angle are in radian, but in the java code snippet, we can see a rad2deg transformation as last step. Also, 50 radians does not make sense. My guess is that the description is a typo and the angle threshold is in degrees, but it would be great if you can confirm it.", "votes": null }, { "id": "694550", "postDate": "12/13/2019 19:12:28", "content": "Thanks
", "rawMarkdown": "Thanks", "votes": null }, { "id": "694865", "postDate": "12/14/2019 08:13:23", "content": "@philculliton hi Phil,
\n\nConfidence score doesn't seem to work as discussed here.
\n\nI changed confidence to 1-confidence and got exact same LB score.\nIs this expected behavior?
Please give us more detail information about metrics!
", "rawMarkdown": "philculliton hi Phil,\n\nConfidence score doesn't seem to work as discussed [here](https://www.kaggle.com/c/pku-autonomous-driving/discussion/120580).\n\nI changed `confidence` to `1-confidence` and got exact same LB score.\nIs this expected behavior?\n\nPlease give us more detail information about metrics!", "votes": null }, { "id": "706147", "postDate": "12/30/2019 01:28:26", "content": "@philculliton , could you kindly give the participants some affirmation concerning the annotation when the test image is cropped as described in a more detail as in: https://www.kaggle.com/c/pku-autonomous-driving/discussion/123653\nMany thanks!
", "rawMarkdown": "philculliton , could you kindly give the participants some affirmation concerning the annotation when the test image is cropped as described in a more detail as in: https://www.kaggle.com/c/pku-autonomous-driving/discussion/123653\nMany thanks!", "votes": null }, { "id": "712760", "postDate": "01/07/2020 15:05:26", "content": "@philculliton Are you planning to answer the question on threshold strictness at https://www.kaggle.com/c/pku-autonomous-driving/discussion/118677 ?
\n\nWe cannot really expect to get any meaningful signal out of thresholding at a few centimeters in position for cars tens of meters away, while the orientation thresholds go all the way up to 50 degrees (although the description says radians).
\n\nIt would be a good idea to think these things through thoroughly and define meaningful eval measures in a transparent way. Overall it looks like this challenge needs some more polishing.
", "rawMarkdown": "philculliton Are you planning to answer the question on threshold strictness at https://www.kaggle.com/c/pku-autonomous-driving/discussion/118677 ?\n\nWe cannot really expect to get any meaningful signal out of thresholding at a few centimeters in position for cars tens of meters away, while the orientation thresholds go all the way up to 50 degrees (although the description says radians).\n\nIt would be a good idea to think these things through thoroughly and define meaningful eval measures in a transparent way. Overall it looks like this challenge needs some more polishing.", "votes": null }, { "id": "713177", "postDate": "01/08/2020 01:58:28", "content": "I think the distance metric is definitely relative. As many pointed out if it is absolute, then 0.10 m threshold at the loosest does not make sense. I recently bumped into the Apollo3DCar dataset, which the competition data is part of, and in it it used 0.01:0.01:0.10 as the relative distance threshold.
\n\nhttps://arxiv.org/pdf/1811.12222.pdf
", "rawMarkdown": "I think the distance metric is definitely relative. As many pointed out if it is absolute, then 0.10 m threshold at the loosest does not make sense. I recently bumped into the Apollo3DCar dataset, which the competition data is part of, and in it it used 0.01:0.01:0.10 as the relative distance threshold. \n\nhttps://arxiv.org/pdf/1811.12222.pdf", "votes": null } ], "comments": [ { "id": 686828, "author_name": "diegojohnson", "author_url": "", "post_date": "12/03/2019 15:42:08", "content": "Good job！👍
", "votes": null, "replies": [] }, { "id": 686842, "author_name": "jackvial", "author_url": "", "post_date": "12/03/2019 15:56:01", "content": "Thanks @philculliton, the scores are much more inline with kitti and nuscenes monocular benchmarks now.
", "votes": null, "replies": [] }, { "id": 687101, "author_name": "yhn112", "author_url": "", "post_date": "12/03/2019 23:35:16", "content": "Great. Many thanks. You changed some unknown formula to another unknown formula. \nWe still have no idea how it works. No, the \"explanation\" on the evaluation page does not help. Does not help even you, as we can see.\nPlease, provide the evaluation code.
", "votes": null, "replies": [] }, { "id": 687112, "author_name": "kwea123", "author_url": "", "post_date": "12/04/2019 00:02:47", "content": "The distance threshold is too strict... This score is more and more meaningless now... It's just ranking randomness??? Please consider my discussion https://www.kaggle.com/c/pku-autonomous-driving/discussion/118677
", "votes": null, "replies": [] }, { "id": 687139, "author_name": "bamps53", "author_url": "", "post_date": "12/04/2019 01:07:16", "content": "Thanks for update, but it's more important to know the metric, or what we need to optimize. Please disclose it with code. Do we need to submit every time to know the progress??
", "votes": null, "replies": [] }, { "id": 687163, "author_name": "scy232119", "author_url": "", "post_date": "12/04/2019 02:06:52", "content": "Please provide the evaluation code. Many thanks
", "votes": null, "replies": [] }, { "id": 687258, "author_name": "ilu000", "author_url": "", "post_date": "12/04/2019 06:00:43", "content": "Thanks @philculliton \nNow the scoring makes sense :)
", "votes": null, "replies": [] }, { "id": 688096, "author_name": "okkband", "author_url": "", "post_date": "12/05/2019 08:22:09", "content": "Please update the overview evaluation page, Thanks!
", "votes": null, "replies": [] }, { "id": 689421, "author_name": "sigma21", "author_url": "", "post_date": "12/06/2019 23:03:37", "content": "@philculliton could you confirm that the angle threshold are in degrees but not in radians?
\n\nIn the description of the evaluation page, it says the angle are in radian, but in the java code snippet, we can see a rad2deg transformation as last step. Also, 50 radians does not make sense. My guess is that the description is a typo and the angle threshold is in degrees, but it would be great if you can confirm it.
", "votes": null, "replies": [] }, { "id": 694550, "author_name": "hatemamine", "author_url": "", "post_date": "12/13/2019 19:12:28", "content": "Thanks
", "votes": null, "replies": [] }, { "id": 694865, "author_name": "its7171", "author_url": "", "post_date": "12/14/2019 08:13:23", "content": "@philculliton hi Phil,
\n\nConfidence score doesn't seem to work as discussed here.
\n\nI changed confidence to 1-confidence and got exact same LB score.\nIs this expected behavior?
Please give us more detail information about metrics!
", "votes": null, "replies": [] }, { "id": 706147, "author_name": "stevenwudi", "author_url": "", "post_date": "12/30/2019 01:28:26", "content": "@philculliton , could you kindly give the participants some affirmation concerning the annotation when the test image is cropped as described in a more detail as in: https://www.kaggle.com/c/pku-autonomous-driving/discussion/123653\nMany thanks!
", "votes": null, "replies": [] }, { "id": 712760, "author_name": "isarandi", "author_url": "", "post_date": "01/07/2020 15:05:26", "content": "@philculliton Are you planning to answer the question on threshold strictness at https://www.kaggle.com/c/pku-autonomous-driving/discussion/118677 ?
\n\nWe cannot really expect to get any meaningful signal out of thresholding at a few centimeters in position for cars tens of meters away, while the orientation thresholds go all the way up to 50 degrees (although the description says radians).
\n\nIt would be a good idea to think these things through thoroughly and define meaningful eval measures in a transparent way. Overall it looks like this challenge needs some more polishing.
", "votes": null, "replies": [] }, { "id": 713177, "author_name": "langechuan", "author_url": "", "post_date": "01/08/2020 01:58:28", "content": "I think the distance metric is definitely relative. As many pointed out if it is absolute, then 0.10 m threshold at the loosest does not make sense. I recently bumped into the Apollo3DCar dataset, which the competition data is part of, and in it it used 0.01:0.01:0.10 as the relative distance threshold.
\n\n", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "686821": "Hi all - as noted earlier, we've updated the LB metric to include false negatives. You'll notice that scores might have dropped markedly; this is working as intended. Please let us know if you run into any trouble. Thanks again for all of the feedback!", "686828": "Good job！👍", "686842": "Thanks @philculliton, the scores are much more inline with kitti and nuscenes monocular benchmarks now.", "687101": "Great. Many thanks. You changed some unknown formula to another unknown formula. \nWe still have no idea how it works. No, the \"explanation\" on the evaluation page does not help. Does not help even you, as we can see.\n**Please, provide the evaluation code**.", "687112": "The distance threshold is too strict... This score is more and more meaningless now... It's just ranking randomness??? Please consider my discussion https://www.kaggle.com/c/pku-autonomous-driving/discussion/118677", "687139": "Thanks for update, but it's more important to know the metric, or what we need to optimize. Please disclose it with code. Do we need to submit every time to know the progress??", "687163": "Please provide the evaluation code. Many thanks", "687258": "Thanks @philculliton \nNow the scoring makes sense :)", "688096": "Please update the overview evaluation page, Thanks!", "689421": "philculliton could you confirm that the angle threshold are in degrees but not in radians?\n\nIn the description of the evaluation page, it says the angle are in radian, but in the java code snippet, we can see a rad2deg transformation as last step. Also, 50 radians does not make sense. My guess is that the description is a typo and the angle threshold is in degrees, but it would be great if you can confirm it.", "694550": "Thanks", "694865": "philculliton hi Phil,\n\nConfidence score doesn't seem to work as discussed [here](https://www.kaggle.com/c/pku-autonomous-driving/discussion/120580).\n\nI changed `confidence` to `1-confidence` and got exact same LB score.\nIs this expected behavior?\n\nPlease give us more detail information about metrics!", "706147": "philculliton , could you kindly give the participants some affirmation concerning the annotation when the test image is cropped as described in a more detail as in: https://www.kaggle.com/c/pku-autonomous-driving/discussion/123653\nMany thanks!", "712760": "philculliton Are you planning to answer the question on threshold strictness at https://www.kaggle.com/c/pku-autonomous-driving/discussion/118677 ?\n\nWe cannot really expect to get any meaningful signal out of thresholding at a few centimeters in position for cars tens of meters away, while the orientation thresholds go all the way up to 50 degrees (although the description says radians).\n\nIt would be a good idea to think these things through thoroughly and define meaningful eval measures in a transparent way. Overall it looks like this challenge needs some more polishing.", "713177": "I think the distance metric is definitely relative. As many pointed out if it is absolute, then 0.10 m threshold at the loosest does not make sense. I recently bumped into the Apollo3DCar dataset, which the competition data is part of, and in it it used 0.01:0.01:0.10 as the relative distance threshold. \n\nhttps://arxiv.org/pdf/1811.12222.pdf" }, "source": "meta" }