{ "id": 418383, "title": "Did someone manage to reproduce the competition metric?", "url": "/competitions/hubmap-hacking-the-human-vasculature/discussion/418383", "author_name": "", "post_date": "2023-06-20T09:46:38.159960900Z", "votes": 7, "comment_count": 6, "views": 0, "content": "

I can't quite understand how a confidence score is used in the metrics calculation.

", "messages": [ { "id": "2310346", "postDate": "06/20/2023 09:46:38", "content": "

I can't quite understand how a confidence score is used in the metrics calculation.

", "rawMarkdown": "I can't quite understand how a confidence score is used in the metrics calculation.", "votes": null }, { "id": "2310357", "postDate": "06/20/2023 09:59:45", "content": "

What I understood is that,

\n

Since there are multiple instances and there can be overlap of one instance of the prediction with the multiple instances in the ground truth, We sort the predicted instances in descending order and then calculate the IOU>0.6 as mentioned in the metric.

\n

So the instance with higher score will be evaluated first with the ground truth instance and then for the further instances in the prediction, it cannot overlap with the previously overlapped instance.

\n

Even though I could not find it clearly here

\n

But under the segmentation metric here they have given the three steps in which these steps will be executed in descending order of the scores.

", "rawMarkdown": "What I understood is that,\n\nSince there are multiple instances and there can be overlap of one instance of the prediction with the multiple instances in the ground truth, We sort the predicted instances in descending order and then calculate the IOU>0.6 as mentioned in the metric.\n\nSo the instance with higher score will be evaluated first with the ground truth instance and then for the further instances in the prediction, it cannot overlap with the previously overlapped instance.\n\nEven though I could not find it clearly [here](https://storage.googleapis.com/openimages/web/evaluation.html#instance_segmentation_eval)\n\nBut under the segmentation metric here they have given the three steps in which these steps will be executed in descending order of the scores.", "votes": null }, { "id": "2310366", "postDate": "06/20/2023 10:06:38", "content": "

https://www.v7labs.com/blog/mean-average-precision#:~:text=Mean%20Average%20Precision(mAP)%20is%20a%20metric%20used%20to%20evaluate,values%20from%200%20to%201

\n

I went through this one as mentioned by @maksimovka during an old discussion

", "rawMarkdown": "https://www.v7labs.com/blog/mean-average-precision#:~:text=Mean%20Average%20Precision(mAP)%20is%20a%20metric%20used%20to%20evaluate,values%20from%200%20to%201\n\nI went through this one as mentioned by @maksimovka during an old discussion", "votes": null }, { "id": "2310405", "postDate": "06/20/2023 10:40:19", "content": "

Does it (roughly) match your LB score?

", "rawMarkdown": "Does it (roughly) match your LB score?", "votes": null }, { "id": "2310474", "postDate": "06/20/2023 11:33:26", "content": "

yes it did roughly match

", "rawMarkdown": "yes it did roughly match", "votes": null }, { "id": "2310484", "postDate": "06/20/2023 11:57:25", "content": "

Ok, are you using a specific implementation? I am using the one from TorchMetrics, but it's way off.

", "rawMarkdown": "Ok, are you using a specific implementation? I am using the one from TorchMetrics, but it's way off.", "votes": null }, { "id": "2311109", "postDate": "06/20/2023 23:08:03", "content": "

You need iou to match fps tps, and confidences you need to build precision recall curve and calculate average precision from it ( so you iterate through confidences, and for each get pair of precision and recall, sort it, after you have the curve and now you can calculate AP)

", "rawMarkdown": "You need iou to match fps tps, and confidences you need to build precision recall curve and calculate average precision from it ( so you iterate through confidences, and for each get pair of precision and recall, sort it, after you have the curve and now you can calculate AP)", "votes": null } ], "comments": [ { "id": 2310357, "author_name": "vishakkbhat", "author_url": "", "post_date": "06/20/2023 09:59:45", "content": "

What I understood is that,

\n

Since there are multiple instances and there can be overlap of one instance of the prediction with the multiple instances in the ground truth, We sort the predicted instances in descending order and then calculate the IOU>0.6 as mentioned in the metric.

\n

So the instance with higher score will be evaluated first with the ground truth instance and then for the further instances in the prediction, it cannot overlap with the previously overlapped instance.

\n

Even though I could not find it clearly here

\n

But under the segmentation metric here they have given the three steps in which these steps will be executed in descending order of the scores.

", "votes": null, "replies": [] }, { "id": 2310366, "author_name": "vishakkbhat", "author_url": "", "post_date": "06/20/2023 10:06:38", "content": "

https://www.v7labs.com/blog/mean-average-precision#:~:text=Mean%20Average%20Precision(mAP)%20is%20a%20metric%20used%20to%20evaluate,values%20from%200%20to%201

\n

I went through this one as mentioned by @maksimovka during an old discussion

", "votes": null, "replies": [ { "id": 2310405, "author_name": "fnands", "author_url": "", "post_date": "06/20/2023 10:40:19", "content": "

Does it (roughly) match your LB score?

", "votes": null, "replies": [ { "id": 2310474, "author_name": "vishakkbhat", "author_url": "", "post_date": "06/20/2023 11:33:26", "content": "

yes it did roughly match

", "votes": null, "replies": [ { "id": 2310484, "author_name": "fnands", "author_url": "", "post_date": "06/20/2023 11:57:25", "content": "

Ok, are you using a specific implementation? I am using the one from TorchMetrics, but it's way off.

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2311109, "author_name": "maksimovka", "author_url": "", "post_date": "06/20/2023 23:08:03", "content": "

You need iou to match fps tps, and confidences you need to build precision recall curve and calculate average precision from it ( so you iterate through confidences, and for each get pair of precision and recall, sort it, after you have the curve and now you can calculate AP)

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2310346": "I can't quite understand how a confidence score is used in the metrics calculation.", "2310357": "What I understood is that,\n\nSince there are multiple instances and there can be overlap of one instance of the prediction with the multiple instances in the ground truth, We sort the predicted instances in descending order and then calculate the IOU>0.6 as mentioned in the metric.\n\nSo the instance with higher score will be evaluated first with the ground truth instance and then for the further instances in the prediction, it cannot overlap with the previously overlapped instance.\n\nEven though I could not find it clearly [here](https://storage.googleapis.com/openimages/web/evaluation.html#instance_segmentation_eval)\n\nBut under the segmentation metric here they have given the three steps in which these steps will be executed in descending order of the scores.", "2310366": "https://www.v7labs.com/blog/mean-average-precision#:~:text=Mean%20Average%20Precision(mAP)%20is%20a%20metric%20used%20to%20evaluate,values%20from%200%20to%201\n\nI went through this one as mentioned by @maksimovka during an old discussion", "2310405": "Does it (roughly) match your LB score?", "2310474": "yes it did roughly match", "2310484": "Ok, are you using a specific implementation? I am using the one from TorchMetrics, but it's way off.", "2311109": "You need iou to match fps tps, and confidences you need to build precision recall curve and calculate average precision from it ( so you iterate through confidences, and for each get pair of precision and recall, sort it, after you have the curve and now you can calculate AP)" }, "source": "meta" }