=> So I reimplement the code to compute mAP for this competition. Note that it only support single class at the moment. I use my Detectron2 model to make predictions as an example and I also compare with Detectron2 built-in CocoEvaluator
\n", "messages": [ { "id": "2316784", "postDate": "06/25/2023 08:26:06", "content": "=> So I reimplement the code to compute mAP for this competition. Note that it only support single class at the moment. I use my Detectron2 model to make predictions as an example and I also compare with Detectron2 built-in CocoEvaluator
\n", "rawMarkdown": "## Although mAP metric is popular, I think we still need to write a new notebook to compute CV score\n\n1. Most of the mAP calculators are integrated within particular frameworks such as detectron2, yolo, mmdet, etc... It might lack flexibility if we want to ensemble various models.\n2. Some standalone codes only compute box-mAP, not segm-mAP\n\n=> So I reimplement the code to compute mAP for this competition. Note that it only support single class at the moment. I use my Detectron2 model to make predictions as an example and I also compare with Detectron2 built-in CocoEvaluator\n\nNotebook: https://www.kaggle.com/code/namgalielei/hubmap-cv-score-map-calculator/notebook?scriptVersionId=134782419", "votes": null }, { "id": "2316852", "postDate": "06/25/2023 09:21:00", "content": "Good discussion!
\nIs the code public?
Yes, please follow the above link
", "rawMarkdown": "Yes, please follow the above link", "votes": null }, { "id": "2316881", "postDate": "06/25/2023 09:44:39", "content": "Actually I have forgot to make it public. Now it is
", "rawMarkdown": "Actually I have forgot to make it public. Now it is", "votes": null }, { "id": "2316943", "postDate": "06/25/2023 10:34:32", "content": "Thanks for sharing!
\nI can now see the code.
I didn't really understand the calculation of the evaluation indicators, so this is a great learning experience.
", "rawMarkdown": "Thanks for sharing!\nI can now see the code.\n\nI didn't really understand the calculation of the evaluation indicators, so this is a great learning experience.", "votes": null }, { "id": "2316979", "postDate": "06/25/2023 11:10:19", "content": "the difference between VOC and COCO map is as follows:
\nhttps://pyimagesearch.com/2022/05/02/mean-average-precision-map-using-the-coco-evaluator/
\nhttps://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation
\nhttps://github.com/rafaelpadilla/review_object_detection_metrics/blob/main/src/evaluators/pascal_voc_evaluator.py
VOC 2007:
\nVOC 2010:
\nCOCO:
\nmaybe this is why your results is difference from detectron2
", "rawMarkdown": "the difference between VOC and COCO map is as follows:\n\nhttps://pyimagesearch.com/2022/05/02/mean-average-precision-map-using-the-coco-evaluator/\nhttps://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation \nhttps://github.com/rafaelpadilla/review_object_detection_metrics/blob/main/src/evaluators/pascal_voc_evaluator.py \n \nVOC 2007: \n- 11-Point Interpolation\n\nVOC 2010:\n- However, from VOC 2010, the computation of AP changed. ...\n\nCOCO: \n- unlike PASCAL VOC, the COCO evaluator uses 101-point interpolated AP (i.e., it calculates the precision values at 101 recall levels [0:0.01:1]).\n\nmaybe this is why your results is difference from detectron2", "votes": null }, { "id": "2316986", "postDate": "06/25/2023 11:16:54", "content": "OpenImagesChallengeEvaluator
\nhttps://github.com/tensorflow/models/blob/d8bb449c0dffc97257560b43b4619c1001c20f15/research/object_detection/utils/object_detection_evaluation.py#L847
def compute_average_precision(precision, recall):
\n \"\"\"Compute Average Precision according to the definition in VOCdevkit.
https://storage.googleapis.com/openimages/web/evaluation.html
\nOpen Images Challenge object detection evaluation
\nThe challenge uses a variant of the standard PASCAL VOC 2010 mean Average Precision (mAP) at IoU > 0.5. …
Ok, let me try the code
", "rawMarkdown": "Ok, let me try the code", "votes": null }, { "id": "2346173", "postDate": "07/16/2023 05:16:38", "content": "Thank you for sharing great code!
\n1) mAP_calc.evaluate(threshods=[0.6])
\n2) mAP_calc.evaluate(thresholds=np.arange(0.6, 1.0, 0.05))
\nwhich metric is correct in this competition?
\nI optimized my code by using (1), but I revisit metric description page, I think (2) might be correct one.
\ndilation leads me into deep fog of confusion….
I think it should be thresholds=[0.6], according to competition's evaluation page
\nSegmentation calculated IoU a threshold of .\nThank you for the comment.
\nI'm very confusing by the description, it states that \"Submissions are evaluated by computing the Average Precision\" (=AP) but linked (described identical) description stated that \"Submissions are evaluated by computing mean Average Precision.\" (=mAP)
\nI think it is not identical.
\nif AP (1) is correct one, I need to find another reason of mismatching between Local validation and LB.
I understand mean computes over all classes, so only one class case, AP and mAP is the same. but I’ｍ still confusing the definition of AP….
", "rawMarkdown": "I understand mean computes over all classes, so only one class case, AP and mAP is the same. but I’ｍ still confusing the definition of AP....", "votes": null } ], "comments": [ { "id": 2316852, "author_name": "yosukeyama", "author_url": "", "post_date": "06/25/2023 09:21:00", "content": "Good discussion!
\nIs the code public?
Yes, please follow the above link
", "votes": null, "replies": [ { "id": 2316881, "author_name": "namgalielei", "author_url": "", "post_date": "06/25/2023 09:44:39", "content": "Actually I have forgot to make it public. Now it is
", "votes": null, "replies": [ { "id": 2316943, "author_name": "yosukeyama", "author_url": "", "post_date": "06/25/2023 10:34:32", "content": "Thanks for sharing!
\nI can now see the code.
I didn't really understand the calculation of the evaluation indicators, so this is a great learning experience.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2316979, "author_name": "hengck23", "author_url": "", "post_date": "06/25/2023 11:10:19", "content": "the difference between VOC and COCO map is as follows:
\nhttps://pyimagesearch.com/2022/05/02/mean-average-precision-map-using-the-coco-evaluator/
\nhttps://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation
\nhttps://github.com/rafaelpadilla/review_object_detection_metrics/blob/main/src/evaluators/pascal_voc_evaluator.py
VOC 2007:
\nVOC 2010:
\nCOCO:
\nmaybe this is why your results is difference from detectron2
", "votes": null, "replies": [ { "id": 2316986, "author_name": "hengck23", "author_url": "", "post_date": "06/25/2023 11:16:54", "content": "OpenImagesChallengeEvaluator
\nhttps://github.com/tensorflow/models/blob/d8bb449c0dffc97257560b43b4619c1001c20f15/research/object_detection/utils/object_detection_evaluation.py#L847
def compute_average_precision(precision, recall):
\n \"\"\"Compute Average Precision according to the definition in VOCdevkit.
https://storage.googleapis.com/openimages/web/evaluation.html
\nOpen Images Challenge object detection evaluation
\nThe challenge uses a variant of the standard PASCAL VOC 2010 mean Average Precision (mAP) at IoU > 0.5. …
Ok, let me try the code
", "votes": null, "replies": [] } ] }, { "id": 2346173, "author_name": "mkurokawa", "author_url": "", "post_date": "07/16/2023 05:16:38", "content": "Thank you for sharing great code!
\n1) mAP_calc.evaluate(threshods=[0.6])
\n2) mAP_calc.evaluate(thresholds=np.arange(0.6, 1.0, 0.05))
\nwhich metric is correct in this competition?
\nI optimized my code by using (1), but I revisit metric description page, I think (2) might be correct one.
\ndilation leads me into deep fog of confusion….
I think it should be thresholds=[0.6], according to competition's evaluation page
\nSegmentation calculated IoU a threshold of .\nThank you for the comment.
\nI'm very confusing by the description, it states that \"Submissions are evaluated by computing the Average Precision\" (=AP) but linked (described identical) description stated that \"Submissions are evaluated by computing mean Average Precision.\" (=mAP)
\nI think it is not identical.
\nif AP (1) is correct one, I need to find another reason of mismatching between Local validation and LB.
I understand mean computes over all classes, so only one class case, AP and mAP is the same. but I’ｍ still confusing the definition of AP….
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2316784": "## Although mAP metric is popular, I think we still need to write a new notebook to compute CV score\n\n1. Most of the mAP calculators are integrated within particular frameworks such as detectron2, yolo, mmdet, etc... It might lack flexibility if we want to ensemble various models.\n2. Some standalone codes only compute box-mAP, not segm-mAP\n\n=> So I reimplement the code to compute mAP for this competition. Note that it only support single class at the moment. I use my Detectron2 model to make predictions as an example and I also compare with Detectron2 built-in CocoEvaluator\n\nNotebook: https://www.kaggle.com/code/namgalielei/hubmap-cv-score-map-calculator/notebook?scriptVersionId=134782419", "2316852": "Good discussion!\nIs the code public?", "2316878": "Yes, please follow the above link", "2316881": "Actually I have forgot to make it public. Now it is", "2316943": "Thanks for sharing!\nI can now see the code.\n\nI didn't really understand the calculation of the evaluation indicators, so this is a great learning experience.", "2316979": "the difference between VOC and COCO map is as follows:\n\nhttps://pyimagesearch.com/2022/05/02/mean-average-precision-map-using-the-coco-evaluator/\nhttps://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation \nhttps://github.com/rafaelpadilla/review_object_detection_metrics/blob/main/src/evaluators/pascal_voc_evaluator.py \n \nVOC 2007: \n- 11-Point Interpolation\n\nVOC 2010:\n- However, from VOC 2010, the computation of AP changed. ...\n\nCOCO: \n- unlike PASCAL VOC, the COCO evaluator uses 101-point interpolated AP (i.e., it calculates the precision values at 101 recall levels [0:0.01:1]).\n\nmaybe this is why your results is difference from detectron2", "2316986": "OpenImagesChallengeEvaluator\nhttps://github.com/tensorflow/models/blob/d8bb449c0dffc97257560b43b4619c1001c20f15/research/object_detection/utils/object_detection_evaluation.py#L847\n\nhttps://github.com/tensorflow/models/blob/d8bb449c0dffc97257560b43b4619c1001c20f15/research/object_detection/utils/metrics.py#L72\n\ndef compute_average_precision(precision, recall):\n \"\"\"Compute Average Precision according to the definition in VOCdevkit.\n\nhttps://storage.googleapis.com/openimages/web/evaluation.html\nOpen Images Challenge object detection evaluation\nThe challenge uses a variant of the standard PASCAL VOC 2010 mean Average Precision (mAP) at IoU > 0.5. ...", "2317109": "Ok, let me try the code", "2346173": "Thank you for sharing great code!\n1) mAP_calc.evaluate(threshods=[0.6])\n2) mAP_calc.evaluate(thresholds=np.arange(0.6, 1.0, 0.05))\nwhich metric is correct in this competition?\nI optimized my code by using (1), but I revisit metric description page, I think (2) might be correct one.\ndilation leads me into deep fog of confusion....", "2346357": "I think it should be thresholds=[0.6], according to competition's evaluation page\n\n```\nSegmentation is calculated using IoU with a threshold of 0.6.\n```", "2346473": "Thank you for the comment. \nI'm very confusing by the [description](https://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature/overview/evaluation), it states that \"Submissions are evaluated by computing the Average Precision\" (=AP) but linked (described identical) [description](https://www.kaggle.com/c/open-images-2019-instance-segmentation/overview/evaluation) stated that \"Submissions are evaluated by computing **mean** Average Precision.\" (=mAP)\nI think it is not identical. \nif AP (1) is correct one, I need to find another reason of mismatching between Local validation and LB.", "2346814": "I understand mean computes over all classes, so only one class case, AP and mAP is the same. but I’ｍ still confusing the definition of AP...." }, "source": "meta" }