I copied the \"Metric implementation\" code from the \"competition metric implementation\" notebook and it works fine if I limit it to the most-likely predicted bbox, but it fails in practice for me, I believe when len(pred_bboxes) > len(gt_boxes). For instance:
pred_bboxes = np.array([\n [.981, 1017, 633, 54, 54],\n [.873, 547, 680, 66, 37],\n [.832, 562, 689, 50, 29],\n [.736, 549, 674, 53, 45],\n [.713, 612, 643, 49, 29],\n [.567, 609, 633, 54, 47]\n])\ngt_bboxes = np.array([[548, 664, 70, 55]])\nf2_score(gt_bboxes, pred_bboxes, True)\nfails. Digging in, it seems that gt_bboxes becomes 0-length in calls to ious = calc_iou(gt_bboxes, pred_bbox[None, 1:]) but I don't see anything obvious in the code that shortens gt_bboxes.
For instance, if you add print(bboxes_1) as the first line in calc_iou, you get:
[[548 664 70 55]]\n[[548 664 70 55]]\n[]\nI copied the \"Metric implementation\" code from the \"competition metric implementation\" notebook and it works fine if I limit it to the most-likely predicted bbox, but it fails in practice for me, I believe when len(pred_bboxes) > len(gt_boxes). For instance:
pred_bboxes = np.array([\n [.981, 1017, 633, 54, 54],\n [.873, 547, 680, 66, 37],\n [.832, 562, 689, 50, 29],\n [.736, 549, 674, 53, 45],\n [.713, 612, 643, 49, 29],\n [.567, 609, 633, 54, 47]\n])\ngt_bboxes = np.array([[548, 664, 70, 55]])\nf2_score(gt_bboxes, pred_bboxes, True)\nfails. Digging in, it seems that gt_bboxes becomes 0-length in calls to ious = calc_iou(gt_bboxes, pred_bbox[None, 1:]) but I don't see anything obvious in the code that shortens gt_bboxes.
For instance, if you add print(bboxes_1) as the first line in calc_iou, you get:
[[548 664 70 55]]\n[[548 664 70 55]]\n[]\nI think you are referring @bamps53 notebook? The first column of pred_bboxes should be probability. [p, x1, y1, x2,y2]. Bassiclypred_bboxes` get sorted by probability and then the score is calculated. Check out this:
In your submission, you are also asked to provide a confidence level for each bounding box. Bounding boxes are evaluated in order of their confidence levels. This means that bounding boxes with higher confidence will be checked first for matches against solutions, which determines what boxes are considered true and false positives.
Thanks for pointing out. I'll check it out later;)
\nBut it's better to mention me directly or raise in the notebook so that I can respond quickly!
\n(I found this because DrHB pointed me:))
\n\nThe first column of pred_bboxes should be probability. [p, x1, y1, x2,y2]
\n
I think the shape of pred_bboxes is correct. I have, for instance, [.981, 1017, 633, 54, 54] -- 98% confidence of a box at [1017,633] of size [54,54]. Isn't that right?
Sorry about that. I think the issue is if a true positive zeroes out the gt_bboxes but the loop doesn't end (because more pred_bboxes), the call to calc_iou is invalid :
for pred_bbox in pred_bboxes:\n ious = calc_iou(gt_bboxes, pred_bbox[None, 1:]) # If `len(gt_bboxes) == 0` this throws\n max_iou = ious.max()\n if max_iou > iou_th:\n tp += 1\n gt_bboxes = np.delete(gt_bboxes, ious.argmax(), axis=0) # This can zero out `gt_bboxes`\n else:\n fp += 1\nIf I understand the logic correctly, you could either alter calc_iou to return 0.0 if len(gt_boxes) == 0 or test at the top of this loop, add len(pred_bboxes) - len(gt_bboxes_original) to fp, and break out of the loop.
that’s correct, my bad i did not notice :)
", "rawMarkdown": "that’s correct, my bad i did not notice :)", "votes": null }, { "id": "1599477", "postDate": "11/29/2021 13:17:52", "content": "@lobrien I've fixed it and also added correct metric. In my first notebook, it calculate image-wise metric but it might be not correct as competition metric is average over all frames. Please check it out!
\nhttps://www.kaggle.com/bamps53/competition-metric-implementation?scriptVersionId=81087805
I think you are referring @bamps53 notebook? The first column of pred_bboxes should be probability. [p, x1, y1, x2,y2]. Bassiclypred_bboxes` get sorted by probability and then the score is calculated. Check out this:
In your submission, you are also asked to provide a confidence level for each bounding box. Bounding boxes are evaluated in order of their confidence levels. This means that bounding boxes with higher confidence will be checked first for matches against solutions, which determines what boxes are considered true and false positives.
\n\nThe first column of pred_bboxes should be probability. [p, x1, y1, x2,y2]
\n
I think the shape of pred_bboxes is correct. I have, for instance, [.981, 1017, 633, 54, 54] -- 98% confidence of a box at [1017,633] of size [54,54]. Isn't that right?
that’s correct, my bad i did not notice :)
", "votes": null, "replies": [] } ] }, { "id": 1598950, "author_name": "bamps53", "author_url": "", "post_date": "11/29/2021 02:58:33", "content": "Thanks for pointing out. I'll check it out later;)
\nBut it's better to mention me directly or raise in the notebook so that I can respond quickly!
\n(I found this because DrHB pointed me:))
Sorry about that. I think the issue is if a true positive zeroes out the gt_bboxes but the loop doesn't end (because more pred_bboxes), the call to calc_iou is invalid :
for pred_bbox in pred_bboxes:\n ious = calc_iou(gt_bboxes, pred_bbox[None, 1:]) # If `len(gt_bboxes) == 0` this throws\n max_iou = ious.max()\n if max_iou > iou_th:\n tp += 1\n gt_bboxes = np.delete(gt_bboxes, ious.argmax(), axis=0) # This can zero out `gt_bboxes`\n else:\n fp += 1\nIf I understand the logic correctly, you could either alter calc_iou to return 0.0 if len(gt_boxes) == 0 or test at the top of this loop, add len(pred_bboxes) - len(gt_bboxes_original) to fp, and break out of the loop.
@lobrien I've fixed it and also added correct metric. In my first notebook, it calculate image-wise metric but it might be not correct as competition metric is average over all frames. Please check it out!
\nhttps://www.kaggle.com/bamps53/competition-metric-implementation?scriptVersionId=81087805