{ "id": 290083, "title": "f2 score implementation", "url": "/competitions/tensorflow-great-barrier-reef/discussion/290083", "author_name": "Camaro", "post_date": "2021-11-23T03:14:09.751000", "votes": 22, "comment_count": 7, "views": 0, "content": "

In this notebook I've implemented the competition metric f2 score.
\nhttps://www.kaggle.com/bamps53/competition-metric-implementation

\n

There is still something I couldn't figure out the detail of metric.
\n(ex. image-wise average or average over whole predictions)
\nIf you found something wrong, please let me know!

", "messages": [ { "id": 1592303, "postDate": "2021-11-23T03:14:09.750Z", "content": "

In this notebook I've implemented the competition metric f2 score.
\nhttps://www.kaggle.com/bamps53/competition-metric-implementation

\n

There is still something I couldn't figure out the detail of metric.
\n(ex. image-wise average or average over whole predictions)
\nIf you found something wrong, please let me know!

", "rawMarkdown": "In this notebook I've implemented the competition metric f2 score.\nhttps://www.kaggle.com/bamps53/competition-metric-implementation\n\nThere is still something I couldn't figure out the detail of metric.\n(ex. image-wise average or average over whole predictions)\nIf you found something wrong, please let me know!", "votes": 22 }, { "id": 1599483, "postDate": "2021-11-29T13:22:20.763Z", "content": "

Hi @bamps53 - I believe you have a nasty bug in the actual metric calculation:

\n

return (1+beta**2)*tp / ((1+beta**2)*tp + beta**2*(fn+fp))
\nshould be
\nreturn (1+beta**2)*tp / ((1+beta**2)*tp + beta**2*fn + fp)

\n

Thanks!

", "rawMarkdown": "Hi @bamps53 - I believe you have a nasty bug in the actual metric calculation:\n\n`return (1+beta**2)*tp / ((1+beta**2)*tp + beta**2*(fn+fp))`\nshould be\n`return (1+beta**2)*tp / ((1+beta**2)*tp + beta**2*fn + fp)`\n\nThanks!", "votes": 5, "replies": [ { "id": 1599525, "postDate": "2021-11-29T14:02:00.293Z", "content": "

@philippsinger Thanks for pointing out, what a stupid bug! I've fixed it.

", "rawMarkdown": "@philippsinger Thanks for pointing out, what a stupid bug! I've fixed it.", "votes": 2 } ] }, { "id": 1620013, "postDate": "2021-12-16T10:36:01.127Z", "content": "

Hi @bamps53 I think i found a bug in your code. In 'calc_is_correct_at_iou_th' function:

\n
for pred_bbox in pred_bboxes:\n        ious = calc_iou(gt_bboxes, pred_bbox[None, 1:])\n        max_iou = ious.max()\n        if max_iou > iou_th:\n            tp += 1\n            gt_bboxes = np.delete(gt_bboxes, ious.argmax(), axis=0)\n            pop_out+=1\n        else:\n            fp += 1\n        if len(gt_bboxes) == 0:\n            fp += len(pred_bboxes)\n            break\n
\n

I think you forget to remove the pred_bbox that is tp and that still counted in fp. Correct me if i wrong. Thanks

", "rawMarkdown": "Hi @bamps53 I think i found a bug in your code. In 'calc_is_correct_at_iou_th' function:\n\n```\nfor pred_bbox in pred_bboxes:\n ious = calc_iou(gt_bboxes, pred_bbox[None, 1:])\n max_iou = ious.max()\n if max_iou > iou_th:\n tp += 1\n gt_bboxes = np.delete(gt_bboxes, ious.argmax(), axis=0)\n pop_out+=1\n else:\n fp += 1\n if len(gt_bboxes) == 0:\n fp += len(pred_bboxes)\n break\n```\n\nI think you forget to remove the pred_bbox that is tp and that still counted in fp. Correct me if i wrong. Thanks", "votes": 4 }, { "id": 1599480, "postDate": "2021-11-29T13:19:19.783Z", "content": "

update: I've added correct metric. In my first notebook, it calculates image-wise metric but it might be not correct as competition metric is average over all frames.
\nhttps://www.kaggle.com/bamps53/competition-metric-implementation?scriptVersionId=81087805

", "rawMarkdown": "update: I've added correct metric. In my first notebook, it calculates image-wise metric but it might be not correct as competition metric is average over all frames.\nhttps://www.kaggle.com/bamps53/competition-metric-implementation?scriptVersionId=81087805", "votes": 1 }, { "id": 1649014, "postDate": "2022-01-13T22:20:41.247Z", "content": "

Hello, @bamps53. Thank you for your work! But I'm not sure about one element.

\n
\n

The final F2 Score is calculated as the mean of the F2 scores at each IoU threshold.

\n
\n

You use all tps, fps, fns to calculate one f2-score instead of calculate f2-score for each iou_th, if I'm not mistaken

", "rawMarkdown": "Hello, @bamps53. Thank you for your work! But I'm not sure about one element.\n> The final F2 Score is calculated as the mean of the F2 scores at each IoU threshold. \n\nYou use all tps, fps, fns to calculate one f2-score instead of calculate f2-score for each iou_th, if I'm not mistaken" }, { "id": 1593690, "postDate": "2021-11-24T07:02:40.987Z", "content": "

Great work its nice to see your work

", "rawMarkdown": "Great work its nice to see your work\n" }, { "id": 1598763, "postDate": "2021-11-28T19:44:24.993Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1599483, "author_name": "Psi", "author_url": "", "post_date": "2021-11-29T13:22:20.763000", "content": "

Hi @bamps53 - I believe you have a nasty bug in the actual metric calculation:

\n

return (1+beta**2)*tp / ((1+beta**2)*tp + beta**2*(fn+fp))
\nshould be
\nreturn (1+beta**2)*tp / ((1+beta**2)*tp + beta**2*fn + fp)

\n

Thanks!

", "votes": 5, "replies": [ { "id": 1599525, "author_name": "Camaro", "author_url": "", "post_date": "2021-11-29T14:02:00.293000", "content": "

@philippsinger Thanks for pointing out, what a stupid bug! I've fixed it.

", "votes": 2, "replies": [] } ] }, { "id": 1620013, "author_name": "Bùi Nhật Trường", "author_url": "", "post_date": "2021-12-16T10:36:01.127000", "content": "

Hi @bamps53 I think i found a bug in your code. In 'calc_is_correct_at_iou_th' function:

\n
for pred_bbox in pred_bboxes:\n        ious = calc_iou(gt_bboxes, pred_bbox[None, 1:])\n        max_iou = ious.max()\n        if max_iou > iou_th:\n            tp += 1\n            gt_bboxes = np.delete(gt_bboxes, ious.argmax(), axis=0)\n            pop_out+=1\n        else:\n            fp += 1\n        if len(gt_bboxes) == 0:\n            fp += len(pred_bboxes)\n            break\n
\n

I think you forget to remove the pred_bbox that is tp and that still counted in fp. Correct me if i wrong. Thanks

", "votes": 4, "replies": [] }, { "id": 1599480, "author_name": "Camaro", "author_url": "", "post_date": "2021-11-29T13:19:19.783000", "content": "

update: I've added correct metric. In my first notebook, it calculates image-wise metric but it might be not correct as competition metric is average over all frames.
\nhttps://www.kaggle.com/bamps53/competition-metric-implementation?scriptVersionId=81087805

", "votes": 1, "replies": [] }, { "id": 1649014, "author_name": "ji411", "author_url": "", "post_date": "2022-01-13T22:20:41.247000", "content": "

Hello, @bamps53. Thank you for your work! But I'm not sure about one element.

\n
\n

The final F2 Score is calculated as the mean of the F2 scores at each IoU threshold.

\n
\n

You use all tps, fps, fns to calculate one f2-score instead of calculate f2-score for each iou_th, if I'm not mistaken

", "votes": 0, "replies": [] }, { "id": 1593690, "author_name": "Dhinahar P", "author_url": "", "post_date": "2021-11-24T07:02:40.987000", "content": "

Great work its nice to see your work

", "votes": 0, "replies": [] }, { "id": 1598763, "author_name": "", "author_url": "", "post_date": "2021-11-28T19:44:24.993000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1592303": "In this notebook I've implemented the competition metric f2 score.\nhttps://www.kaggle.com/bamps53/competition-metric-implementation\n\nThere is still something I couldn't figure out the detail of metric.\n(ex. image-wise average or average over whole predictions)\nIf you found something wrong, please let me know!", "1599483": "Hi @bamps53 - I believe you have a nasty bug in the actual metric calculation:\n\n`return (1+beta**2)*tp / ((1+beta**2)*tp + beta**2*(fn+fp))`\nshould be\n`return (1+beta**2)*tp / ((1+beta**2)*tp + beta**2*fn + fp)`\n\nThanks!", "1620013": "Hi @bamps53 I think i found a bug in your code. In 'calc_is_correct_at_iou_th' function:\n\n```\nfor pred_bbox in pred_bboxes:\n ious = calc_iou(gt_bboxes, pred_bbox[None, 1:])\n max_iou = ious.max()\n if max_iou > iou_th:\n tp += 1\n gt_bboxes = np.delete(gt_bboxes, ious.argmax(), axis=0)\n pop_out+=1\n else:\n fp += 1\n if len(gt_bboxes) == 0:\n fp += len(pred_bboxes)\n break\n```\n\nI think you forget to remove the pred_bbox that is tp and that still counted in fp. Correct me if i wrong. Thanks", "1599480": "update: I've added correct metric. In my first notebook, it calculates image-wise metric but it might be not correct as competition metric is average over all frames.\nhttps://www.kaggle.com/bamps53/competition-metric-implementation?scriptVersionId=81087805", "1649014": "Hello, @bamps53. Thank you for your work! But I'm not sure about one element.\n> The final F2 Score is calculated as the mean of the F2 scores at each IoU threshold. \n\nYou use all tps, fps, fns to calculate one f2-score instead of calculate f2-score for each iou_th, if I'm not mistaken", "1593690": "Great work its nice to see your work\n", "1598763": "" } }