{ "id": 304143, "title": "Difference Between AP and this Competition's Evaluation Metrics", "url": "/competitions/tensorflow-great-barrier-reef/discussion/304143", "author_name": "", "post_date": "2022-01-31T00:33:44.179607200Z", "votes": 2, "comment_count": 2, "views": 0, "content": "

In short

\n

AP and this competition's evaluation metrics differs as follows:

\n\n

Example

\n

Assume there are two models named A and B.
\nThe first model predicts TPs with higher confidence than that of FPs, whereas the second model predicts TPs with lower confidence than that of FPs (Table 1).

\n

The APs calculated for model A and B are 1.0 and 0.5 respectively (Table 2 right and Fig 1-2).
\nSince AP considers your model's prediction order, it evaluates higher score for model A than model B.

\n

On the contrary, the calculated F2 score for A and B are the same, i.e. F2=0.83 (Table 2 left).

\n

Thus, this competition metric does not take into account the order of prediction of your model. This means the high-mAP model doesn't necessarily performs better on the competition LB.

\n

(Strictly speaking, the higher mAP model tends to perform better on LB, but the model of enhanced mAP doesn't necessarily performs well on LB if the enhancement is about the order of your model's prediction.)

\n

Reference

\n\n
\n

Table 1.: The model's prediction. It only shows order of the prediction instead of confidence for convenience.
\n\"Screen-Shot-2022-01-31-at-8-59-33\"

\n

Table 2.: left: calculated AP for model A and B. right: calculated F2 for both model A and B.
\n\"Screen-Shot-2022-01-31-at-8-59-48\"

\n

Fig 1.: AP for model A visualized. The AP is area under the red line.

\n

\"AP-for-Model-A\"

\n

Fig 2.: AP for model B visualized. The AP is area under the red line.

\n

\"AP-for-Model-B\"

", "messages": [ { "id": "1669802", "postDate": "01/31/2022 00:33:44", "content": "

In short

\n

AP and this competition's evaluation metrics differs as follows:

\n\n

Example

\n

Assume there are two models named A and B.
\nThe first model predicts TPs with higher confidence than that of FPs, whereas the second model predicts TPs with lower confidence than that of FPs (Table 1).

\n

The APs calculated for model A and B are 1.0 and 0.5 respectively (Table 2 right and Fig 1-2).
\nSince AP considers your model's prediction order, it evaluates higher score for model A than model B.

\n

On the contrary, the calculated F2 score for A and B are the same, i.e. F2=0.83 (Table 2 left).

\n

Thus, this competition metric does not take into account the order of prediction of your model. This means the high-mAP model doesn't necessarily performs better on the competition LB.

\n

(Strictly speaking, the higher mAP model tends to perform better on LB, but the model of enhanced mAP doesn't necessarily performs well on LB if the enhancement is about the order of your model's prediction.)

\n

Reference

\n\n
\n

Table 1.: The model's prediction. It only shows order of the prediction instead of confidence for convenience.
\n\"Screen-Shot-2022-01-31-at-8-59-33\"

\n

Table 2.: left: calculated AP for model A and B. right: calculated F2 for both model A and B.
\n\"Screen-Shot-2022-01-31-at-8-59-48\"

\n

Fig 1.: AP for model A visualized. The AP is area under the red line.

\n

\"AP-for-Model-A\"

\n

Fig 2.: AP for model B visualized. The AP is area under the red line.

\n

\"AP-for-Model-B\"

", "rawMarkdown": "# In short \n\nAP and this competition's evaluation metrics differs as follows:\n\n* AP considers prediction order of your model, but this competition's metrics doesn't (at least in a limited way)\n* AP don't care FPs if the confidence of them are sufficiently less. AP only considers your model's prediction's order of confidence. This means AP is indifferent to confidence threshold.\n\n# Example\n\nAssume there are two models named A and B.\nThe first model predicts TPs with higher confidence than that of FPs, whereas the second model predicts TPs with lower confidence than that of FPs (Table 1). \n\nThe APs calculated for model A and B are 1.0 and 0.5 respectively (Table 2 right and Fig 1-2).\nSince AP considers your model's prediction order, it evaluates higher score for model A than model B.\n\nOn the contrary, the calculated F2 score for A and B are the same, i.e. F2=0.83 (Table 2 left).\n\nThus, this competition metric does not take into account the order of prediction of your model. This means the high-mAP model doesn't necessarily performs better on the competition LB.\n\n(Strictly speaking, the higher mAP model tends to perform better on LB, but the model of enhanced mAP doesn't necessarily performs well on LB if the enhancement is about the order of your model's prediction.)\n\n# Reference\n\n* [1] calculation of AP/mAP: https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173\n\n----\n\n**Table 1.**: The model's prediction. It only shows order of the prediction instead of confidence for convenience.\n\"Screen-Shot-2022-01-31-at-8-59-33\"\n\n**Table 2.**: left: calculated AP for model A and B. right: calculated F2 for both model A and B.\n\"Screen-Shot-2022-01-31-at-8-59-48\"\n\n**Fig 1.**: AP for model A visualized. The AP is area under the red line.\n\n\"AP-for-Model-A\"\n\n**Fig 2.**: AP for model B visualized. The AP is area under the red line.\n\n\"AP-for-Model-B\"", "votes": null }, { "id": "1669807", "postDate": "01/31/2022 00:47:37", "content": "

What I really want to prove is \"Does mAP well-reflects the performance of our model on LB?\".

\n

You know, mAP is convenient metric because we doesn't have to consider confidence threshold while training our model.

\n

But one question occurs: \"Does the model selected by mAP really performs well on the LB?\".
\nIf it doesn't, we have to consider better model-selection criterion instead of mAP.

\n

Adopting this competition metrics (based on F2) is one option. But then, we have to choose proper confidence threshold before we train the model. That is a nuisance.

", "rawMarkdown": "What I really want to prove is \"Does mAP well-reflects the performance of our model on LB?\".\n\nYou know, mAP is convenient metric because we doesn't have to consider confidence threshold while training our model.\n\nBut one question occurs: \"Does the model selected by mAP really performs well on the LB?\".\nIf it doesn't, we have to consider better model-selection criterion instead of mAP.\n\nAdopting this competition metrics (based on F2) is one option. But then, we have to choose proper confidence threshold before we train the model. That is a nuisance.", "votes": null }, { "id": "1669817", "postDate": "01/31/2022 01:10:16", "content": "

Candidates of better model-selection criterion

\n

Max value of F2 on the F2-confidence curve

\n

This is a high-return strategy because it tries to maximize the maximum return: maximizing F2 if your confidence-threshold is optimal. It's also a high-risk strategy because this metrics won't assure your model performs also well if the confidence threshold is not optimal.

\n

Area under the F2-confidence curve

\n

That strategy tries to maximize the expected value of F2 with respect to confidence threshold.
\nIf F2 is higher for all the confidence threshold, the model is expected to perform well on the LB if your confidence threshold is not optimal.
\nOn the contrary, it doesn't assure your model performs the best on the LB even if the confidence threshold is the optimal value.
\nThus is a low-risk, low-return strategy.

", "rawMarkdown": "# Candidates of better model-selection criterion\n\n## Max value of F2 on the F2-confidence curve\n\nThis is a high-return strategy because it tries to maximize the maximum return: maximizing F2 if your confidence-threshold is optimal. It's also a high-risk strategy because this metrics won't assure your model performs also well if the confidence threshold is not optimal.\n\n## Area under the F2-confidence curve\n\nThat strategy tries to maximize the expected value of F2 with respect to confidence threshold. \nIf F2 is higher for all the confidence threshold, the model is expected to perform well on the LB if your confidence threshold is not optimal.\nOn the contrary, it doesn't assure your model performs the best on the LB even if the confidence threshold is the optimal value.\nThus is a low-risk, low-return strategy.", "votes": null } ], "comments": [ { "id": 1669807, "author_name": "tatamikenn", "author_url": "", "post_date": "01/31/2022 00:47:37", "content": "

What I really want to prove is \"Does mAP well-reflects the performance of our model on LB?\".

\n

You know, mAP is convenient metric because we doesn't have to consider confidence threshold while training our model.

\n

But one question occurs: \"Does the model selected by mAP really performs well on the LB?\".
\nIf it doesn't, we have to consider better model-selection criterion instead of mAP.

\n

Adopting this competition metrics (based on F2) is one option. But then, we have to choose proper confidence threshold before we train the model. That is a nuisance.

", "votes": null, "replies": [ { "id": 1669817, "author_name": "tatamikenn", "author_url": "", "post_date": "01/31/2022 01:10:16", "content": "

Candidates of better model-selection criterion

\n

Max value of F2 on the F2-confidence curve

\n

This is a high-return strategy because it tries to maximize the maximum return: maximizing F2 if your confidence-threshold is optimal. It's also a high-risk strategy because this metrics won't assure your model performs also well if the confidence threshold is not optimal.

\n

Area under the F2-confidence curve

\n

That strategy tries to maximize the expected value of F2 with respect to confidence threshold.
\nIf F2 is higher for all the confidence threshold, the model is expected to perform well on the LB if your confidence threshold is not optimal.
\nOn the contrary, it doesn't assure your model performs the best on the LB even if the confidence threshold is the optimal value.
\nThus is a low-risk, low-return strategy.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1669802": "# In short \n\nAP and this competition's evaluation metrics differs as follows:\n\n* AP considers prediction order of your model, but this competition's metrics doesn't (at least in a limited way)\n* AP don't care FPs if the confidence of them are sufficiently less. AP only considers your model's prediction's order of confidence. This means AP is indifferent to confidence threshold.\n\n# Example\n\nAssume there are two models named A and B.\nThe first model predicts TPs with higher confidence than that of FPs, whereas the second model predicts TPs with lower confidence than that of FPs (Table 1). \n\nThe APs calculated for model A and B are 1.0 and 0.5 respectively (Table 2 right and Fig 1-2).\nSince AP considers your model's prediction order, it evaluates higher score for model A than model B.\n\nOn the contrary, the calculated F2 score for A and B are the same, i.e. F2=0.83 (Table 2 left).\n\nThus, this competition metric does not take into account the order of prediction of your model. This means the high-mAP model doesn't necessarily performs better on the competition LB.\n\n(Strictly speaking, the higher mAP model tends to perform better on LB, but the model of enhanced mAP doesn't necessarily performs well on LB if the enhancement is about the order of your model's prediction.)\n\n# Reference\n\n* [1] calculation of AP/mAP: https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173\n\n----\n\n**Table 1.**: The model's prediction. It only shows order of the prediction instead of confidence for convenience.\n\"Screen-Shot-2022-01-31-at-8-59-33\"\n\n**Table 2.**: left: calculated AP for model A and B. right: calculated F2 for both model A and B.\n\"Screen-Shot-2022-01-31-at-8-59-48\"\n\n**Fig 1.**: AP for model A visualized. The AP is area under the red line.\n\n\"AP-for-Model-A\"\n\n**Fig 2.**: AP for model B visualized. The AP is area under the red line.\n\n\"AP-for-Model-B\"", "1669807": "What I really want to prove is \"Does mAP well-reflects the performance of our model on LB?\".\n\nYou know, mAP is convenient metric because we doesn't have to consider confidence threshold while training our model.\n\nBut one question occurs: \"Does the model selected by mAP really performs well on the LB?\".\nIf it doesn't, we have to consider better model-selection criterion instead of mAP.\n\nAdopting this competition metrics (based on F2) is one option. But then, we have to choose proper confidence threshold before we train the model. That is a nuisance.", "1669817": "# Candidates of better model-selection criterion\n\n## Max value of F2 on the F2-confidence curve\n\nThis is a high-return strategy because it tries to maximize the maximum return: maximizing F2 if your confidence-threshold is optimal. It's also a high-risk strategy because this metrics won't assure your model performs also well if the confidence threshold is not optimal.\n\n## Area under the F2-confidence curve\n\nThat strategy tries to maximize the expected value of F2 with respect to confidence threshold. \nIf F2 is higher for all the confidence threshold, the model is expected to perform well on the LB if your confidence threshold is not optimal.\nOn the contrary, it doesn't assure your model performs the best on the LB even if the confidence threshold is the optimal value.\nThus is a low-risk, low-return strategy." }, "source": "meta" }