{ "id": 414877, "title": "Confidence Prediction in evaluation score", "url": "/competitions/hubmap-hacking-the-human-vasculature/discussion/414877", "author_name": "", "post_date": "2023-06-03T19:40:13.259716900Z", "votes": 5, "comment_count": 5, "views": 0, "content": "

I have read the open images evaluation metric but still can't figure how would box prediction confidence in the prediction string impact the overall score. If you have any information in this regard please advise.

", "messages": [ { "id": "2286807", "postDate": "06/03/2023 19:40:13", "content": "

I have read the open images evaluation metric but still can't figure how would box prediction confidence in the prediction string impact the overall score. If you have any information in this regard please advise.

", "rawMarkdown": "I have read the open images evaluation metric but still can't figure how would box prediction confidence in the prediction string impact the overall score. If you have any information in this regard please advise.", "votes": null }, { "id": "2287368", "postDate": "06/04/2023 11:35:03", "content": "

pls take a look at this explanation of AP calculation (open images metric in general is the same but with some small changes) - https://www.v7labs.com/blog/mean-average-precision#:~:text=Mean%20Average%20Precision(mAP)%20is%20a%20metric%20used%20to%20evaluate,values%20from%200%20to%201.

", "rawMarkdown": "pls take a look at this explanation of AP calculation (open images metric in general is the same but with some small changes) - https://www.v7labs.com/blog/mean-average-precision#:~:text=Mean%20Average%20Precision(mAP)%20is%20a%20metric%20used%20to%20evaluate,values%20from%200%20to%201.", "votes": null }, { "id": "2287409", "postDate": "06/04/2023 12:28:59", "content": "

I like to think of the basic case below for some intuition when you are doing AP calculation. There are a few stages.

\n
\n
    \n

  1. Choose an IoU (you will perhaps choose a number of them, then you repeat the calculations below and average). Let us suppose it is some value 0<I<1. This is often denoted as mAP@I.

    2. \n

  2. Plot precision-recall graph.

    3. \n
\n

Let us begin in the case without any confidence or IoU.

\n

--- The case without confidence or IoU ---
\nSuppose you had no confidence attached. Your model did the following:
\nInputs: an image, whose ground truth has 10 blood vessels.
\nOutputs:

\n\n

The key point:
\nWe have to say what it means for a polygon to be \"(un)successful prediction\". Let us suppose (un)successful means your prediction polygon intersection has (no) an intersection with the of some blood vessel in ground truth. We will see later, we use the confidence values to define what is successful. I'm sure you know the term for this is just True Positive.

\n

For example: we may deduce

\n\n

Then:
\nPrecision=TP/(TP+FP)=5/(5+4) ~56%
\nRecall =TP/(TP+FN)= 5/10 =50%

\n

Note one can think of Recall as
\n number of predictions/number of vessels in the ground truth.

\n
\n

--- The case with confidence and IoU threshold:=I ----
\nIn this case you consider the confidence attached. Then we plot Precision-recall(PR) as a *function of choices of confidence threshold. *

\n

Note how I italicized \"successfully, unsuccessfully\" in the case without confidence.
\nHere are your
\nOutputs:

\n\n

Now suppose we have the following IoUs (one way we can define this is say, for each predicted polygon, pick the largest IoU with a polygon of blood vessel in the ground truth) (0.8,0.7,0.6,0.5,0.5,0.9,0.1,0.5,0.1)

\n

For a choice of confidence threshold, C, a prediction is considered
\nTP :: confidence >C and IoU >= I
\nFP :: confidence >C but IoU< I

\n

Now we can compute PR for each choice of confidence.
\nPR for 0.9: (TP,FP)=(1,0): Precision=100%, Recall = 10%
\nPR for 0.1:(TP,FP,FN)= (7,2) : Precision ~ 78%, Recall ~ 70%

\n

The last step is to get an \"averaged\" value by considering a weighted sum. (Other ways to encode all of this data could be F1 and AUC(area under curve), but AP seems to be the convention these days.)

\n
\n

Observe if the confidence threshold gets smaller; generally, your prediction precision decreases (since you make more predictions), and your recall increases (you potentially detect more predictions)

\n

Hope this helps!

", "rawMarkdown": "I like to think of the basic case below for some intuition when you are doing AP calculation. There are a few stages. \n\n--- \n1. Choose an IoU (you will perhaps choose a number of them, then you repeat the calculations below and average). Let us suppose it is some value 0C and IoU >= I\nFP :: confidence >C but IoU< I \n\nNow we can compute PR for each choice of confidence. \nPR for 0.9: (TP,FP)=(1,0): Precision=100%, Recall = 10%\nPR for 0.1:(TP,FP,FN)= (7,2) : Precision ~ 78%, Recall ~ 70%\n\nThe last step is to get an \"averaged\" value by considering a weighted sum. (Other ways to encode all of this data could be F1 and AUC(area under curve), but AP seems to be the convention these days.)\n \n--- \nObserve if the confidence threshold gets smaller; generally, your prediction precision decreases (since you make more predictions), and your recall increases (you potentially detect more predictions)\n\nHope this helps!", "votes": null }, { "id": "2293714", "postDate": "06/09/2023 12:27:50", "content": "

@eatalittlel Thank you so much, so what you mean is confidence is incorporated the same way iou threshold is? or this is just some sort of analogy?

", "rawMarkdown": "eatalittlel Thank you so much, so what you mean is confidence is incorporated the same way iou threshold is? or this is just some sort of analogy?", "votes": null }, { "id": "2293881", "postDate": "06/09/2023 14:56:40", "content": "

Yes! It is not an analogy, but is exactly what IoU and confidence are doing: they are two types of thresholds for saying what makes a classification successful or unsuccessful. (Hence allowing one to compute recall and precision)

", "rawMarkdown": "Yes! It is not an analogy, but is exactly what IoU and confidence are doing: they are two types of thresholds for saying what makes a classification successful or unsuccessful. (Hence allowing one to compute recall and precision)", "votes": null }, { "id": "2294026", "postDate": "06/09/2023 17:03:15", "content": "

I found this article helpful: https://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation

", "rawMarkdown": "I found this article helpful: [https://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation](url)", "votes": null } ], "comments": [ { "id": 2287368, "author_name": "maksimovka", "author_url": "", "post_date": "06/04/2023 11:35:03", "content": "

pls take a look at this explanation of AP calculation (open images metric in general is the same but with some small changes) - https://www.v7labs.com/blog/mean-average-precision#:~:text=Mean%20Average%20Precision(mAP)%20is%20a%20metric%20used%20to%20evaluate,values%20from%200%20to%201.

", "votes": null, "replies": [] }, { "id": 2287409, "author_name": "eatalittlel", "author_url": "", "post_date": "06/04/2023 12:28:59", "content": "

I like to think of the basic case below for some intuition when you are doing AP calculation. There are a few stages.

\n
\n
    \n

  1. Choose an IoU (you will perhaps choose a number of them, then you repeat the calculations below and average). Let us suppose it is some value 0<I<1. This is often denoted as mAP@I.

    2. \n

  2. Plot precision-recall graph.

    3. \n
\n

Let us begin in the case without any confidence or IoU.

\n

--- The case without confidence or IoU ---
\nSuppose you had no confidence attached. Your model did the following:
\nInputs: an image, whose ground truth has 10 blood vessels.
\nOutputs:

\n\n

The key point:
\nWe have to say what it means for a polygon to be \"(un)successful prediction\". Let us suppose (un)successful means your prediction polygon intersection has (no) an intersection with the of some blood vessel in ground truth. We will see later, we use the confidence values to define what is successful. I'm sure you know the term for this is just True Positive.

\n

For example: we may deduce

\n\n

Then:
\nPrecision=TP/(TP+FP)=5/(5+4) ~56%
\nRecall =TP/(TP+FN)= 5/10 =50%

\n

Note one can think of Recall as
\n number of predictions/number of vessels in the ground truth.

\n
\n

--- The case with confidence and IoU threshold:=I ----
\nIn this case you consider the confidence attached. Then we plot Precision-recall(PR) as a *function of choices of confidence threshold. *

\n

Note how I italicized \"successfully, unsuccessfully\" in the case without confidence.
\nHere are your
\nOutputs:

\n\n

Now suppose we have the following IoUs (one way we can define this is say, for each predicted polygon, pick the largest IoU with a polygon of blood vessel in the ground truth) (0.8,0.7,0.6,0.5,0.5,0.9,0.1,0.5,0.1)

\n

For a choice of confidence threshold, C, a prediction is considered
\nTP :: confidence >C and IoU >= I
\nFP :: confidence >C but IoU< I

\n

Now we can compute PR for each choice of confidence.
\nPR for 0.9: (TP,FP)=(1,0): Precision=100%, Recall = 10%
\nPR for 0.1:(TP,FP,FN)= (7,2) : Precision ~ 78%, Recall ~ 70%

\n

The last step is to get an \"averaged\" value by considering a weighted sum. (Other ways to encode all of this data could be F1 and AUC(area under curve), but AP seems to be the convention these days.)

\n
\n

Observe if the confidence threshold gets smaller; generally, your prediction precision decreases (since you make more predictions), and your recall increases (you potentially detect more predictions)

\n

Hope this helps!

", "votes": null, "replies": [ { "id": 2293714, "author_name": "mhmdsab", "author_url": "", "post_date": "06/09/2023 12:27:50", "content": "

@eatalittlel Thank you so much, so what you mean is confidence is incorporated the same way iou threshold is? or this is just some sort of analogy?

", "votes": null, "replies": [ { "id": 2293881, "author_name": "eatalittlel", "author_url": "", "post_date": "06/09/2023 14:56:40", "content": "

Yes! It is not an analogy, but is exactly what IoU and confidence are doing: they are two types of thresholds for saying what makes a classification successful or unsuccessful. (Hence allowing one to compute recall and precision)

", "votes": null, "replies": [] } ] } ] }, { "id": 2294026, "author_name": "tsobolev", "author_url": "", "post_date": "06/09/2023 17:03:15", "content": "

I found this article helpful: https://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2286807": "I have read the open images evaluation metric but still can't figure how would box prediction confidence in the prediction string impact the overall score. If you have any information in this regard please advise.", "2287368": "pls take a look at this explanation of AP calculation (open images metric in general is the same but with some small changes) - https://www.v7labs.com/blog/mean-average-precision#:~:text=Mean%20Average%20Precision(mAP)%20is%20a%20metric%20used%20to%20evaluate,values%20from%200%20to%201.", "2287409": "I like to think of the basic case below for some intuition when you are doing AP calculation. There are a few stages. \n\n--- \n1. Choose an IoU (you will perhaps choose a number of them, then you repeat the calculations below and average). Let us suppose it is some value 0C and IoU >= I\nFP :: confidence >C but IoU< I \n\nNow we can compute PR for each choice of confidence. \nPR for 0.9: (TP,FP)=(1,0): Precision=100%, Recall = 10%\nPR for 0.1:(TP,FP,FN)= (7,2) : Precision ~ 78%, Recall ~ 70%\n\nThe last step is to get an \"averaged\" value by considering a weighted sum. (Other ways to encode all of this data could be F1 and AUC(area under curve), but AP seems to be the convention these days.)\n \n--- \nObserve if the confidence threshold gets smaller; generally, your prediction precision decreases (since you make more predictions), and your recall increases (you potentially detect more predictions)\n\nHope this helps!", "2293714": "eatalittlel Thank you so much, so what you mean is confidence is incorporated the same way iou threshold is? or this is just some sort of analogy?", "2293881": "Yes! It is not an analogy, but is exactly what IoU and confidence are doing: they are two types of thresholds for saying what makes a classification successful or unsuccessful. (Hence allowing one to compute recall and precision)", "2294026": "I found this article helpful: [https://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation](url)" }, "source": "meta" }