This is slightly different from the metric in this competition, but since I wanted to read up more on MAP as a metric in general, I sieved through multiple articles like the one by Jonathon and here by Kaggler Tito.
\nOne part I cannot grasp immediately is the need to sort the confidence scores of the bounding box in descending order. What is the reason behind sorting here? Amongst the > 10 articles I read, only @its7171 notebook mentioned about sorting, but I cannot grasp it also.
\nIs it because we want to have a cumulative precision and recall based off the cut-off threshold by the confidence scores since we are discarding all predictions under a certain confidence score?
", "messages": [ { "id": "1736385", "postDate": "03/27/2022 08:37:39", "content": "This is slightly different from the metric in this competition, but since I wanted to read up more on MAP as a metric in general, I sieved through multiple articles like the one by Jonathon and here by Kaggler Tito.
\nOne part I cannot grasp immediately is the need to sort the confidence scores of the bounding box in descending order. What is the reason behind sorting here? Amongst the > 10 articles I read, only @its7171 notebook mentioned about sorting, but I cannot grasp it also.
\nIs it because we want to have a cumulative precision and recall based off the cut-off threshold by the confidence scores since we are discarding all predictions under a certain confidence score?
", "rawMarkdown": "This is slightly different from the metric in this competition, but since I wanted to read up more on MAP as a metric in general, I sieved through multiple articles like [the one by Jonathon](https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173) and [here by Kaggler Tito](https://www.kaggle.com/code/its7171/map-understanding-with-code-and-its-tips?scriptVersionId=54650292).\n\nOne part I cannot grasp immediately is the need to sort the confidence scores of the bounding box in descending order. What is the reason behind sorting here? Amongst the > 10 articles I read, only @its7171 notebook mentioned about sorting, but I cannot grasp it also.\n\nIs it because we want to have a cumulative precision and recall based off the cut-off threshold by the confidence scores since we are discarding all predictions under a certain confidence score?", "votes": null }, { "id": "1736396", "postDate": "03/27/2022 09:00:51", "content": "I feel the answer lies in the statement: 'The lower the confidence threshold, the higher the mAP we will always get', and your doubt is infact the answer to your question.
", "rawMarkdown": "I feel the answer lies in the statement: 'The lower the confidence threshold, the higher the mAP we will always get', and your doubt is infact the answer to your question.", "votes": null }, { "id": "1736419", "postDate": "03/27/2022 09:24:19", "content": "Thanks, this is kind of confusing for me since, amongst the 10+ articles I read, none actually mentioned the rationale of sorting. Tito's notebook mentioned though, but I cannot grasp it also, I read through a youtuber's source code on this, but they indeed sorted them (without explaining).
", "rawMarkdown": "Thanks, this is kind of confusing for me since, amongst the 10+ articles I read, none actually mentioned the rationale of sorting. Tito's notebook mentioned though, but I cannot grasp it also, I read through a youtuber's source code on this, but they indeed sorted them (without explaining).", "votes": null }, { "id": "1736602", "postDate": "03/27/2022 14:15:06", "content": "The metric MAP answers the questions,
\n\"how good is your prediction if you only guess 1 bounding box\"?,
\n\"how good is your prediction if you only guess 2 bounding boxes\"?
\n\"how good is your prediction if you only guess 3 bounding boxes\"?
\n\"how good is your prediction if you only guess 4 bounding boxes\"?
\netc etc etc
\nthen MAP takes the average of these questions
When evaluating \"how good is your prediction if you only guess X bound boxes\", then of course we would like to use our best X bounding boxes. Our best X bounding boxes are the X bounding boxes with best confidence. Therefore we sort by confidence before answering the above questions.
", "rawMarkdown": "The metric MAP answers the questions, \n\"how good is your prediction if you only guess 1 bounding box\"?, \n\"how good is your prediction if you only guess 2 bounding boxes\"?\n\"how good is your prediction if you only guess 3 bounding boxes\"?\n\"how good is your prediction if you only guess 4 bounding boxes\"?\netc etc etc\nthen MAP takes the average of these questions\n\nWhen evaluating \"how good is your prediction if you only guess X bound boxes\", then of course we would like to use our best X bounding boxes. Our best X bounding boxes are the X bounding boxes with best confidence. Therefore we sort by confidence before answering the above questions.", "votes": null }, { "id": "1736725", "postDate": "03/27/2022 16:45:47", "content": "It gives the precision value from the rank of outcomes we derive.
", "rawMarkdown": "It gives the precision value from the rank of outcomes we derive.", "votes": null }, { "id": "1737045", "postDate": "03/28/2022 04:27:39", "content": "Thanks Chris for the clear answer. In that case, are we able to ask
\n\"at each confidence level (threshold), what is the precision-recall score of the predictions of all bounding boxes while discarding off those below the threshold, then average them over all thresholds?\"
", "rawMarkdown": "Thanks Chris for the clear answer. In that case, are we able to ask\n\n\"at each confidence level (threshold), what is the precision-recall score of the predictions of all bounding boxes while discarding off those below the threshold, then average them over all thresholds?\"", "votes": null }, { "id": "1737049", "postDate": "03/28/2022 04:33:20", "content": "Yes basically yes. The exact statement is \"\"at each confidence level (threshold), if recall increases from previous confidence level, then include this precision and average all previous included precisions\". But what you said, is basically correct.
", "rawMarkdown": "Yes basically yes. The exact statement is \"\"at each confidence level (threshold), if recall increases from previous confidence level, then include this precision and average all previous included precisions\". But what you said, is basically correct.", "votes": null }, { "id": "1737147", "postDate": "03/28/2022 06:30:57", "content": "@cdeotte Thanks a lot for this, it clears some air. I will go and read the source code again and update my understanding…
", "rawMarkdown": "cdeotte Thanks a lot for this, it clears some air. I will go and read the source code again and update my understanding...", "votes": null } ], "comments": [ { "id": 1736396, "author_name": "gianetan", "author_url": "", "post_date": "03/27/2022 09:00:51", "content": "I feel the answer lies in the statement: 'The lower the confidence threshold, the higher the mAP we will always get', and your doubt is infact the answer to your question.
", "votes": null, "replies": [ { "id": 1736419, "author_name": "reighns", "author_url": "", "post_date": "03/27/2022 09:24:19", "content": "Thanks, this is kind of confusing for me since, amongst the 10+ articles I read, none actually mentioned the rationale of sorting. Tito's notebook mentioned though, but I cannot grasp it also, I read through a youtuber's source code on this, but they indeed sorted them (without explaining).
", "votes": null, "replies": [] } ] }, { "id": 1736602, "author_name": "cdeotte", "author_url": "", "post_date": "03/27/2022 14:15:06", "content": "The metric MAP answers the questions,
\n\"how good is your prediction if you only guess 1 bounding box\"?,
\n\"how good is your prediction if you only guess 2 bounding boxes\"?
\n\"how good is your prediction if you only guess 3 bounding boxes\"?
\n\"how good is your prediction if you only guess 4 bounding boxes\"?
\netc etc etc
\nthen MAP takes the average of these questions
When evaluating \"how good is your prediction if you only guess X bound boxes\", then of course we would like to use our best X bounding boxes. Our best X bounding boxes are the X bounding boxes with best confidence. Therefore we sort by confidence before answering the above questions.
", "votes": null, "replies": [ { "id": 1737045, "author_name": "reighns", "author_url": "", "post_date": "03/28/2022 04:27:39", "content": "Thanks Chris for the clear answer. In that case, are we able to ask
\n\"at each confidence level (threshold), what is the precision-recall score of the predictions of all bounding boxes while discarding off those below the threshold, then average them over all thresholds?\"
", "votes": null, "replies": [] }, { "id": 1737049, "author_name": "cdeotte", "author_url": "", "post_date": "03/28/2022 04:33:20", "content": "Yes basically yes. The exact statement is \"\"at each confidence level (threshold), if recall increases from previous confidence level, then include this precision and average all previous included precisions\". But what you said, is basically correct.
", "votes": null, "replies": [] }, { "id": 1737147, "author_name": "reighns", "author_url": "", "post_date": "03/28/2022 06:30:57", "content": "@cdeotte Thanks a lot for this, it clears some air. I will go and read the source code again and update my understanding…
", "votes": null, "replies": [] } ] }, { "id": 1736725, "author_name": "jainishsavalia", "author_url": "", "post_date": "03/27/2022 16:45:47", "content": "It gives the precision value from the rank of outcomes we derive.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1736385": "This is slightly different from the metric in this competition, but since I wanted to read up more on MAP as a metric in general, I sieved through multiple articles like [the one by Jonathon](https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173) and [here by Kaggler Tito](https://www.kaggle.com/code/its7171/map-understanding-with-code-and-its-tips?scriptVersionId=54650292).\n\nOne part I cannot grasp immediately is the need to sort the confidence scores of the bounding box in descending order. What is the reason behind sorting here? Amongst the > 10 articles I read, only @its7171 notebook mentioned about sorting, but I cannot grasp it also.\n\nIs it because we want to have a cumulative precision and recall based off the cut-off threshold by the confidence scores since we are discarding all predictions under a certain confidence score?", "1736396": "I feel the answer lies in the statement: 'The lower the confidence threshold, the higher the mAP we will always get', and your doubt is infact the answer to your question.", "1736419": "Thanks, this is kind of confusing for me since, amongst the 10+ articles I read, none actually mentioned the rationale of sorting. Tito's notebook mentioned though, but I cannot grasp it also, I read through a youtuber's source code on this, but they indeed sorted them (without explaining).", "1736602": "The metric MAP answers the questions, \n\"how good is your prediction if you only guess 1 bounding box\"?, \n\"how good is your prediction if you only guess 2 bounding boxes\"?\n\"how good is your prediction if you only guess 3 bounding boxes\"?\n\"how good is your prediction if you only guess 4 bounding boxes\"?\netc etc etc\nthen MAP takes the average of these questions\n\nWhen evaluating \"how good is your prediction if you only guess X bound boxes\", then of course we would like to use our best X bounding boxes. Our best X bounding boxes are the X bounding boxes with best confidence. Therefore we sort by confidence before answering the above questions.", "1736725": "It gives the precision value from the rank of outcomes we derive.", "1737045": "Thanks Chris for the clear answer. In that case, are we able to ask\n\n\"at each confidence level (threshold), what is the precision-recall score of the predictions of all bounding boxes while discarding off those below the threshold, then average them over all thresholds?\"", "1737049": "Yes basically yes. The exact statement is \"\"at each confidence level (threshold), if recall increases from previous confidence level, then include this precision and average all previous included precisions\". But what you said, is basically correct.", "1737147": "cdeotte Thanks a lot for this, it clears some air. I will go and read the source code again and update my understanding..." }, "source": "meta" }