As this is my first time to use mAP metric, I am striving to implement the exact competition metric in my local CV settings as close as possible to LB. I would like to share what I understand til now and to discuss confusing points, especially about study-level mAP. Would be appreciated to correct me if anything wrongs below and clarify few things.
\nAccording to the hosts, the calculation is based on PASCAL VOC 2010 competition metric. The PASCAL VOC 2010 is one version of several mAP versions. The mAP metric is modified over the past competitions and literatures. The basic common concept of mAP is nothing but the area under the recall-precision curve. However, we cannot draw this curve continuously. Depending on how to deal with this approximation of recall-precision curve and its auc values, the mAP versions have their own methods.
\nOur competition metric mAP is drawn by interpolating (recall, precision) points at some IoU values. These IoU values usually between 0.5 and 0.75 with step size 0.05. (i.e. [0.5, 0.55, 0.60, 0.65, 0.7, 0.75]) However, in this competition, we use IoU values > 0.5 as described in evaluation section. Step size is not mentioned, but mostly, the step size is 0.05. I am not sure how much the gaps between mAP values with different step sizes (0.01, 0.05, 0.1, ,,,), but It would be grateful if the hosts clarify the step size. I am taking step size 0.05 and the IoU values to calculate recall-precision curve are [0.5, 0.55, ,,, 0.95, 1.0]. Note that this mAP in PASCAL VOC 2010 make the recall-precision curve monotone decreasing form when calculating AUC of recall-precision curve. (By just taking the maximum recall among recall values at next IoU steps) The above is done for each class and the mean value of auc values for each class is the final mAP of our model.
One can find a nice detailed explanation here (https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173) and a great kernel (https://www.kaggle.com/pestipeti/competition-metric-details-script).
\nFor evaluating image-level labels, the above kernel's mAP is valid, but, for this competition, it is needed additional implementation of monotone decrease function for recall values would be enough to implement the right mAP implementation for this competition. Note that the monotone decreased recall is not included in most mAP calculation libraries sklearn and ZFTurbo's Mean-Average-Precision-for-Boxes.
\nI am still not getting how to calculate this and more detailed explanations or formulas need to be revealed by the host IHMO.
\nCurrently, I am assuming that the IoU values is replaced to confidence values and calculated in one-versus-rest manner since no IoU can be calculated. That is, for each class, we can draw recall-precision curve using points at each confidence(correspond to IoU in PASCAL VOC 2010 mAP) step. [0.5, 0.55, ,,, 0.95, 1.0] If this is true, we should make prediction results not with a single-label prediction with confidence 1, (i.e. [0.9, 0.6, 0.1, 0.3] -> \"negative 1 0 0 1 1\") but multi-label predictions (i.e. [0.9, 0.6, 0.1, 0.3] -> \"negative 0.9 0 0 1 1 typical 0.6 0 0 1 1\")
Also, I am wondering there could be multiple labels for a single study in test set. If so, are the test set labeled with confidence or just 1? From the evaluation section in overview tab, the example says it is possible to make PredictionString contains multi labels. Before the LB updates, some people say a study can have multiple images and thus, multiple study-level labels can be predicted for a study (from discussion here). However, now in the test set there are only one image according to the host here. Then, do we have single study-level label in test set as the training set does?
\nThank you for reading and any comments will be welcomed :)
", "messages": [ { "id": "1362763", "postDate": "06/23/2021 16:09:09", "content": "As this is my first time to use mAP metric, I am striving to implement the exact competition metric in my local CV settings as close as possible to LB. I would like to share what I understand til now and to discuss confusing points, especially about study-level mAP. Would be appreciated to correct me if anything wrongs below and clarify few things.
\nAccording to the hosts, the calculation is based on PASCAL VOC 2010 competition metric. The PASCAL VOC 2010 is one version of several mAP versions. The mAP metric is modified over the past competitions and literatures. The basic common concept of mAP is nothing but the area under the recall-precision curve. However, we cannot draw this curve continuously. Depending on how to deal with this approximation of recall-precision curve and its auc values, the mAP versions have their own methods.
\nOur competition metric mAP is drawn by interpolating (recall, precision) points at some IoU values. These IoU values usually between 0.5 and 0.75 with step size 0.05. (i.e. [0.5, 0.55, 0.60, 0.65, 0.7, 0.75]) However, in this competition, we use IoU values > 0.5 as described in evaluation section. Step size is not mentioned, but mostly, the step size is 0.05. I am not sure how much the gaps between mAP values with different step sizes (0.01, 0.05, 0.1, ,,,), but It would be grateful if the hosts clarify the step size. I am taking step size 0.05 and the IoU values to calculate recall-precision curve are [0.5, 0.55, ,,, 0.95, 1.0]. Note that this mAP in PASCAL VOC 2010 make the recall-precision curve monotone decreasing form when calculating AUC of recall-precision curve. (By just taking the maximum recall among recall values at next IoU steps) The above is done for each class and the mean value of auc values for each class is the final mAP of our model.
One can find a nice detailed explanation here (https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173) and a great kernel (https://www.kaggle.com/pestipeti/competition-metric-details-script).
\nFor evaluating image-level labels, the above kernel's mAP is valid, but, for this competition, it is needed additional implementation of monotone decrease function for recall values would be enough to implement the right mAP implementation for this competition. Note that the monotone decreased recall is not included in most mAP calculation libraries sklearn and ZFTurbo's Mean-Average-Precision-for-Boxes.
\nI am still not getting how to calculate this and more detailed explanations or formulas need to be revealed by the host IHMO.
\nCurrently, I am assuming that the IoU values is replaced to confidence values and calculated in one-versus-rest manner since no IoU can be calculated. That is, for each class, we can draw recall-precision curve using points at each confidence(correspond to IoU in PASCAL VOC 2010 mAP) step. [0.5, 0.55, ,,, 0.95, 1.0] If this is true, we should make prediction results not with a single-label prediction with confidence 1, (i.e. [0.9, 0.6, 0.1, 0.3] -> \"negative 1 0 0 1 1\") but multi-label predictions (i.e. [0.9, 0.6, 0.1, 0.3] -> \"negative 0.9 0 0 1 1 typical 0.6 0 0 1 1\")
Also, I am wondering there could be multiple labels for a single study in test set. If so, are the test set labeled with confidence or just 1? From the evaluation section in overview tab, the example says it is possible to make PredictionString contains multi labels. Before the LB updates, some people say a study can have multiple images and thus, multiple study-level labels can be predicted for a study (from discussion here). However, now in the test set there are only one image according to the host here. Then, do we have single study-level label in test set as the training set does?
\nThank you for reading and any comments will be welcomed :)
", "rawMarkdown": "As this is my first time to use mAP metric, I am striving to implement the exact competition metric in my local CV settings as close as possible to LB. I would like to share what I understand til now and to discuss confusing points, especially about study-level mAP. Would be appreciated to correct me if anything wrongs below and clarify few things.\n\n## **Image-level mean Average Precision**\n\nAccording to the hosts, the calculation is based on PASCAL VOC 2010 competition metric. The PASCAL VOC 2010 is one version of several mAP versions. The mAP metric is modified over the past competitions and literatures. The basic common concept of mAP is nothing but **the area under the recall-precision curve**. However, we cannot draw this curve continuously. Depending on how to deal with this approximation of recall-precision curve and its auc values, the mAP versions have their own methods. \nOur competition metric mAP is drawn by interpolating (recall, precision) points at some IoU values. These IoU values usually between 0.5 and 0.75 with step size 0.05. (i.e. [0.5, 0.55, 0.60, 0.65, 0.7, 0.75]) However, in this competition, we use IoU values > 0.5 as described in evaluation section. Step size is not mentioned, but mostly, the step size is 0.05. I am not sure how much the gaps between mAP values with different step sizes (0.01, 0.05, 0.1, ,,,), but It would be grateful if the hosts clarify the step size. I am taking step size 0.05 and the IoU values to calculate recall-precision curve are [0.5, 0.55, ,,, 0.95, 1.0]. Note that this mAP in PASCAL VOC 2010 make the recall-precision curve monotone decreasing form when calculating AUC of recall-precision curve. (By just taking the maximum recall among recall values at next IoU steps) The above is done for each class and the mean value of auc values for each class is the final mAP of our model.\n\nOne can find a nice detailed explanation here ([https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173](https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173)) and a great kernel ([https://www.kaggle.com/pestipeti/competition-metric-details-script]( https://www.kaggle.com/pestipeti/competition-metric-details-script)).\n\nFor evaluating image-level labels, the above kernel's mAP is valid, but, for this competition, it is needed additional implementation of monotone decrease function for recall values would be enough to implement the right mAP implementation for this competition. Note that the monotone decreased recall is not included in most mAP calculation libraries sklearn and ZFTurbo's [Mean-Average-Precision-for-Boxes](https://github.com/ZFTurbo/Mean-Average-Precision-for-Boxes).\n\n## **Study-level mean Average Precision?**\n\nI am still not getting how to calculate this and more detailed explanations or formulas need to be revealed by the host IHMO. \nCurrently, I am assuming that the IoU values is replaced to confidence values and calculated in one-versus-rest manner since no IoU can be calculated. That is, for each class, we can draw recall-precision curve using points at each confidence(correspond to IoU in PASCAL VOC 2010 mAP) step. [0.5, 0.55, ,,, 0.95, 1.0] If this is true, we should make prediction results not with a single-label prediction with confidence 1, (i.e. [0.9, 0.6, 0.1, 0.3] -> \"negative 1 0 0 1 1\") but multi-label predictions (i.e. [0.9, 0.6, 0.1, 0.3] -> \"negative 0.9 0 0 1 1 typical 0.6 0 0 1 1\")\n\nAlso, I am wondering there could be multiple labels for a single study in test set. If so, are the test set labeled with confidence or just 1? From the evaluation section in overview tab, the example says it is possible to make PredictionString contains multi labels. Before the LB updates, some people say a study can have multiple images and thus, multiple study-level labels can be predicted for a study (from discussion [here](https://www.kaggle.com/c/siim-covid19-detection/discussion/240329)). However, now in the test set there are only one image according to the host [here](https://www.kaggle.com/c/siim-covid19-detection/discussion/246597). Then, do we have single study-level label in test set as the training set does?\n\nThank you for reading and any comments will be welcomed :)", "votes": null }, { "id": "1362916", "postDate": "06/23/2021 18:13:12", "content": "Hi! Good questions.
\n1) There is only one IoU threshold - 0.5. The AP calculation is done using a P/R curve - specifically, counting precision at each unique recall value. There is clarification around this in this document, which is also linked in the Evaluation page.
2) For the study-level labels, to maximize your score, you should be making ONE correct prediction. Making multiple predictions will result in false positives. Confidence scores are used only for sorting predictions (i.e. so that your most confident prediction will be tried against the ground truth first) and have no bearing on the correctness of the prediction outside of that, or on additional predictions being counted as FPs. Because we use placeholder bounding boxes the IoU should always be 1.0, so getting the class correct is key.
\nHopefully the linked document helps. Note that there is also MATLAB evaluation code associated with that document, which may help clarify some of the issues in a more concrete way.
", "rawMarkdown": "Hi! Good questions.\n\n1) There is only one IoU threshold - `0.5`. The AP calculation is done using a P/R curve - specifically, counting precision at each unique recall value. There is clarification around this in [this document](https://storage.googleapis.com/kaggle-media/competitions/SIIM2021/VOC2012_doc.pdf), which is also linked in the Evaluation page.\n\n2) For the study-level labels, to maximize your score, you should be making ONE correct prediction. Making multiple predictions will result in false positives. Confidence scores are used only for sorting predictions (i.e. so that your most confident prediction will be tried against the ground truth first) and have no bearing on the correctness of the prediction outside of that, or on additional predictions being counted as FPs. Because we use placeholder bounding boxes the IoU should always be 1.0, so getting the class correct is key.\n\nHopefully the linked document helps. Note that there is also MATLAB evaluation code associated with that document, which may help clarify some of the issues in a more concrete way.", "votes": null }, { "id": "1363379", "postDate": "06/24/2021 06:09:15", "content": "Thank you for the kind and quick response.
\nI thoroughly re-read the document in the link and your comment, I could realize which part is missing in my understanding. I was confused the role of IoU during mAP calculation and PR-curve calculation.
\nNow I think I finally get the right PR-curve and AP implementations. thank you so much :))
Hello! after some submissions, I have another question.
\nPublic LB gives better results when I gave prediction for all four classes with sigmoid confidence values (LB 0.442) than argmax single prediction with confidence value 1 (LB 0.318).
for instance, if we get [0.7, 0.5, 0.4, 0.1],
\nnegative 0.7 0 0 1 1 typical 0.5 0 0 1 1 indeterminate 0.4 0 0 1 1 atypical 0.1 0 0 1 1 # Higher LB with this row's format\nnegative 1 0 0 1 1\nare there any internal thresholding process with confidence value for study-level prediction? According to your comment, the false positive will be always 3. and IMHO it should be harmful to the prediction results.
", "rawMarkdown": "Hello! after some submissions, I have another question.\nPublic LB gives better results when I gave prediction for all four classes with sigmoid confidence values (LB 0.442) than argmax single prediction with confidence value 1 (LB 0.318).\n\nfor instance, if we get [0.7, 0.5, 0.4, 0.1],\n```\nnegative 0.7 0 0 1 1 typical 0.5 0 0 1 1 indeterminate 0.4 0 0 1 1 atypical 0.1 0 0 1 1 # Higher LB with this row's format\nnegative 1 0 0 1 1\n```\n\nare there any internal thresholding process with confidence value for study-level prediction? According to your comment, the false positive will be always 3. and IMHO it should be harmful to the prediction results.", "votes": null }, { "id": "1398775", "postDate": "07/24/2021 13:28:47", "content": "Hi,
\nOriginally I thought the so-called mAP is calculated as the mean of the precision values at each IoU threshold, but now for this competition, mAP calculation is actually using one single loU threshold 0.5 right? This is called mean average precision just becuase we are averaging the precision when we are selecting different recall values, instead of different loU, do I get it right? Thanks!
", "rawMarkdown": "Hi, \n\nOriginally I thought the so-called mAP is calculated as the mean of the precision values at each IoU threshold, but now for this competition, mAP calculation is actually using one single loU threshold 0.5 right? This is called mean average precision just becuase we are averaging the precision when we are selecting different recall values, instead of different loU, do I get it right? Thanks!", "votes": null } ], "comments": [ { "id": 1362916, "author_name": "philculliton", "author_url": "", "post_date": "06/23/2021 18:13:12", "content": "Hi! Good questions.
\n1) There is only one IoU threshold - 0.5. The AP calculation is done using a P/R curve - specifically, counting precision at each unique recall value. There is clarification around this in this document, which is also linked in the Evaluation page.
2) For the study-level labels, to maximize your score, you should be making ONE correct prediction. Making multiple predictions will result in false positives. Confidence scores are used only for sorting predictions (i.e. so that your most confident prediction will be tried against the ground truth first) and have no bearing on the correctness of the prediction outside of that, or on additional predictions being counted as FPs. Because we use placeholder bounding boxes the IoU should always be 1.0, so getting the class correct is key.
\nHopefully the linked document helps. Note that there is also MATLAB evaluation code associated with that document, which may help clarify some of the issues in a more concrete way.
", "votes": null, "replies": [ { "id": 1363379, "author_name": "jihunlorenzopark", "author_url": "", "post_date": "06/24/2021 06:09:15", "content": "Thank you for the kind and quick response.
\nI thoroughly re-read the document in the link and your comment, I could realize which part is missing in my understanding. I was confused the role of IoU during mAP calculation and PR-curve calculation.
\nNow I think I finally get the right PR-curve and AP implementations. thank you so much :))
Hello! after some submissions, I have another question.
\nPublic LB gives better results when I gave prediction for all four classes with sigmoid confidence values (LB 0.442) than argmax single prediction with confidence value 1 (LB 0.318).
for instance, if we get [0.7, 0.5, 0.4, 0.1],
\nnegative 0.7 0 0 1 1 typical 0.5 0 0 1 1 indeterminate 0.4 0 0 1 1 atypical 0.1 0 0 1 1 # Higher LB with this row's format\nnegative 1 0 0 1 1\nare there any internal thresholding process with confidence value for study-level prediction? According to your comment, the false positive will be always 3. and IMHO it should be harmful to the prediction results.
", "votes": null, "replies": [] }, { "id": 1398775, "author_name": "yus002", "author_url": "", "post_date": "07/24/2021 13:28:47", "content": "Hi,
\nOriginally I thought the so-called mAP is calculated as the mean of the precision values at each IoU threshold, but now for this competition, mAP calculation is actually using one single loU threshold 0.5 right? This is called mean average precision just becuase we are averaging the precision when we are selecting different recall values, instead of different loU, do I get it right? Thanks!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1362763": "As this is my first time to use mAP metric, I am striving to implement the exact competition metric in my local CV settings as close as possible to LB. I would like to share what I understand til now and to discuss confusing points, especially about study-level mAP. Would be appreciated to correct me if anything wrongs below and clarify few things.\n\n## **Image-level mean Average Precision**\n\nAccording to the hosts, the calculation is based on PASCAL VOC 2010 competition metric. The PASCAL VOC 2010 is one version of several mAP versions. The mAP metric is modified over the past competitions and literatures. The basic common concept of mAP is nothing but **the area under the recall-precision curve**. However, we cannot draw this curve continuously. Depending on how to deal with this approximation of recall-precision curve and its auc values, the mAP versions have their own methods. \nOur competition metric mAP is drawn by interpolating (recall, precision) points at some IoU values. These IoU values usually between 0.5 and 0.75 with step size 0.05. (i.e. [0.5, 0.55, 0.60, 0.65, 0.7, 0.75]) However, in this competition, we use IoU values > 0.5 as described in evaluation section. Step size is not mentioned, but mostly, the step size is 0.05. I am not sure how much the gaps between mAP values with different step sizes (0.01, 0.05, 0.1, ,,,), but It would be grateful if the hosts clarify the step size. I am taking step size 0.05 and the IoU values to calculate recall-precision curve are [0.5, 0.55, ,,, 0.95, 1.0]. Note that this mAP in PASCAL VOC 2010 make the recall-precision curve monotone decreasing form when calculating AUC of recall-precision curve. (By just taking the maximum recall among recall values at next IoU steps) The above is done for each class and the mean value of auc values for each class is the final mAP of our model.\n\nOne can find a nice detailed explanation here ([https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173](https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173)) and a great kernel ([https://www.kaggle.com/pestipeti/competition-metric-details-script]( https://www.kaggle.com/pestipeti/competition-metric-details-script)).\n\nFor evaluating image-level labels, the above kernel's mAP is valid, but, for this competition, it is needed additional implementation of monotone decrease function for recall values would be enough to implement the right mAP implementation for this competition. Note that the monotone decreased recall is not included in most mAP calculation libraries sklearn and ZFTurbo's [Mean-Average-Precision-for-Boxes](https://github.com/ZFTurbo/Mean-Average-Precision-for-Boxes).\n\n## **Study-level mean Average Precision?**\n\nI am still not getting how to calculate this and more detailed explanations or formulas need to be revealed by the host IHMO. \nCurrently, I am assuming that the IoU values is replaced to confidence values and calculated in one-versus-rest manner since no IoU can be calculated. That is, for each class, we can draw recall-precision curve using points at each confidence(correspond to IoU in PASCAL VOC 2010 mAP) step. [0.5, 0.55, ,,, 0.95, 1.0] If this is true, we should make prediction results not with a single-label prediction with confidence 1, (i.e. [0.9, 0.6, 0.1, 0.3] -> \"negative 1 0 0 1 1\") but multi-label predictions (i.e. [0.9, 0.6, 0.1, 0.3] -> \"negative 0.9 0 0 1 1 typical 0.6 0 0 1 1\")\n\nAlso, I am wondering there could be multiple labels for a single study in test set. If so, are the test set labeled with confidence or just 1? From the evaluation section in overview tab, the example says it is possible to make PredictionString contains multi labels. Before the LB updates, some people say a study can have multiple images and thus, multiple study-level labels can be predicted for a study (from discussion [here](https://www.kaggle.com/c/siim-covid19-detection/discussion/240329)). However, now in the test set there are only one image according to the host [here](https://www.kaggle.com/c/siim-covid19-detection/discussion/246597). Then, do we have single study-level label in test set as the training set does?\n\nThank you for reading and any comments will be welcomed :)", "1362916": "Hi! Good questions.\n\n1) There is only one IoU threshold - `0.5`. The AP calculation is done using a P/R curve - specifically, counting precision at each unique recall value. There is clarification around this in [this document](https://storage.googleapis.com/kaggle-media/competitions/SIIM2021/VOC2012_doc.pdf), which is also linked in the Evaluation page.\n\n2) For the study-level labels, to maximize your score, you should be making ONE correct prediction. Making multiple predictions will result in false positives. Confidence scores are used only for sorting predictions (i.e. so that your most confident prediction will be tried against the ground truth first) and have no bearing on the correctness of the prediction outside of that, or on additional predictions being counted as FPs. Because we use placeholder bounding boxes the IoU should always be 1.0, so getting the class correct is key.\n\nHopefully the linked document helps. Note that there is also MATLAB evaluation code associated with that document, which may help clarify some of the issues in a more concrete way.", "1363379": "Thank you for the kind and quick response. \nI thoroughly re-read the document in the link and your comment, I could realize which part is missing in my understanding. I was confused the role of IoU during mAP calculation and PR-curve calculation. \nNow I think I finally get the right PR-curve and AP implementations. thank you so much :))", "1364453": "Hello! after some submissions, I have another question.\nPublic LB gives better results when I gave prediction for all four classes with sigmoid confidence values (LB 0.442) than argmax single prediction with confidence value 1 (LB 0.318).\n\nfor instance, if we get [0.7, 0.5, 0.4, 0.1],\n```\nnegative 0.7 0 0 1 1 typical 0.5 0 0 1 1 indeterminate 0.4 0 0 1 1 atypical 0.1 0 0 1 1 # Higher LB with this row's format\nnegative 1 0 0 1 1\n```\n\nare there any internal thresholding process with confidence value for study-level prediction? According to your comment, the false positive will be always 3. and IMHO it should be harmful to the prediction results.", "1398775": "Hi, \n\nOriginally I thought the so-called mAP is calculated as the mean of the precision values at each IoU threshold, but now for this competition, mAP calculation is actually using one single loU threshold 0.5 right? This is called mean average precision just becuase we are averaging the precision when we are selecting different recall values, instead of different loU, do I get it right? Thanks!" }, "source": "meta" }