I searched for it all over the competition but can't find the exact difference the confidences makes in the result.
", "messages": [ { "id": 1205460, "postDate": "2021-02-16T17:24:01.470Z", "content": "I searched for it all over the competition but can't find the exact difference the confidences makes in the result.
", "rawMarkdown": "I searched for it all over the competition but can't find the exact difference the confidences makes in the result.", "votes": 5 }, { "id": 1210956, "postDate": "2021-02-19T20:37:36.637Z", "content": "This video explains how confidence scores are used.
\nCheck out the first 8 minutes for an explanation of mAP.
\nThe rest of the video contains implementation code.
This one was helpful. For people(like me) who don't know what mAP is I'd recommend checking this out 😀
", "rawMarkdown": "This one was helpful. For people(like me) who don't know what mAP is I'd recommend checking this out 😀", "votes": 1 } ] }, { "id": 1208850, "postDate": "2021-02-18T14:14:19.447Z", "content": "@lukereijnen I read somewhere that confidence doesn't affect the score, you can even hardcode it to an arbitrary value. I will update you when I find the source. Edit: https://www.kaggle.com/thedrcat/hpa-single-cell-classification-eda
\nIt says \"we can always set the confidence to 1 without impacting the score\".
", "rawMarkdown": "@lukereijnen I read somewhere that confidence doesn't affect the score, you can even hardcode it to an arbitrary value. I will update you when I find the source. Edit: https://www.kaggle.com/thedrcat/hpa-single-cell-classification-eda\n\nIt says \"we can always set the confidence to 1 without impacting the score\".", "votes": 1, "replies": [ { "id": 1208939, "postDate": "2021-02-18T15:14:54.133Z", "content": "There may be some misunderstanding here.
\nI don't think mAP can work without confidence score.
Okay, then maybe this is an older version. I'll check with the author of that kernel. 😅
", "rawMarkdown": "Okay, then maybe this is an older version. I'll check with the author of that kernel. 😅", "votes": 1 }, { "id": 1208996, "postDate": "2021-02-18T15:59:26.810Z", "content": "Or maybe I'm wrong.😅
\nI'll set the confidence to 1 and submit it later, and let you know the result here.
Thanks, that should clear things up :)
", "rawMarkdown": "Thanks, that should clear things up :)" }, { "id": 1209158, "postDate": "2021-02-18T18:10:39.860Z", "content": "Like all things statistical - I think the correct answer is it depends.
\nI think this is how it depends for this competition.
You cannot use a confidence of 1 to select the predicted class(es) for an cell. Hopefully it's obvious that this value will result in no classes being selected for a cell.
\nBut
\nOnce you have made a selection and are reporting the class in your submission file than PERHAPS you can put a value of 1 in the file without hurting the score.
\nThis metric is really confusing for me - someone post a nice explanation that seemed to clear up my mind - but once I found source code that might be in use I got all confused again :)
", "rawMarkdown": "Like all things statistical - I think the correct answer is it depends.\nI think this is how it depends for this competition.\n\nYou cannot use a confidence of 1 to select the predicted class(es) for an cell. Hopefully it's obvious that this value will result in no classes being selected for a cell.\n\nBut\n\nOnce you have made a selection and are reporting the class in your submission file than **PERHAPS** you can put a value of 1 in the file without hurting the score.\n\n- not sure why you would do that - you should have code access to the computed probability when you made the decision to select the class(es) - so why not just report that value rather than fake it with a 1? (Not likely that a metric change will occur for the third time but always a slim chance of a change where the \"1\" does not really hurt)\n\nThis metric is really confusing for me - someone post a nice explanation that seemed to clear up my mind - but once I found source code that might be in use I got all confused again :)" }, { "id": 1209222, "postDate": "2021-02-18T18:59:08.633Z", "content": "\n\nI'll set the confidence to 1 and submit it later, and let you know the result here.
\n
I obtained very low LB score of 0.116.
\nSo confidence score should be used in LB.
This is a complex subject and the discussion is often confusing, so It might be a good idea to read reliable source code for mAP.
\nAgain, I hope this notebook helps.
\n\nI obtained very low LB score of 0.116.
\n
\n So confidence score should be used in LB.
I see, thanks @its7171
", "rawMarkdown": "> I obtained very low LB score of 0.116.\n> So confidence score should be used in LB.\n\nI see, thanks @its7171 ", "votes": 1 }, { "id": 1209785, "postDate": "2021-02-19T03:33:29.403Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 1209829, "postDate": "2021-02-19T04:04:50.423Z", "content": "yes - important to include probability 👍😃
", "rawMarkdown": "yes - important to include probability 👍😃", "votes": 1 }, { "id": 1209923, "postDate": "2021-02-19T05:35:19.457Z", "content": "@arka47 @its7171 thanks for confirming that confidence is relevant! I was confused by a comment by the host:
\n\n\nConfidence is just ranking of scoring (which mask/label got scored first), so you can set them all to 1 like this and it won't affect your score.
\n
@lnhtrang - can you reconfirm the relevance of the confidence score? Based on the discussion above your comment appears to be incorrect.
", "rawMarkdown": "@arka47 @its7171 thanks for confirming that confidence is relevant! I was confused by a [comment by the host](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/215141): \n> Confidence is just ranking of scoring (which mask/label got scored first), so you can set them all to 1 like this and it won't affect your score.\n\n@lnhtrang - can you reconfirm the relevance of the confidence score? Based on the discussion above your comment appears to be incorrect. ", "votes": 2 }, { "id": 1209951, "postDate": "2021-02-19T05:56:51.743Z", "content": "@its7171
\nThank you very much for your (all confidence==1) submission and the amazing notebook.
\nNow I understand the situation as following:
\n(All confidence==1) means random ranking.
\nFor one class, if there are many \"False\" accidentally around the beginning of the ranking (#1, #2, #3…), the left part of the PR curve closes to 0, where AP score cannot be calculated properly (underestimation).
With the initial scoring system, the confidence didn't affect the score. After the switch to mAP, the confidence score is indeed being used as part of the scoring.
", "rawMarkdown": "With the initial scoring system, the confidence didn't affect the score. After the switch to mAP, the confidence score is indeed being used as part of the scoring.", "votes": 4 }, { "id": 1210255, "postDate": "2021-02-19T09:46:28.910Z", "content": "Thanks for confirming!
", "rawMarkdown": "Thanks for confirming!", "votes": 1 }, { "id": 1210261, "postDate": "2021-02-19T09:53:38.700Z", "content": "@cwinsnes,
\nThanks for the clarification.
\nIt's all clear now.
For one class, if there are many \"False\" accidentally around the beginning of the ranking (#1, #2, #3…), the left part of the PR curve closes to 0\n\n
Yes, box with a high confidence value have high precision, so Precision/Recall curve is usually a decreasing curve.
\nBut if false_positives and ture_positives lists are not sorted,Precision/Recall curve will be a flat curve.
\nThis situation can be checked by commenting out the following 3 lines in my notebook.
indices = np.argsort(-scores)\nfalse_positives = false_positives[indices]\ntrue_positives = true_positives[indices]\n", "rawMarkdown": "@cwinsnes,\nThanks for the clarification.\nIt's all clear now.\n\n@drtausamaru,\n
\nFor one class, if there are many \"False\" accidentally around the beginning of the ranking (#1, #2, #3…), the left part of the PR curve closes to 0\n\nYes, box with a high confidence value have high precision, so Precision/Recall curve is usually a decreasing curve.\nBut if false_positives and ture_positives lists are not sorted,Precision/Recall curve will be a flat curve.\nThis situation can be checked by commenting out the following 3 lines in my notebook.\n
\nindices = np.argsort(-scores)\nfalse_positives = false_positives[indices]\ntrue_positives = true_positives[indices]\n\n", "votes": 2 } ] }, { "id": 1226587, "postDate": "2021-03-04T17:10:21.200Z", "content": "
So does it makes sense then not to use a threshold for the cell-wise confidences and just put every label and it's according confidence there?
", "rawMarkdown": "So does it makes sense then not to use a threshold for the cell-wise confidences and just put every label and it's according confidence there?", "replies": [ { "id": 1226949, "postDate": "2021-03-05T03:46:47.117Z", "content": "Yes - for me this led to the best score. The only caveat is that you need to get the model to work well on negative cells.
", "rawMarkdown": "Yes - for me this led to the best score. The only caveat is that you need to get the model to work well on negative cells. " } ] }, { "id": 1205607, "postDate": "2021-02-16T20:16:40.437Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 1207198, "postDate": "2021-02-17T18:35:43Z", "content": "Nice paper - thanks for the share. Guess the thing we are missing to do local measurement of mAP is some ground truth segment images. There are images on the Human Atlas site with segments - have not found how to download any of those to create a set of ground truth's. Also of course not certain that they are \"annotated\" segments.
", "rawMarkdown": "Nice paper - thanks for the share. Guess the thing we are missing to do local measurement of mAP is some ground truth segment images. There are images on the Human Atlas site with segments - have not found how to download any of those to create a set of ground truth's. Also of course not certain that they are \"annotated\" segments.", "votes": 1 }, { "id": 1208583, "postDate": "2021-02-18T10:48:36.617Z", "content": "Great read, but I still don't understand how the confidence influences our final score.
", "rawMarkdown": "Great read, but I still don't understand how the confidence influences our final score." }, { "id": 1208838, "postDate": "2021-02-18T14:06:12.747Z", "content": "Hi @lukereijnen,
\nConfidence scores is used to sort for the following two purposes.
\nIt is a bit difficult to explain, so I have added this explanation to this note.
\nmAP understanding with code and its tips
\nThis is an mAP description of object detection, not instance segmentation, but it is essentially the same.
I hope this helps.
", "rawMarkdown": "Hi @lukereijnen,\n\nConfidence scores is used to sort for the following two purposes.\n\n1. detected bounding boxes should be sorted by confidence scores before cehck if TP or FP.\n2. FP and TP list should be sorted by confidence scores before Average Precision Calculation.\n\nIt is a bit difficult to explain, so I have added this explanation to this note.\n[mAP understanding with code and its tips](https://www.kaggle.com/its7171/map-understanding-with-code-and-its-tips)\nThis is an mAP description of object detection, not instance segmentation, but it is essentially the same.\n\nI hope this helps.", "votes": 5 } ] } ], "comments": [ { "id": 1210956, "author_name": "CroDoc", "author_url": "", "post_date": "2021-02-19T20:37:36.637000", "content": "This video explains how confidence scores are used.
\nCheck out the first 8 minutes for an explanation of mAP.
\nThe rest of the video contains implementation code.
This one was helpful. For people(like me) who don't know what mAP is I'd recommend checking this out 😀
", "votes": 1, "replies": [] } ] }, { "id": 1208850, "author_name": "Arka Saha", "author_url": "", "post_date": "2021-02-18T14:14:19.447000", "content": "@lukereijnen I read somewhere that confidence doesn't affect the score, you can even hardcode it to an arbitrary value. I will update you when I find the source. Edit: https://www.kaggle.com/thedrcat/hpa-single-cell-classification-eda
\nIt says \"we can always set the confidence to 1 without impacting the score\".
", "votes": 1, "replies": [ { "id": 1208939, "author_name": "tito", "author_url": "", "post_date": "2021-02-18T15:14:54.133000", "content": "There may be some misunderstanding here.
\nI don't think mAP can work without confidence score.
Okay, then maybe this is an older version. I'll check with the author of that kernel. 😅
", "votes": 1, "replies": [] }, { "id": 1208996, "author_name": "tito", "author_url": "", "post_date": "2021-02-18T15:59:26.810000", "content": "Or maybe I'm wrong.😅
\nI'll set the confidence to 1 and submit it later, and let you know the result here.
Thanks, that should clear things up :)
", "votes": 0, "replies": [] }, { "id": 1209158, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-02-18T18:10:39.860000", "content": "Like all things statistical - I think the correct answer is it depends.
\nI think this is how it depends for this competition.
You cannot use a confidence of 1 to select the predicted class(es) for an cell. Hopefully it's obvious that this value will result in no classes being selected for a cell.
\nBut
\nOnce you have made a selection and are reporting the class in your submission file than PERHAPS you can put a value of 1 in the file without hurting the score.
\nThis metric is really confusing for me - someone post a nice explanation that seemed to clear up my mind - but once I found source code that might be in use I got all confused again :)
", "votes": 0, "replies": [] }, { "id": 1209222, "author_name": "tito", "author_url": "", "post_date": "2021-02-18T18:59:08.633000", "content": "\n\nI'll set the confidence to 1 and submit it later, and let you know the result here.
\n
I obtained very low LB score of 0.116.
\nSo confidence score should be used in LB.
This is a complex subject and the discussion is often confusing, so It might be a good idea to read reliable source code for mAP.
\nAgain, I hope this notebook helps.
\n\nI obtained very low LB score of 0.116.
\n
\n So confidence score should be used in LB.
I see, thanks @its7171
", "votes": 1, "replies": [] }, { "id": 1209785, "author_name": "", "author_url": "", "post_date": "2021-02-19T03:33:29.403000", "content": "", "votes": 2, "replies": [] }, { "id": 1209829, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-02-19T04:04:50.423000", "content": "yes - important to include probability 👍😃
", "votes": 1, "replies": [] }, { "id": 1209923, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2021-02-19T05:35:19.457000", "content": "@arka47 @its7171 thanks for confirming that confidence is relevant! I was confused by a comment by the host:
\n\n\nConfidence is just ranking of scoring (which mask/label got scored first), so you can set them all to 1 like this and it won't affect your score.
\n
@lnhtrang - can you reconfirm the relevance of the confidence score? Based on the discussion above your comment appears to be incorrect.
", "votes": 2, "replies": [] }, { "id": 1209951, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-02-19T05:56:51.743000", "content": "@its7171
\nThank you very much for your (all confidence==1) submission and the amazing notebook.
\nNow I understand the situation as following:
\n(All confidence==1) means random ranking.
\nFor one class, if there are many \"False\" accidentally around the beginning of the ranking (#1, #2, #3…), the left part of the PR curve closes to 0, where AP score cannot be calculated properly (underestimation).
With the initial scoring system, the confidence didn't affect the score. After the switch to mAP, the confidence score is indeed being used as part of the scoring.
", "votes": 4, "replies": [] }, { "id": 1210255, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2021-02-19T09:46:28.910000", "content": "Thanks for confirming!
", "votes": 1, "replies": [] }, { "id": 1210261, "author_name": "tito", "author_url": "", "post_date": "2021-02-19T09:53:38.700000", "content": "@cwinsnes,
\nThanks for the clarification.
\nIt's all clear now.
For one class, if there are many \"False\" accidentally around the beginning of the ranking (#1, #2, #3…), the left part of the PR curve closes to 0\n\n
Yes, box with a high confidence value have high precision, so Precision/Recall curve is usually a decreasing curve.
\nBut if false_positives and ture_positives lists are not sorted,Precision/Recall curve will be a flat curve.
\nThis situation can be checked by commenting out the following 3 lines in my notebook.
indices = np.argsort(-scores)\nfalse_positives = false_positives[indices]\ntrue_positives = true_positives[indices]\n", "votes": 2, "replies": [] } ] }, { "id": 1226587, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-03-04T17:10:21.200000", "content": "
So does it makes sense then not to use a threshold for the cell-wise confidences and just put every label and it's according confidence there?
", "votes": 0, "replies": [ { "id": 1226949, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2021-03-05T03:46:47.117000", "content": "Yes - for me this led to the best score. The only caveat is that you need to get the model to work well on negative cells.
", "votes": 0, "replies": [] } ] }, { "id": 1205607, "author_name": "", "author_url": "", "post_date": "2021-02-16T20:16:40.437000", "content": "", "votes": 1, "replies": [ { "id": 1207198, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-02-17T18:35:43", "content": "Nice paper - thanks for the share. Guess the thing we are missing to do local measurement of mAP is some ground truth segment images. There are images on the Human Atlas site with segments - have not found how to download any of those to create a set of ground truth's. Also of course not certain that they are \"annotated\" segments.
", "votes": 1, "replies": [] }, { "id": 1208583, "author_name": "SmileEveryDay", "author_url": "", "post_date": "2021-02-18T10:48:36.617000", "content": "Great read, but I still don't understand how the confidence influences our final score.
", "votes": 0, "replies": [] }, { "id": 1208838, "author_name": "tito", "author_url": "", "post_date": "2021-02-18T14:06:12.747000", "content": "Hi @lukereijnen,
\nConfidence scores is used to sort for the following two purposes.
\nIt is a bit difficult to explain, so I have added this explanation to this note.
\nmAP understanding with code and its tips
\nThis is an mAP description of object detection, not instance segmentation, but it is essentially the same.
I hope this helps.
", "votes": 5, "replies": [] } ] } ], "raw_markdown_by_id": { "1205460": "I searched for it all over the competition but can't find the exact difference the confidences makes in the result.", "1210956": "This [video](https://www.youtube.com/watch?v=FppOzcDvaDI) explains how confidence scores are used.\nCheck out the first 8 minutes for an explanation of mAP.\nThe rest of the video contains implementation code.", "1208850": "@lukereijnen I read somewhere that confidence doesn't affect the score, you can even hardcode it to an arbitrary value. I will update you when I find the source. Edit: https://www.kaggle.com/thedrcat/hpa-single-cell-classification-eda\n\nIt says \"we can always set the confidence to 1 without impacting the score\".", "1226587": "So does it makes sense then not to use a threshold for the cell-wise confidences and just put every label and it's according confidence there?", "1205607": "" } }