Edit: See the comments for the discussion first. Probing expectations of 1/6 here is wrong. Instead use this notebook for verification with LB probing below.
\nI did some LB probing with all my 5 submissions from today here is a summary:
\nHere is the notebook
\nLB:0.0 Expectation:0.0
\nLB:0.0 Expectation:1/6 (Assuming equal class weights mAP)
\nLB: 0.0 also supports the existing problem with image level score calculations. Many say it only contributes 0.001 but infact its not contributing at all, not sure when we will have an explanation about this.
\nLB:0.050 Expectation:1/6 (Assuming equal class weights mAP)
\nThis means mAP is weighted by the number of samples of that class in test set. Probe 4 and 5 supports this idea.
\nLB:0.079 Expectation:1/6 (Assuming equal class weights mAP)
\nThis means mAP is weighted.
\nLB negative probe / LB typical probe = 0.050 / 0.079 = 0.63
\nno. of negative in train / no. of typical in train = 1676 / 2855 = 0.58
\nWe can see that ratios of LB scores and number of samples are very close. They are not exactly same since class distributions of training and test are not same, we probably have more negatives in test set.
\nLB:0.031 Expectation:1/6 (Assuming equal class weights mAP)
\nThis means mAP is weighted.
\nLB negative probe / LB indeterminate probe = 0.050 / 0.031 = 1.61
\nno. of negative in train / no. of indeterminate in train = 1676 / 2855 = 1.59
\nLB:0.031 Expectation:1/6 (Assuming equal class weights mAP)
\nThis means mAP is weighted.
\nLB negative probe / LB atypical probe = 0.050 / X = Y (Tomorrow: out of submissions)
\nno. of negative in train / no. of atypical in train = 1676 / 2855 = 3.53
\nX should be close to 0.050 / 3.53 ~= 0.014
\nAbove we see that mAP is not weigthed equally, hence not 1/6. We can see the relative contributions within study level classes, but image level classes might be differently weighted compared to study level classes.
\nWe kind of now know that study level predictions are weighted proportionally to their number of samples, if we also assume this for all the 6 classes then here is a rough calculation:
\nLB (probe 3): 0.050 - Expectation: no. of negative in train / all train samples = 0.136
\nLB (probe 4): 0.079 - Expectation: no. of negative in train / all train samples = 0.230
\nLB (probe 4): 0.031 - Expectation: no. of negative in train / all train samples = 0.084
\nWhen we assume equal weight between study and image level predictions numbers don't match. Here if we assume a weight of 0.35 for study level predictions:
\nLB (probe 3): 0.050 - Expectation: 0.35*(no. of negative in train / all train samples) = 0.047
\nLB (probe 4): 0.079 - Expectation: 0.35*(no. of negative in train / all train samples) = 0.080
\nLB (probe 4): 0.031 - Expectation: 0.35*(no. of negative in train / all train samples) = 0.029
\nAgain numbers are slightly different because train and test distribution differences.
\nScore: 0.35x(Weighted mAP Study) + 0.65x(Weighted mAP Image) ???
\n(We can't do much probing around images since evaluation is currently broken.)
\nEverything would be much easier if we could get more answers about metric calculation. It is well known that key to succeeding in any Kaggle competition is to construct a good validation set, and it would only be possible by having the questions around the metric answered.
\nLet me know what you think about my hypothesis in the comments, maybe I am missing something. Thanks.
", "messages": [ { "id": "1336698", "postDate": "06/05/2021 06:51:11", "content": "Edit: See the comments for the discussion first. Probing expectations of 1/6 here is wrong. Instead use this notebook for verification with LB probing below.
\nI did some LB probing with all my 5 submissions from today here is a summary:
\nHere is the notebook
\nLB:0.0 Expectation:0.0
\nLB:0.0 Expectation:1/6 (Assuming equal class weights mAP)
\nLB: 0.0 also supports the existing problem with image level score calculations. Many say it only contributes 0.001 but infact its not contributing at all, not sure when we will have an explanation about this.
\nLB:0.050 Expectation:1/6 (Assuming equal class weights mAP)
\nThis means mAP is weighted by the number of samples of that class in test set. Probe 4 and 5 supports this idea.
\nLB:0.079 Expectation:1/6 (Assuming equal class weights mAP)
\nThis means mAP is weighted.
\nLB negative probe / LB typical probe = 0.050 / 0.079 = 0.63
\nno. of negative in train / no. of typical in train = 1676 / 2855 = 0.58
\nWe can see that ratios of LB scores and number of samples are very close. They are not exactly same since class distributions of training and test are not same, we probably have more negatives in test set.
\nLB:0.031 Expectation:1/6 (Assuming equal class weights mAP)
\nThis means mAP is weighted.
\nLB negative probe / LB indeterminate probe = 0.050 / 0.031 = 1.61
\nno. of negative in train / no. of indeterminate in train = 1676 / 2855 = 1.59
\nLB:0.031 Expectation:1/6 (Assuming equal class weights mAP)
\nThis means mAP is weighted.
\nLB negative probe / LB atypical probe = 0.050 / X = Y (Tomorrow: out of submissions)
\nno. of negative in train / no. of atypical in train = 1676 / 2855 = 3.53
\nX should be close to 0.050 / 3.53 ~= 0.014
\nAbove we see that mAP is not weigthed equally, hence not 1/6. We can see the relative contributions within study level classes, but image level classes might be differently weighted compared to study level classes.
\nWe kind of now know that study level predictions are weighted proportionally to their number of samples, if we also assume this for all the 6 classes then here is a rough calculation:
\nLB (probe 3): 0.050 - Expectation: no. of negative in train / all train samples = 0.136
\nLB (probe 4): 0.079 - Expectation: no. of negative in train / all train samples = 0.230
\nLB (probe 4): 0.031 - Expectation: no. of negative in train / all train samples = 0.084
\nWhen we assume equal weight between study and image level predictions numbers don't match. Here if we assume a weight of 0.35 for study level predictions:
\nLB (probe 3): 0.050 - Expectation: 0.35*(no. of negative in train / all train samples) = 0.047
\nLB (probe 4): 0.079 - Expectation: 0.35*(no. of negative in train / all train samples) = 0.080
\nLB (probe 4): 0.031 - Expectation: 0.35*(no. of negative in train / all train samples) = 0.029
\nAgain numbers are slightly different because train and test distribution differences.
\nScore: 0.35x(Weighted mAP Study) + 0.65x(Weighted mAP Image) ???
\n(We can't do much probing around images since evaluation is currently broken.)
\nEverything would be much easier if we could get more answers about metric calculation. It is well known that key to succeeding in any Kaggle competition is to construct a good validation set, and it would only be possible by having the questions around the metric answered.
\nLet me know what you think about my hypothesis in the comments, maybe I am missing something. Thanks.
", "rawMarkdown": "**Edit:** See the comments for the discussion first. Probing expectations of 1/6 here is wrong. Instead use this [notebook](https://www.kaggle.com/keremt/competition-metric-map-0-5?scriptVersionId=64929405) for verification with LB probing below.\n\nI did some LB probing with all my 5 submissions from today here is a summary: \n\nHere is the [notebook](https://www.kaggle.com/keremt/understanding-evaluation/output?scriptVersionId=64877409)\n\n### LB Probes\n\n#### Probe 1: study (1,1,2,2 invalid boxes) and image (1,1,2,2 invalid boxes): \n\nLB:0.0 Expectation:0.0\n\n#### Probe 2: study (1,1,2,2 invalid boxes) and image (none,1,0,0,1,1): \n\nLB:0.0 Expectation:1/6 (Assuming equal class weights mAP)\n\nLB: 0.0 also supports the existing problem with image level score calculations. Many say it only contributes 0.001 but infact its not contributing at all, not sure when we will have an explanation about this.\n\n#### Probe 3: study (negative,1,1,0,0) and image (1,1,2,2 boxes): \n\nLB:0.050 Expectation:1/6 (Assuming equal class weights mAP)\n\nThis means mAP is weighted by the number of samples of that class in test set. Probe 4 and 5 supports this idea.\n\n#### Probe 4: study (typical,1,1,0,0) and image (1,1,2,2 boxes): \n\nLB:0.079 Expectation:1/6 (Assuming equal class weights mAP)\n\nThis means mAP is weighted.\n\nLB negative probe / LB typical probe = 0.050 / 0.079 = 0.63\n\nno. of negative in train / no. of typical in train = 1676 / 2855 = 0.58\n\nWe can see that ratios of LB scores and number of samples are very close. They are not exactly same since class distributions of training and test are not same, we probably have more negatives in test set.\n\n#### Probe 5: study (indeterminate,1,1,0,0) and image (1,1,2,2 boxes): \n\nLB:0.031 Expectation:1/6 (Assuming equal class weights mAP)\n\nThis means mAP is weighted.\n\nLB negative probe / LB indeterminate probe = 0.050 / 0.031 = 1.61\n\nno. of negative in train / no. of indeterminate in train = 1676 / 2855 = 1.59\n\n#### Probe 6: study (atypical,1,1,0,0) and image (1,1,2,2 boxes) (TODO): \n\nLB:0.031 Expectation:1/6 (Assuming equal class weights mAP)\n\nThis means mAP is weighted.\n\nLB negative probe / LB atypical probe = 0.050 / X = Y (Tomorrow: out of submissions)\n\nno. of negative in train / no. of atypical in train = 1676 / 2855 = 3.53\n\nX should be close to 0.050 / 3.53 ~= 0.014\n\n\n### What are weights for image and study level predictions?\n\nAbove we see that mAP is not weigthed equally, hence not 1/6. We can see the relative contributions within study level classes, but image level classes might be differently weighted compared to study level classes.\n\n\nWe kind of now know that study level predictions are weighted proportionally to their number of samples, if we also assume this for all the 6 classes then here is a rough calculation:\n\n\nLB (probe 3): 0.050 - Expectation: no. of negative in train / all train samples = 0.136\n\nLB (probe 4): 0.079 - Expectation: no. of negative in train / all train samples = 0.230\n\nLB (probe 4): 0.031 - Expectation: no. of negative in train / all train samples = 0.084\n\n\nWhen we assume equal weight between study and image level predictions numbers don't match. Here if we assume a weight of 0.35 for study level predictions:\n\nLB (probe 3): 0.050 - Expectation: 0.35*(no. of negative in train / all train samples) = 0.047\n\nLB (probe 4): 0.079 - Expectation: 0.35*(no. of negative in train / all train samples) = 0.080\n\nLB (probe 4): 0.031 - Expectation: 0.35*(no. of negative in train / all train samples) = 0.029\n\nAgain numbers are slightly different because train and test distribution differences.\n\n\n### Evaluation Formula Hypothesis\n\nScore: 0.35x(Weighted mAP Study) + 0.65x(Weighted mAP Image) ???\n\n\n(We can't do much probing around images since evaluation is currently broken.)\n\nEverything would be much easier if we could get more answers about metric calculation. It is well known that key to succeeding in any Kaggle competition is to construct a good validation set, and it would only be possible by having the questions around the metric answered.\n\nLet me know what you think about my hypothesis in the comments, maybe I am missing something. Thanks.", "votes": null }, { "id": "1337242", "postDate": "06/05/2021 13:46:25", "content": "Many say it only contributes 0.001 but infact its not contributing at all\nThe score of the none class is 0, but the score of the opacity class is 0.001.
\nAnd there is no difference in weight between the classes, only in the percentage of positive samples.
So, does it mean number of samples is impacting the mAP ?
", "rawMarkdown": "So, does it mean number of samples is impacting the **mAP** ?", "votes": null }, { "id": "1337509", "postDate": "06/05/2021 16:54:18", "content": "Thanks for the reply. After your comment I realize that it might not be possible to probe and expect AP=1.00 for (negative,typical,indeterminate or atypical) classes when we predict all studies as same class since precision and recall will depend on the number of positive cases of that class as you explained. Also, ordering will matter during AP calculation and since we predict all confidences as 1 we can't know how predictions will be ordered, and how TPs and FPs will be ordered. Different orders will have different AP and ranking TP higher will result in higher score. How to optimize mAP in a classification setting is little complicated for me at the moment, thanks for your comment which made me think about it :)
", "rawMarkdown": "Thanks for the reply. After your comment I realize that it might not be possible to probe and expect AP=1.00 for (negative,typical,indeterminate or atypical) classes when we predict all studies as same class since precision and recall will depend on the number of positive cases of that class as you explained. Also, ordering will matter during AP calculation and since we predict all confidences as 1 we can't know how predictions will be ordered, and how TPs and FPs will be ordered. Different orders will have different AP and ranking TP higher will result in higher score. How to optimize mAP in a classification setting is little complicated for me at the moment, thanks for your comment which made me think about it :)", "votes": null }, { "id": "1337530", "postDate": "06/05/2021 17:12:54", "content": "I think in case of this probing, although recall is 1 precision is the ratio of positive samples / number of predictions. To my understanding, thats why LB is proportional to number of positive cases. F1 would be more intuitive but with mAP in classification setting I am not very sure whats happening exactly.
", "rawMarkdown": "I think in case of this probing, although recall is 1 precision is the ratio of positive samples / number of predictions. To my understanding, thats why LB is proportional to number of positive cases. F1 would be more intuitive but with mAP in classification setting I am not very sure whats happening exactly.", "votes": null }, { "id": "1337703", "postDate": "06/05/2021 19:53:14", "content": "I am able to replicate my LB probing results using this notebook. Results using training dataset vs LB are very close ~-+0.001 indicating that distribution of training and public test set are very close.
", "rawMarkdown": "I am able to replicate my LB probing results using this [notebook](https://www.kaggle.com/keremt/competition-metric-map-0-5?scriptVersionId=64929087). Results using training dataset vs LB are very close ~-+0.001 indicating that distribution of training and public test set are very close.", "votes": null }, { "id": "1337774", "postDate": "06/05/2021 22:17:45", "content": "It certainly seems to be close to the train distribution. Thanks for the info!
", "rawMarkdown": "It certainly seems to be close to the train distribution. Thanks for the info!", "votes": null }, { "id": "1338563", "postDate": "06/06/2021 14:48:03", "content": "Thanks for sharing with the community!
", "rawMarkdown": "Thanks for sharing with the community!", "votes": null }, { "id": "1343203", "postDate": "06/10/2021 05:33:03", "content": "Great job!👍
", "rawMarkdown": "Great job!👍", "votes": null } ], "comments": [ { "id": 1337242, "author_name": "yujiariyasu", "author_url": "", "post_date": "06/05/2021 13:46:25", "content": "Many say it only contributes 0.001 but infact its not contributing at all\nThe score of the none class is 0, but the score of the opacity class is 0.001.
\nAnd there is no difference in weight between the classes, only in the percentage of positive samples.
So, does it mean number of samples is impacting the mAP ?
", "votes": null, "replies": [] }, { "id": 1337509, "author_name": "keremt", "author_url": "", "post_date": "06/05/2021 16:54:18", "content": "Thanks for the reply. After your comment I realize that it might not be possible to probe and expect AP=1.00 for (negative,typical,indeterminate or atypical) classes when we predict all studies as same class since precision and recall will depend on the number of positive cases of that class as you explained. Also, ordering will matter during AP calculation and since we predict all confidences as 1 we can't know how predictions will be ordered, and how TPs and FPs will be ordered. Different orders will have different AP and ranking TP higher will result in higher score. How to optimize mAP in a classification setting is little complicated for me at the moment, thanks for your comment which made me think about it :)
", "votes": null, "replies": [] }, { "id": 1337530, "author_name": "keremt", "author_url": "", "post_date": "06/05/2021 17:12:54", "content": "I think in case of this probing, although recall is 1 precision is the ratio of positive samples / number of predictions. To my understanding, thats why LB is proportional to number of positive cases. F1 would be more intuitive but with mAP in classification setting I am not very sure whats happening exactly.
", "votes": null, "replies": [] }, { "id": 1337703, "author_name": "keremt", "author_url": "", "post_date": "06/05/2021 19:53:14", "content": "I am able to replicate my LB probing results using this notebook. Results using training dataset vs LB are very close ~-+0.001 indicating that distribution of training and public test set are very close.
", "votes": null, "replies": [] }, { "id": 1337774, "author_name": "yujiariyasu", "author_url": "", "post_date": "06/05/2021 22:17:45", "content": "It certainly seems to be close to the train distribution. Thanks for the info!
", "votes": null, "replies": [] } ] }, { "id": 1338563, "author_name": "saurabhbagchi", "author_url": "", "post_date": "06/06/2021 14:48:03", "content": "Thanks for sharing with the community!
", "votes": null, "replies": [] }, { "id": 1343203, "author_name": "shangweichen", "author_url": "", "post_date": "06/10/2021 05:33:03", "content": "Great job!👍
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1336698": "**Edit:** See the comments for the discussion first. Probing expectations of 1/6 here is wrong. Instead use this [notebook](https://www.kaggle.com/keremt/competition-metric-map-0-5?scriptVersionId=64929405) for verification with LB probing below.\n\nI did some LB probing with all my 5 submissions from today here is a summary: \n\nHere is the [notebook](https://www.kaggle.com/keremt/understanding-evaluation/output?scriptVersionId=64877409)\n\n### LB Probes\n\n#### Probe 1: study (1,1,2,2 invalid boxes) and image (1,1,2,2 invalid boxes): \n\nLB:0.0 Expectation:0.0\n\n#### Probe 2: study (1,1,2,2 invalid boxes) and image (none,1,0,0,1,1): \n\nLB:0.0 Expectation:1/6 (Assuming equal class weights mAP)\n\nLB: 0.0 also supports the existing problem with image level score calculations. Many say it only contributes 0.001 but infact its not contributing at all, not sure when we will have an explanation about this.\n\n#### Probe 3: study (negative,1,1,0,0) and image (1,1,2,2 boxes): \n\nLB:0.050 Expectation:1/6 (Assuming equal class weights mAP)\n\nThis means mAP is weighted by the number of samples of that class in test set. Probe 4 and 5 supports this idea.\n\n#### Probe 4: study (typical,1,1,0,0) and image (1,1,2,2 boxes): \n\nLB:0.079 Expectation:1/6 (Assuming equal class weights mAP)\n\nThis means mAP is weighted.\n\nLB negative probe / LB typical probe = 0.050 / 0.079 = 0.63\n\nno. of negative in train / no. of typical in train = 1676 / 2855 = 0.58\n\nWe can see that ratios of LB scores and number of samples are very close. They are not exactly same since class distributions of training and test are not same, we probably have more negatives in test set.\n\n#### Probe 5: study (indeterminate,1,1,0,0) and image (1,1,2,2 boxes): \n\nLB:0.031 Expectation:1/6 (Assuming equal class weights mAP)\n\nThis means mAP is weighted.\n\nLB negative probe / LB indeterminate probe = 0.050 / 0.031 = 1.61\n\nno. of negative in train / no. of indeterminate in train = 1676 / 2855 = 1.59\n\n#### Probe 6: study (atypical,1,1,0,0) and image (1,1,2,2 boxes) (TODO): \n\nLB:0.031 Expectation:1/6 (Assuming equal class weights mAP)\n\nThis means mAP is weighted.\n\nLB negative probe / LB atypical probe = 0.050 / X = Y (Tomorrow: out of submissions)\n\nno. of negative in train / no. of atypical in train = 1676 / 2855 = 3.53\n\nX should be close to 0.050 / 3.53 ~= 0.014\n\n\n### What are weights for image and study level predictions?\n\nAbove we see that mAP is not weigthed equally, hence not 1/6. We can see the relative contributions within study level classes, but image level classes might be differently weighted compared to study level classes.\n\n\nWe kind of now know that study level predictions are weighted proportionally to their number of samples, if we also assume this for all the 6 classes then here is a rough calculation:\n\n\nLB (probe 3): 0.050 - Expectation: no. of negative in train / all train samples = 0.136\n\nLB (probe 4): 0.079 - Expectation: no. of negative in train / all train samples = 0.230\n\nLB (probe 4): 0.031 - Expectation: no. of negative in train / all train samples = 0.084\n\n\nWhen we assume equal weight between study and image level predictions numbers don't match. Here if we assume a weight of 0.35 for study level predictions:\n\nLB (probe 3): 0.050 - Expectation: 0.35*(no. of negative in train / all train samples) = 0.047\n\nLB (probe 4): 0.079 - Expectation: 0.35*(no. of negative in train / all train samples) = 0.080\n\nLB (probe 4): 0.031 - Expectation: 0.35*(no. of negative in train / all train samples) = 0.029\n\nAgain numbers are slightly different because train and test distribution differences.\n\n\n### Evaluation Formula Hypothesis\n\nScore: 0.35x(Weighted mAP Study) + 0.65x(Weighted mAP Image) ???\n\n\n(We can't do much probing around images since evaluation is currently broken.)\n\nEverything would be much easier if we could get more answers about metric calculation. It is well known that key to succeeding in any Kaggle competition is to construct a good validation set, and it would only be possible by having the questions around the metric answered.\n\nLet me know what you think about my hypothesis in the comments, maybe I am missing something. Thanks.", "1337242": "```\nMany say it only contributes 0.001 but infact its not contributing at all\n```\nThe score of the none class is 0, but the score of the opacity class is 0.001.\nAnd there is no difference in weight between the classes, only in the percentage of positive samples.", "1337499": "So, does it mean number of samples is impacting the **mAP** ?", "1337509": "Thanks for the reply. After your comment I realize that it might not be possible to probe and expect AP=1.00 for (negative,typical,indeterminate or atypical) classes when we predict all studies as same class since precision and recall will depend on the number of positive cases of that class as you explained. Also, ordering will matter during AP calculation and since we predict all confidences as 1 we can't know how predictions will be ordered, and how TPs and FPs will be ordered. Different orders will have different AP and ranking TP higher will result in higher score. How to optimize mAP in a classification setting is little complicated for me at the moment, thanks for your comment which made me think about it :)", "1337530": "I think in case of this probing, although recall is 1 precision is the ratio of positive samples / number of predictions. To my understanding, thats why LB is proportional to number of positive cases. F1 would be more intuitive but with mAP in classification setting I am not very sure whats happening exactly.", "1337703": "I am able to replicate my LB probing results using this [notebook](https://www.kaggle.com/keremt/competition-metric-map-0-5?scriptVersionId=64929087). Results using training dataset vs LB are very close ~-+0.001 indicating that distribution of training and public test set are very close.", "1337774": "It certainly seems to be close to the train distribution. Thanks for the info!", "1338563": "Thanks for sharing with the community!", "1343203": "Great job!👍" }, "source": "meta" }