![\"Figure](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4358411%2F0534dd3b14db64f13db6f484182e96d9%2FScreenshot%202024-04-08%20at%208.47.30AM.png?generation=1712591269797592&alt=media\")
I got the metric from here and I can see my CV around ~0.99 but the LB is ~0.6.
\nBecause AUC of 0.6 indicates that model is barely getting the ranking of predictions correct, I wanted to confirm if the implementation of metric provided is correct, or if submission ground truth is somewhat incorrect, or if its just me making a mistake in either CV notebook or inference notebook getting 0.99 CV and 0.6 LB or if the test dataset has extremely hard samples
", "messages": [ { "id": "2738533", "postDate": "04/06/2024 11:55:50", "content": "I got the metric from here and I can see my CV around ~0.99 but the LB is ~0.6.
\nBecause AUC of 0.6 indicates that model is barely getting the ranking of predictions correct, I wanted to confirm if the implementation of metric provided is correct, or if submission ground truth is somewhat incorrect, or if its just me making a mistake in either CV notebook or inference notebook getting 0.99 CV and 0.6 LB or if the test dataset has extremely hard samples
", "rawMarkdown": "I got the metric from [here](https://www.kaggle.com/code/metric/birdclef-roc-auc) and I can see my CV around ~0.99 but the LB is ~0.6.\n\nBecause AUC of 0.6 indicates that model is barely getting the ranking of predictions correct, I wanted to confirm if the implementation of metric provided is correct, or if submission ground truth is somewhat incorrect, or if its just me making a mistake in either CV notebook or inference notebook getting 0.99 CV and 0.6 LB or if the test dataset has extremely hard samples", "votes": null }, { "id": "2738586", "postDate": "04/06/2024 12:36:56", "content": "I’m curious, on what data do you calculate the CV? We don't have test data. And the training ones are much better in quality than the test ones (judging by past competitions)
", "rawMarkdown": "I’m curious, on what data do you calculate the CV? We don't have test data. And the training ones are much better in quality than the test ones (judging by past competitions)", "votes": null }, { "id": "2738588", "postDate": "04/06/2024 12:37:59", "content": "I have similar observations, my CV score is around 0.975 and LB is around 0.6
", "rawMarkdown": "I have similar observations, my CV score is around 0.975 and LB is around 0.6", "votes": null }, { "id": "2738633", "postDate": "04/06/2024 13:07:06", "content": "Split is a stratified 5 Fold and for each file first 5 second of audio is used for validation, this is the most common and pretty much the consistent way in past birdclef competitions to calculate CV
", "rawMarkdown": "Split is a stratified 5 Fold and for each file first 5 second of audio is used for validation, this is the most common and pretty much the consistent way in past birdclef competitions to calculate CV", "votes": null }, { "id": "2739050", "postDate": "04/06/2024 19:33:49", "content": "I got val_auc at 0.82 lb at 0.63. This is without using unlabeled data, but with data from previous competitions
", "rawMarkdown": "I got val_auc at 0.82 lb at 0.63. This is without using unlabeled data, but with data from previous competitions", "votes": null }, { "id": "2741562", "postDate": "04/08/2024 13:01:33", "content": "Any thoughts on this @tomdenton?
", "rawMarkdown": "Any thoughts on this @tomdenton?", "votes": null }, { "id": "2741799", "postDate": "04/08/2024 15:49:16", "content": "Domain shift is hard. :)
\nIt's very common for models to do well on held-out Xeno-Canto data and then fall on their face on passive acoustic (PAM) data.
\nSee Figure 4 here, for example: https://arxiv.org/pdf/2312.07439.pdf
\nThe yellow line is eval on held-out XC data, the green line is held-out XC data with label balance shifted to match the class distribution of the Powdermill dataset, and the red line is for the actual Powdermill PAM dataset.
What is weird is that the evaluation curves are flat.
", "rawMarkdown": "What is weird is that the evaluation curves are flat.", "votes": null }, { "id": "2764104", "postDate": "04/20/2024 22:41:19", "content": "@tomdenton how can the croc-auc metric vary with label rebalance? Isn't it rather a change in the set of species included in the evaluation?
", "rawMarkdown": "tomdenton how can the croc-auc metric vary with label rebalance? Isn't it rather a change in the set of species included in the evaluation?", "votes": null }, { "id": "2768384", "postDate": "04/22/2024 19:59:24", "content": "There's a couple things going on here which might contribute to a difference between eval (iid) and eval (label-shifted).
\nFirst, I think you're likely correct that the main effect is focusing on Powdermill species, instead of general XC species. ROC-AUC should (with some asterisks) be invariant to pos/neg label balance within a class.
\nSecond, the asterisk - When there are very few positive examples, the ROC-AUC measurement is likely to be noisy. eg, if we only have one positive example and it is a relatively low-ranked one, we'll see a different ROC-AUC than if we happened to pick a high-ranked example. So, when we label balance, we lose some examples per class, and wind up with noisier ROC-AUC estimates.
\nThird, the paper is using the geometric mean of ROC-AUC over classes, which accentuates the contribution of the 'worst' classes. This will also accentuate the impact of 'negative' noisiness in the ROC-AUC computation.
\nWe were (with this ablation) trying to disentangle the effects of label shift from covariate shift; the overall story in the figure is that most of the drop in metrics is due to covariate shift. (It's also worth keeping in mind that the actual task in the paper isn't classification but retrieval, which tends to be harder/noisier than classification.)
\nHope that helps.
", "rawMarkdown": "There's a couple things going on here which might contribute to a difference between eval (iid) and eval (label-shifted).\n\nFirst, I think you're likely correct that the main effect is focusing on Powdermill species, instead of general XC species. ROC-AUC should (with some asterisks) be invariant to pos/neg label balance within a class.\n\nSecond, the asterisk - When there are very few positive examples, the ROC-AUC measurement is likely to be noisy. eg, if we only have one positive example and it is a relatively low-ranked one, we'll see a different ROC-AUC than if we happened to pick a high-ranked example. So, when we label balance, we lose some examples per class, and wind up with noisier ROC-AUC estimates.\n\nThird, the paper is using the geometric mean of ROC-AUC over classes, which accentuates the contribution of the 'worst' classes. This will also accentuate the impact of 'negative' noisiness in the ROC-AUC computation.\n\nWe were (with this ablation) trying to disentangle the effects of label shift from covariate shift; the overall story in the figure is that most of the drop in metrics is due to covariate shift. (It's also worth keeping in mind that the actual task in the paper isn't classification but retrieval, which tends to be harder/noisier than classification.)\n\nHope that helps.", "votes": null }, { "id": "2768458", "postDate": "04/22/2024 21:08:53", "content": "It does help, thanks! I missed the geo mean bit when I read the paper.
\nI was looking for BIRB code but it is not public yet. Will you keep it private till the end for this competition?
", "rawMarkdown": "It does help, thanks! I missed the geo mean bit when I read the paper.\n\nI was looking for BIRB code but it is not public yet. Will you keep it private till the end for this competition?", "votes": null }, { "id": "2768487", "postDate": "04/22/2024 21:48:09", "content": "It's already in our Github repository. There's a readme here:
\nhttps://docs.google.com/document/d/1RasVkxIKKlUToFlJ8gZxaHqIE-mMy9G1MZwfK98Gb-I/edit#heading=h.3jyo0amqlt70
The main part of the code for BIRB is here:
\nhttps://github.com/google-research/perch/tree/main/chirp/eval
Did you have solved this question？
", "rawMarkdown": "Did you have solved this question？", "votes": null } ], "comments": [ { "id": 2738586, "author_name": "sapr3s", "author_url": "", "post_date": "04/06/2024 12:36:56", "content": "I’m curious, on what data do you calculate the CV? We don't have test data. And the training ones are much better in quality than the test ones (judging by past competitions)
", "votes": null, "replies": [ { "id": 2738633, "author_name": "harshitsheoran", "author_url": "", "post_date": "04/06/2024 13:07:06", "content": "Split is a stratified 5 Fold and for each file first 5 second of audio is used for validation, this is the most common and pretty much the consistent way in past birdclef competitions to calculate CV
", "votes": null, "replies": [] } ] }, { "id": 2738588, "author_name": "martynoveduard", "author_url": "", "post_date": "04/06/2024 12:37:59", "content": "I have similar observations, my CV score is around 0.975 and LB is around 0.6
", "votes": null, "replies": [ { "id": 2790858, "author_name": "deepshark", "author_url": "", "post_date": "05/03/2024 11:12:12", "content": "Did you have solved this question？
", "votes": null, "replies": [] } ] }, { "id": 2739050, "author_name": "aikhmelnytskyy", "author_url": "", "post_date": "04/06/2024 19:33:49", "content": "I got val_auc at 0.82 lb at 0.63. This is without using unlabeled data, but with data from previous competitions
", "votes": null, "replies": [] }, { "id": 2741562, "author_name": "hichambellafkir", "author_url": "", "post_date": "04/08/2024 13:01:33", "content": "Any thoughts on this @tomdenton?
", "votes": null, "replies": [ { "id": 2741799, "author_name": "tomdenton", "author_url": "", "post_date": "04/08/2024 15:49:16", "content": "Domain shift is hard. :)
\nIt's very common for models to do well on held-out Xeno-Canto data and then fall on their face on passive acoustic (PAM) data.
\nSee Figure 4 here, for example: https://arxiv.org/pdf/2312.07439.pdf
\nThe yellow line is eval on held-out XC data, the green line is held-out XC data with label balance shifted to match the class distribution of the Powdermill dataset, and the red line is for the actual Powdermill PAM dataset.
What is weird is that the evaluation curves are flat.
", "votes": null, "replies": [] }, { "id": 2764104, "author_name": "cpmpml", "author_url": "", "post_date": "04/20/2024 22:41:19", "content": "@tomdenton how can the croc-auc metric vary with label rebalance? Isn't it rather a change in the set of species included in the evaluation?
", "votes": null, "replies": [ { "id": 2768384, "author_name": "tomdenton", "author_url": "", "post_date": "04/22/2024 19:59:24", "content": "There's a couple things going on here which might contribute to a difference between eval (iid) and eval (label-shifted).
\nFirst, I think you're likely correct that the main effect is focusing on Powdermill species, instead of general XC species. ROC-AUC should (with some asterisks) be invariant to pos/neg label balance within a class.
\nSecond, the asterisk - When there are very few positive examples, the ROC-AUC measurement is likely to be noisy. eg, if we only have one positive example and it is a relatively low-ranked one, we'll see a different ROC-AUC than if we happened to pick a high-ranked example. So, when we label balance, we lose some examples per class, and wind up with noisier ROC-AUC estimates.
\nThird, the paper is using the geometric mean of ROC-AUC over classes, which accentuates the contribution of the 'worst' classes. This will also accentuate the impact of 'negative' noisiness in the ROC-AUC computation.
\nWe were (with this ablation) trying to disentangle the effects of label shift from covariate shift; the overall story in the figure is that most of the drop in metrics is due to covariate shift. (It's also worth keeping in mind that the actual task in the paper isn't classification but retrieval, which tends to be harder/noisier than classification.)
\nHope that helps.
", "votes": null, "replies": [ { "id": 2768458, "author_name": "cpmpml", "author_url": "", "post_date": "04/22/2024 21:08:53", "content": "It does help, thanks! I missed the geo mean bit when I read the paper.
\nI was looking for BIRB code but it is not public yet. Will you keep it private till the end for this competition?
", "votes": null, "replies": [ { "id": 2768487, "author_name": "tomdenton", "author_url": "", "post_date": "04/22/2024 21:48:09", "content": "It's already in our Github repository. There's a readme here:
\nhttps://docs.google.com/document/d/1RasVkxIKKlUToFlJ8gZxaHqIE-mMy9G1MZwfK98Gb-I/edit#heading=h.3jyo0amqlt70
The main part of the code for BIRB is here:
\nhttps://github.com/google-research/perch/tree/main/chirp/eval