Hi!
\nIn some discussions, we can notice that people struggle with 2 problems:
\nOne possible reason, which already was mentioned in many posts, is the domain shift between Xeno Canto recordings and test soundscapes. But, I suspect that this reason is not only one and can be even not crucial as the next one
\nIf we take a look at competition metric a version of macro-averaged ROC-AUC that skips classes which have no true positive labels - we can notice that skips classes which have no true positive labels. And this is really important!
On local validation we might use stratified split by primary_label and we will have ALL classes with at least one true positive label for EACH fold. So in such a case competition metric will be equivalent to simple macro-averaged ROC-AUC. But it is not the case for Public or/and Private subset. On these subsets we might expect \"missing classes\" (classes which does not have at least one true positive label) and they will be just missed while metric computation
Now let's module next situation: We have 3 subset of classes: C_A, C_B, C_K and 2 models: M_I, M_P. Let len(C_A) > len(C_K) and M_I outperforms M_P on C_A BUT M_P outperforms M_I on C_K and both models perform same on C_B. But there no samples, which contains C_A on Public (or just a few classes are present). In such a case we will have much better local CV for M_I but much worse Public for M_I.
\nSuch behaviour is more probable for Public set, because it is smaller. But in such case there no reason to rely on Public score and no way to find good correlation, without sophisticated Public probing
\nSorry for such a long read :) But let's finalize it with some questions to @stefankahl and @tomdenton:
\nIf the answer for last question No - I completely understand
\nLooking forward to your answers!
\nP.S.
\nSorry if I have missed it somewhere in other Discussion
Hi!
\nIn some discussions, we can notice that people struggle with 2 problems:
\nOne possible reason, which already was mentioned in many posts, is the domain shift between Xeno Canto recordings and test soundscapes. But, I suspect that this reason is not only one and can be even not crucial as the next one
\nIf we take a look at competition metric a version of macro-averaged ROC-AUC that skips classes which have no true positive labels - we can notice that skips classes which have no true positive labels. And this is really important!
On local validation we might use stratified split by primary_label and we will have ALL classes with at least one true positive label for EACH fold. So in such a case competition metric will be equivalent to simple macro-averaged ROC-AUC. But it is not the case for Public or/and Private subset. On these subsets we might expect \"missing classes\" (classes which does not have at least one true positive label) and they will be just missed while metric computation
Now let's module next situation: We have 3 subset of classes: C_A, C_B, C_K and 2 models: M_I, M_P. Let len(C_A) > len(C_K) and M_I outperforms M_P on C_A BUT M_P outperforms M_I on C_K and both models perform same on C_B. But there no samples, which contains C_A on Public (or just a few classes are present). In such a case we will have much better local CV for M_I but much worse Public for M_I.
\nSuch behaviour is more probable for Public set, because it is smaller. But in such case there no reason to rely on Public score and no way to find good correlation, without sophisticated Public probing
\nSorry for such a long read :) But let's finalize it with some questions to @stefankahl and @tomdenton:
\nIf the answer for last question No - I completely understand
\nLooking forward to your answers!
\nP.S.
\nSorry if I have missed it somewhere in other Discussion
I have a bug in code evaluate => OOF CV ~ 0.67, I think I've found a eval method correlate with PB. But no, after fix this bug, OOF CV ~ 0.982 😅
", "rawMarkdown": "I have a bug in code evaluate => OOF CV ~ 0.67, I think I've found a eval method correlate with PB. But no, after fix this bug, OOF CV ~ 0.982 😅", "votes": null }, { "id": "2809369", "postDate": "05/12/2024 17:26:31", "content": "Hello!
\nWe're ultimately interested in understanding how to better transfer lab-trained models to real-world data. This means that, like in the real world, you have a list of possible species, but no idea which ones or how many actually appear in the test set. (More generally, you can think of this as a label distribution shift, where the probability of any given label in the test set could got up or down from the training data, but can fall to an epsilon small enough that it is unobserved in the data.)
", "rawMarkdown": "Hello!\n\nWe're ultimately interested in understanding how to better transfer lab-trained models to real-world data. This means that, like in the real world, you have a list of possible species, but no idea which ones or how many actually appear in the test set. (More generally, you can think of this as a label distribution shift, where the probability of any given label in the test set could got up or down from the training data, but can fall to an epsilon small enough that it is unobserved in the data.)", "votes": null }, { "id": "2809373", "postDate": "05/12/2024 17:27:58", "content": "Not all species from the train data have a label in the test data. Which ones - that's for you to find out :)
\nEven though this might seem weird, it reflects the actual real-world use case where we do have a good understanding of which species to expect, but we don't know exactly if they're present in the recordings we made. So we have to train with all species regardless.
", "rawMarkdown": "Not all species from the train data have a label in the test data. Which ones - that's for you to find out :) \n\nEven though this might seem weird, it reflects the actual real-world use case where we do have a good understanding of which species to expect, but we don't know exactly if they're present in the recordings we made. So we have to train with all species regardless.", "votes": null }, { "id": "2809408", "postDate": "05/12/2024 17:50:50", "content": "Posted within a minute of each other. Jinx!
", "rawMarkdown": "Posted within a minute of each other. Jinx!", "votes": null }, { "id": "2809433", "postDate": "05/12/2024 18:19:33", "content": "Thanks for your reply. I understand the purpose of holding this competition.
\nI think what really matters is the third question.
\n\nDo we expect to have at least one sample for each class on Private and Public union ?
\n
Or else, the private leaderboard will result to be a dice role because of the difference in distribution.
\nI appreciate if you can give a further explaination on it.
I see! It makes perfect sense. But maybe metric choice is not optimal for such a case. Previous year metric with padding (or the same analog for ROC AUC) makes more sense for me
\nAnyway thanks for clarclarification
", "rawMarkdown": "I see! It makes perfect sense. But maybe metric choice is not optimal for such a case. Previous year metric with padding (or the same analog for ROC AUC) makes more sense for me\n\nAnyway thanks for clarclarification", "votes": null }, { "id": "2809481", "postDate": "05/12/2024 19:25:28", "content": "There were two metric changes here: First, in previous years, we have used mAP, which is extremely punishing in the low-data regime. In fact, you can show that changing the label balance with mAP has a direct influence on the outcome, which makes it hard to disentangle model performance from label balance. This, in turn, means that when you look at a list of per-species mAP scores, there's no way to pull out guesses about what might be affecting model performance.
\n(In more detail: mAP generalizes mean-reciprocal-rank, so you get a non-linear punishment for ranking negative examples higher. And if you've got more negatives, it's more chances for ranking them higher. And if you've got very few positives, it's less chances to get 'easier' examples to put at the top of the list.)
\nROC-AUC, on the other hand, is basically coming from mean rank, and is thus linear. It isn't biased by label balance, so you can more directly compare cross-species scores. And it has a lovely probabilistic interpretation which makes it much easier to reason about: ROC-AUC is the probability that a uniformly-chosen positive example has a higher score/rank than a uniformly chosen negative example[1]. (Because the positive and negative are chosen uniformly and independently, you can see that there's no label balance bias right in the definition!) This is a far better definition than 'integrate over all thresholds,' IMO; for example, you can use it to (carefully) construct confidence intervals over ROC-AUCs in the low-data regime, and get a sense of whether one species' ROC-AUC is /actually/ better than another's, or just under-sampled.
\nYou can also express the class-averaged roc-auc as a probabilistic process: choose a species uniformly, and then pick a positive and negative example uniformly at random. Then the overall score is the expected value of that process. You can modify the metric by different class selection strategies: eg, weighting the class choice by dataset prevalence. We have chosen to weight all species which appear in the dataset equally, to avoid focusing overly on the most common species. And any species which do not appear have zero weight, so do not affect the score. (Padding, by contrast, doesn't have such a nice probabilistic interpretation, so will end up muddying the waters for cross-species comparison.)
\nOf course, since the scores and submissions are hidden, y'all don't get to do that kind of per-species analysis, but we do, and will! :)
\n(Thank you for coming to my TED Talk, brought to you by this fine Wendy's establishment.)
", "rawMarkdown": "There were two metric changes here: First, in previous years, we have used mAP, which is extremely punishing in the low-data regime. In fact, you can show that changing the label balance with mAP has a direct influence on the outcome, which makes it hard to disentangle model performance from label balance. This, in turn, means that when you look at a list of per-species mAP scores, there's no way to pull out guesses about what might be affecting model performance.\n\n(In more detail: mAP generalizes mean-reciprocal-rank, so you get a non-linear punishment for ranking negative examples higher. And if you've got more negatives, it's more chances for ranking them higher. And if you've got very few positives, it's less chances to get 'easier' examples to put at the top of the list.)\n\nROC-AUC, on the other hand, is basically coming from mean rank, and is thus linear. It isn't biased by label balance, so you can more directly compare cross-species scores. And it has a lovely probabilistic interpretation which makes it much easier to reason about: ROC-AUC is the probability that a uniformly-chosen positive example has a higher score/rank than a uniformly chosen negative example[[1](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Probabilistic_interpretation)]. (Because the positive and negative are chosen uniformly and independently, you can see that there's no label balance bias right in the definition!) This is a far better definition than 'integrate over all thresholds,' IMO; for example, you can use it to (carefully) construct confidence intervals over ROC-AUCs in the low-data regime, and get a sense of whether one species' ROC-AUC is /actually/ better than another's, or just under-sampled.\n\nYou can also express the class-averaged roc-auc as a probabilistic process: choose a species uniformly, and then pick a positive and negative example uniformly at random. Then the overall score is the expected value of that process. You can modify the metric by different class selection strategies: eg, weighting the class choice by dataset prevalence. We have chosen to weight all species which appear in the dataset equally, to avoid focusing overly on the most common species. And any species which do not appear have zero weight, so do not affect the score. (Padding, by contrast, doesn't have such a nice probabilistic interpretation, so will end up muddying the waters for cross-species comparison.) \n\nOf course, since the scores and submissions are hidden, y'all don't get to do that kind of per-species analysis, but we do, and will! :)\n\n(Thank you for coming to my TED Talk, brought to you by this fine Wendy's establishment.)", "votes": null }, { "id": "2811666", "postDate": "05/13/2024 20:03:46", "content": "Thanks for your idea
\nI need ur upvote on my notebook & datasets
I have a bug in code evaluate => OOF CV ~ 0.67, I think I've found a eval method correlate with PB. But no, after fix this bug, OOF CV ~ 0.982 😅
", "votes": null, "replies": [] }, { "id": 2809369, "author_name": "tomdenton", "author_url": "", "post_date": "05/12/2024 17:26:31", "content": "Hello!
\nWe're ultimately interested in understanding how to better transfer lab-trained models to real-world data. This means that, like in the real world, you have a list of possible species, but no idea which ones or how many actually appear in the test set. (More generally, you can think of this as a label distribution shift, where the probability of any given label in the test set could got up or down from the training data, but can fall to an epsilon small enough that it is unobserved in the data.)
", "votes": null, "replies": [ { "id": 2809433, "author_name": "honglihang", "author_url": "", "post_date": "05/12/2024 18:19:33", "content": "Thanks for your reply. I understand the purpose of holding this competition.
\nI think what really matters is the third question.
\n\nDo we expect to have at least one sample for each class on Private and Public union ?
\n
Or else, the private leaderboard will result to be a dice role because of the difference in distribution.
\nI appreciate if you can give a further explaination on it.
I see! It makes perfect sense. But maybe metric choice is not optimal for such a case. Previous year metric with padding (or the same analog for ROC AUC) makes more sense for me
\nAnyway thanks for clarclarification
", "votes": null, "replies": [ { "id": 2809481, "author_name": "tomdenton", "author_url": "", "post_date": "05/12/2024 19:25:28", "content": "There were two metric changes here: First, in previous years, we have used mAP, which is extremely punishing in the low-data regime. In fact, you can show that changing the label balance with mAP has a direct influence on the outcome, which makes it hard to disentangle model performance from label balance. This, in turn, means that when you look at a list of per-species mAP scores, there's no way to pull out guesses about what might be affecting model performance.
\n(In more detail: mAP generalizes mean-reciprocal-rank, so you get a non-linear punishment for ranking negative examples higher. And if you've got more negatives, it's more chances for ranking them higher. And if you've got very few positives, it's less chances to get 'easier' examples to put at the top of the list.)
\nROC-AUC, on the other hand, is basically coming from mean rank, and is thus linear. It isn't biased by label balance, so you can more directly compare cross-species scores. And it has a lovely probabilistic interpretation which makes it much easier to reason about: ROC-AUC is the probability that a uniformly-chosen positive example has a higher score/rank than a uniformly chosen negative example[1]. (Because the positive and negative are chosen uniformly and independently, you can see that there's no label balance bias right in the definition!) This is a far better definition than 'integrate over all thresholds,' IMO; for example, you can use it to (carefully) construct confidence intervals over ROC-AUCs in the low-data regime, and get a sense of whether one species' ROC-AUC is /actually/ better than another's, or just under-sampled.
\nYou can also express the class-averaged roc-auc as a probabilistic process: choose a species uniformly, and then pick a positive and negative example uniformly at random. Then the overall score is the expected value of that process. You can modify the metric by different class selection strategies: eg, weighting the class choice by dataset prevalence. We have chosen to weight all species which appear in the dataset equally, to avoid focusing overly on the most common species. And any species which do not appear have zero weight, so do not affect the score. (Padding, by contrast, doesn't have such a nice probabilistic interpretation, so will end up muddying the waters for cross-species comparison.)
\nOf course, since the scores and submissions are hidden, y'all don't get to do that kind of per-species analysis, but we do, and will! :)
\n(Thank you for coming to my TED Talk, brought to you by this fine Wendy's establishment.)
", "votes": null, "replies": [] } ] } ] }, { "id": 2809373, "author_name": "stefankahl", "author_url": "", "post_date": "05/12/2024 17:27:58", "content": "Not all species from the train data have a label in the test data. Which ones - that's for you to find out :)
\nEven though this might seem weird, it reflects the actual real-world use case where we do have a good understanding of which species to expect, but we don't know exactly if they're present in the recordings we made. So we have to train with all species regardless.
", "votes": null, "replies": [ { "id": 2809408, "author_name": "tomdenton", "author_url": "", "post_date": "05/12/2024 17:50:50", "content": "Posted within a minute of each other. Jinx!
", "votes": null, "replies": [] } ] }, { "id": 2811666, "author_name": "xyznihal", "author_url": "", "post_date": "05/13/2024 20:03:46", "content": "Thanks for your idea
\nI need ur upvote on my notebook & datasets