So - the competition describes:
\n\"a version of macro-averaged ROC-AUC that skips classes which have no true positive labels.\"
\nhttps://www.kaggle.com/code/metric/birdclef-roc-auc
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-exploring-scoring
That code would seem to suggest a single DF goes in for scoring - and then kicks out an LB score.
\nOn that assumption - all the soundscapes would be scored together.
\nIn this case - for a class to be \"skipped\" - it must not have any true positive labels in any of the soundscapes. Predicting a Purple Pigeon in a soundscape with no Purple Pigeons will result in your score being penalized (assuming another soundscape has a Purple Pigeon). Scores for a given species would be relative to all scores for that species across soundscapes.
\nAnother possibility is for each soundscape to be scored separately - and then have the scores for the soundscapes averaged together.
\nIn that case - any species not present in a given soundscape is exempt from scoring for that soundscape. Further - predictions for the same species across different soundscapes could be scaled or offset differently from each other without impact on the LB score.
\nDoes anyone have any thoughts on which it is (or something different)? Or better yet - is there something documented for the competition?
", "messages": [ { "id": "2793825", "postDate": "05/05/2024 01:42:13", "content": "So - the competition describes:
\n\"a version of macro-averaged ROC-AUC that skips classes which have no true positive labels.\"
\nhttps://www.kaggle.com/code/metric/birdclef-roc-auc
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-exploring-scoring
That code would seem to suggest a single DF goes in for scoring - and then kicks out an LB score.
\nOn that assumption - all the soundscapes would be scored together.
\nIn this case - for a class to be \"skipped\" - it must not have any true positive labels in any of the soundscapes. Predicting a Purple Pigeon in a soundscape with no Purple Pigeons will result in your score being penalized (assuming another soundscape has a Purple Pigeon). Scores for a given species would be relative to all scores for that species across soundscapes.
\nAnother possibility is for each soundscape to be scored separately - and then have the scores for the soundscapes averaged together.
\nIn that case - any species not present in a given soundscape is exempt from scoring for that soundscape. Further - predictions for the same species across different soundscapes could be scaled or offset differently from each other without impact on the LB score.
\nDoes anyone have any thoughts on which it is (or something different)? Or better yet - is there something documented for the competition?
", "rawMarkdown": "So - the competition describes: \n\"a version of macro-averaged ROC-AUC that skips classes which have no true positive labels.\"\nhttps://www.kaggle.com/code/metric/birdclef-roc-auc\nhttps://www.kaggle.com/code/richolson/birdclef-2024-exploring-scoring\n\nThat code would seem to suggest a single DF goes in for scoring - and then kicks out an LB score.\n\nOn that assumption - all the soundscapes would be scored together.\n\nIn this case - for a class to be \"skipped\" - it must not have any true positive labels in any of the soundscapes. Predicting a Purple Pigeon in a soundscape with no Purple Pigeons will result in your score being penalized (assuming another soundscape has a Purple Pigeon). Scores for a given species would be relative to all scores for that species across soundscapes.\n\nAnother possibility is for each soundscape to be scored separately - and then have the scores for the soundscapes averaged together.\n\nIn that case - any species not present in a given soundscape is exempt from scoring for that soundscape. Further - predictions for the same species across different soundscapes could be scaled or offset differently from each other without impact on the LB score.\n\nDoes anyone have any thoughts on which it is (or something different)? Or better yet - is there something documented for the competition?", "votes": null }, { "id": "2793902", "postDate": "05/05/2024 03:58:47", "content": "As far as I understand, complete results (solution and submission) are submitted to calculate the metric. That is, all the soundscapes.
OK - so I did some tests - and I got some curious results…
\nVersion 6: 0.61 LB \"Baseline\"
\nAn existing model I've used / is used for all other versions in this test.
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment
Version 7: 0.50 LB
\nI select a random number between 0 and 1 for each soundscape - and added it to all the predictions for that soundscape.
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment?scriptVersionId=175709383
Version 8: 0.61 LB (\"Best Score\"?!)
\nIn this notebook I choose a random number between 0.5 and 1 for each soundscape - and then -multiply- each prediction in that soundscape by it. So - each soundscape will randomly have its score scaled to between 50% and 100% of what the model predicted.
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment?scriptVersionId=175888223
Version 9: 0.60 LB
\nHere we scale each soundscapes prediction by a multiplier between 0.1 and 1. Each soundscape has all it's scored scaled between 10% and 100% of what the model predicted.
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment?scriptVersionId=175899536
So - I'm not 100% sure what this means - but maybe there are some clues about the nature of the test data / scoring. I think my code is correct…
\n0.50 LB score in Version 7 is what I'd expect for combined scoring.
\nBut.. I find that scaling predictions per-soundscape over a factor of 10x would have relatively little impact on score surprising. That could be explained if separated-by-soundscape scoring was used. Alternately - maybe it indicates something about the data (or is maybe a side-effect of doing these tests on a relatively-low-scoring model).
\nAnyways - tossing it out there for discussion….
", "rawMarkdown": "OK - so I did some tests - and I got some curious results...\n\nVersion 6: 0.61 LB \"Baseline\"\nAn existing model I've used / is used for all other versions in this test.\nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment\n\nVersion 7: 0.50 LB\nI select a random number between 0 and 1 for each soundscape - and added it to all the predictions for that soundscape. \nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment?scriptVersionId=175709383\n\nVersion 8: 0.61 LB (\"Best Score\"?!)\nIn this notebook I choose a random number between 0.5 and 1 for each soundscape - and then -multiply- each prediction in that soundscape by it. So - each soundscape will randomly have its score scaled to between 50% and 100% of what the model predicted. \nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment?scriptVersionId=175888223\n\nVersion 9: 0.60 LB\nHere we scale each soundscapes prediction by a multiplier between 0.1 and 1. Each soundscape has all it's scored scaled between 10% and 100% of what the model predicted.\nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment?scriptVersionId=175899536\n\nSo - I'm not 100% sure what this means - but maybe there are some clues about the nature of the test data / scoring. I think my code is correct...\n\n0.50 LB score in Version 7 is what I'd expect for combined scoring.\n\nBut.. I find that scaling predictions per-soundscape over a factor of 10x would have relatively little impact on score surprising. That could be explained if separated-by-soundscape scoring was used. Alternately - maybe it indicates something about the data (or is maybe a side-effect of doing these tests on a relatively-low-scoring model).\n\nAnyways - tossing it out there for discussion....", "votes": null } ], "comments": [ { "id": 2793902, "author_name": "sapr3s", "author_url": "", "post_date": "05/05/2024 03:58:47", "content": "As far as I understand, complete results (solution and submission) are submitted to calculate the metric. That is, all the soundscapes.
OK - so I did some tests - and I got some curious results…
\nVersion 6: 0.61 LB \"Baseline\"
\nAn existing model I've used / is used for all other versions in this test.
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment
Version 7: 0.50 LB
\nI select a random number between 0 and 1 for each soundscape - and added it to all the predictions for that soundscape.
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment?scriptVersionId=175709383
Version 8: 0.61 LB (\"Best Score\"?!)
\nIn this notebook I choose a random number between 0.5 and 1 for each soundscape - and then -multiply- each prediction in that soundscape by it. So - each soundscape will randomly have its score scaled to between 50% and 100% of what the model predicted.
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment?scriptVersionId=175888223
Version 9: 0.60 LB
\nHere we scale each soundscapes prediction by a multiplier between 0.1 and 1. Each soundscape has all it's scored scaled between 10% and 100% of what the model predicted.
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment?scriptVersionId=175899536
So - I'm not 100% sure what this means - but maybe there are some clues about the nature of the test data / scoring. I think my code is correct…
\n0.50 LB score in Version 7 is what I'd expect for combined scoring.
\nBut.. I find that scaling predictions per-soundscape over a factor of 10x would have relatively little impact on score surprising. That could be explained if separated-by-soundscape scoring was used. Alternately - maybe it indicates something about the data (or is maybe a side-effect of doing these tests on a relatively-low-scoring model).
\nAnyways - tossing it out there for discussion….
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2793825": "So - the competition describes: \n\"a version of macro-averaged ROC-AUC that skips classes which have no true positive labels.\"\nhttps://www.kaggle.com/code/metric/birdclef-roc-auc\nhttps://www.kaggle.com/code/richolson/birdclef-2024-exploring-scoring\n\nThat code would seem to suggest a single DF goes in for scoring - and then kicks out an LB score.\n\nOn that assumption - all the soundscapes would be scored together.\n\nIn this case - for a class to be \"skipped\" - it must not have any true positive labels in any of the soundscapes. Predicting a Purple Pigeon in a soundscape with no Purple Pigeons will result in your score being penalized (assuming another soundscape has a Purple Pigeon). Scores for a given species would be relative to all scores for that species across soundscapes.\n\nAnother possibility is for each soundscape to be scored separately - and then have the scores for the soundscapes averaged together.\n\nIn that case - any species not present in a given soundscape is exempt from scoring for that soundscape. Further - predictions for the same species across different soundscapes could be scaled or offset differently from each other without impact on the LB score.\n\nDoes anyone have any thoughts on which it is (or something different)? Or better yet - is there something documented for the competition?", "2793902": "As far as I understand, complete results (`solution` and `submission`) are submitted to calculate the metric. That is, all the soundscapes.", "2795881": "OK - so I did some tests - and I got some curious results...\n\nVersion 6: 0.61 LB \"Baseline\"\nAn existing model I've used / is used for all other versions in this test.\nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment\n\nVersion 7: 0.50 LB\nI select a random number between 0 and 1 for each soundscape - and added it to all the predictions for that soundscape. \nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment?scriptVersionId=175709383\n\nVersion 8: 0.61 LB (\"Best Score\"?!)\nIn this notebook I choose a random number between 0.5 and 1 for each soundscape - and then -multiply- each prediction in that soundscape by it. So - each soundscape will randomly have its score scaled to between 50% and 100% of what the model predicted. \nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment?scriptVersionId=175888223\n\nVersion 9: 0.60 LB\nHere we scale each soundscapes prediction by a multiplier between 0.1 and 1. Each soundscape has all it's scored scaled between 10% and 100% of what the model predicted.\nhttps://www.kaggle.com/code/richolson/birdclef-2024-run-v2-scoring-experiment?scriptVersionId=175899536\n\nSo - I'm not 100% sure what this means - but maybe there are some clues about the nature of the test data / scoring. I think my code is correct...\n\n0.50 LB score in Version 7 is what I'd expect for combined scoring.\n\nBut.. I find that scaling predictions per-soundscape over a factor of 10x would have relatively little impact on score surprising. That could be explained if separated-by-soundscape scoring was used. Alternately - maybe it indicates something about the data (or is maybe a side-effect of doing these tests on a relatively-low-scoring model).\n\nAnyways - tossing it out there for discussion...." }, "source": "meta" }