In the current competition we have much more time available to make predictions. So, it seems, that we have to build more accurate models, instead of 'fast' models, that were used in the last competitions.
\n| Year | \nNumber of test records | \nRecord length, min | \nTotal, min | \nInference time limit, min | \n
|---|---|---|---|---|
| 2025 | \n700 | \n1 | \n700 | \n90 | \n
| 2024 | \n1100 | \n4 | \n4400 | \n120 | \n
| 2023 | \n200 | \n10 | \n2000 | \n120 | \n
| 2022 | \n5500 | \n1 | \n5500 | \n540 | \n
| 2021 | \n80 | \n10 | \n800 | \n540 | \n
I decided to start with using common models like BirdNet and Bird Vocalization Classifier.
\nI used birdnetlib in the starter notebook.
\nHowever, it shows pure performance giving the score of 0.610. The obvious reason is that this model doesn't cover all the existing species. It was trained on birds only and doesn't recognize shghum1. So it provides the predictions on 145 of 206 total species. Keeping this in mind, one can assess the public score of the model as \\( (0.61\\cdot206-61\\cdot0.5)/145 = 0.656 \\). This is not a top score, but it is somewhat higher than current public solutions.
BVC is implemented here. This NN is able to predict 143 of 206 species. Also it is several times slower than BirdNet. Even when executed in several threads, it takes about 10-12 sec to process one recording. So, just for test, I have dropped a quarter of each recording. This gave me the public score of 0.644. Recalculating the public score on the predicted data, we get
\n\\( (0.644\\cdot206-(206-143\\cdot0.75)\\cdot0.5)/(143\\cdot 0.75) = 0.776 \\). Not a top score, but better, that BirdNet.
Have anyone tried these models or something else? What are your results?
", "messages": [ { "id": 3148686, "postDate": "2025-03-13T12:31:24.050Z", "content": "In the current competition we have much more time available to make predictions. So, it seems, that we have to build more accurate models, instead of 'fast' models, that were used in the last competitions.
\n| Year | \nNumber of test records | \nRecord length, min | \nTotal, min | \nInference time limit, min | \n
|---|---|---|---|---|
| 2025 | \n700 | \n1 | \n700 | \n90 | \n
| 2024 | \n1100 | \n4 | \n4400 | \n120 | \n
| 2023 | \n200 | \n10 | \n2000 | \n120 | \n
| 2022 | \n5500 | \n1 | \n5500 | \n540 | \n
| 2021 | \n80 | \n10 | \n800 | \n540 | \n
I decided to start with using common models like BirdNet and Bird Vocalization Classifier.
\nI used birdnetlib in the starter notebook.
\nHowever, it shows pure performance giving the score of 0.610. The obvious reason is that this model doesn't cover all the existing species. It was trained on birds only and doesn't recognize shghum1. So it provides the predictions on 145 of 206 total species. Keeping this in mind, one can assess the public score of the model as \\( (0.61\\cdot206-61\\cdot0.5)/145 = 0.656 \\). This is not a top score, but it is somewhat higher than current public solutions.
BVC is implemented here. This NN is able to predict 143 of 206 species. Also it is several times slower than BirdNet. Even when executed in several threads, it takes about 10-12 sec to process one recording. So, just for test, I have dropped a quarter of each recording. This gave me the public score of 0.644. Recalculating the public score on the predicted data, we get
\n\\( (0.644\\cdot206-(206-143\\cdot0.75)\\cdot0.5)/(143\\cdot 0.75) = 0.776 \\). Not a top score, but better, that BirdNet.
Have anyone tried these models or something else? What are your results?
", "rawMarkdown": "In the current competition we have much more time available to make predictions. So, it seems, that we have to build more accurate models, instead of 'fast' models, that were used in the last competitions.\n\n| Year | Number of test records | Record length, min| Total, min| Inference time limit, min|\n| --- | --- |\n| **2025**| **700** | **1** | **700**| **90**|\n| 2024 | 1100 | 4 | 4400 | 120|\n| 2023 | 200 | 10 | 2000 | 120|\n| 2022 | 5500| 1 | 5500 | 540|\n| 2021 | 80| 10 | 800 | 540|\n\n\nI decided to start with using common models like BirdNet and Bird Vocalization Classifier. \n\n## BirdNet\nI used `birdnetlib` in [**the starter notebook**](https://www.kaggle.com/code/kdmitrie/bc25-birdnet-starter). \nHowever, it shows pure performance giving the score of 0.610. The obvious reason is that this model doesn't cover all the existing species. It was trained on birds only and doesn't recognize shghum1. So it provides the predictions on 145 of 206 total species. Keeping this in mind, one can assess the public score of the model as \\\\( (0.61\\cdot206-61\\cdot0.5)/145 = 0.656 \\\\). This is not a top score, but it is somewhat higher than current public solutions.\n\n## Bird Vocalization Classifier\nBVC is implemented [**here**](https://www.kaggle.com/code/kdmitrie/bc25-bvc-starter). This NN is able to predict 143 of 206 species. Also it is several times slower than BirdNet. Even when executed in several threads, it takes about 10-12 sec to process one recording. So, just for test, I have dropped a quarter of each recording. This gave me the public score of 0.644. Recalculating the public score on the predicted data, we get\n\\\\( (0.644\\cdot206-(206-143\\cdot0.75)\\cdot0.5)/(143\\cdot 0.75) = 0.776 \\\\). Not a top score, but better, that BirdNet.\n\nHave anyone tried these models or something else? What are your results?", "votes": 34 }, { "id": 3164567, "postDate": "2025-03-31T19:54:57.760Z", "content": "Could you help with output format? In BirdNet workbook output almost all values are zeros. In BVC only unknown are zeros. How could these two workbooks have similar score? None of them look like probabilities and don't sum to 1 in a row, however its a requirement.
", "rawMarkdown": "Could you help with output format? In BirdNet workbook output almost all values are zeros. In BVC only unknown are zeros. How could these two workbooks have similar score? None of them look like probabilities and don't sum to 1 in a row, however its a requirement.", "replies": [ { "id": 3166983, "postDate": "2025-04-01T07:37:55.453Z", "content": "In this competition, AUC ROC metrics is used. It doesn't need the model's outputs to be probabilities, and it is not sensitive to the exact values. It depends on the order of true and false detections.
\nA monotonic function doesn't change this order, so it doesn't matter if we use raw logits of model or process them with sigmoid. BVC produces logits while BirdNet produces probabilities, but the AUC ROC score may be similar.
Thank you for providing a birdnetlib starter notebook! Are there any ways to optimize the BVC model itself so we can run all the test soundscapes?
Yes, and the most obvious is to use the embeddings provided by BVC model. Please, have a look at the documentation. The model_outputs['embedding'] contains vector corresponding to the input audio. You can train your own classifier (not necessarily a neural network) to predict whatever you want.
Awesome, thank you! Building some simple supervised models on the embeddings first makes a lot of sense.
", "rawMarkdown": "Awesome, thank you! Building some simple supervised models on the embeddings first makes a lot of sense." }, { "id": 3153246, "postDate": "2025-03-18T15:04:05.957Z", "content": "Why use the embeddings rather than fully finetuning the model?
", "rawMarkdown": "Why use the embeddings rather than fully finetuning the model?", "votes": 1 }, { "id": 3153302, "postDate": "2025-03-18T16:23:46.193Z", "content": "Both can make sense, but since BVC is trained on bioacoustics (i.e. birds) the embeddings should already be pretty good.
\nAn upside of using the embeddings directly is that we can fit very lightweight models on it, like logistic regression or decision trees. It also allow you to iterate very quickly and explore differrent models. Finetuning the model will have a longer iteration cycle.
\nTo get started I would also recommend checking out openl3. An elegant way to get audio embeddings. Use content_type=\"env\" to access a model trained on environmental sound.
Could you help with output format? In BirdNet workbook output almost all values are zeros. In BVC only unknown are zeros. How could these two workbooks have similar score? None of them look like probabilities and don't sum to 1 in a row, however its a requirement.
", "votes": 0, "replies": [ { "id": 3166983, "author_name": "Konstantin Dmitriev", "author_url": "", "post_date": "2025-04-01T07:37:55.453000", "content": "In this competition, AUC ROC metrics is used. It doesn't need the model's outputs to be probabilities, and it is not sensitive to the exact values. It depends on the order of true and false detections.
\nA monotonic function doesn't change this order, so it doesn't matter if we use raw logits of model or process them with sigmoid. BVC produces logits while BirdNet produces probabilities, but the AUC ROC score may be similar.
Thank you for providing a birdnetlib starter notebook! Are there any ways to optimize the BVC model itself so we can run all the test soundscapes?
Yes, and the most obvious is to use the embeddings provided by BVC model. Please, have a look at the documentation. The model_outputs['embedding'] contains vector corresponding to the input audio. You can train your own classifier (not necessarily a neural network) to predict whatever you want.
Awesome, thank you! Building some simple supervised models on the embeddings first makes a lot of sense.
", "votes": 0, "replies": [] }, { "id": 3153246, "author_name": "thacrobatheskis", "author_url": "", "post_date": "2025-03-18T15:04:05.957000", "content": "Why use the embeddings rather than fully finetuning the model?
", "votes": 1, "replies": [] }, { "id": 3153302, "author_name": "Carlo", "author_url": "", "post_date": "2025-03-18T16:23:46.193000", "content": "Both can make sense, but since BVC is trained on bioacoustics (i.e. birds) the embeddings should already be pretty good.
\nAn upside of using the embeddings directly is that we can fit very lightweight models on it, like logistic regression or decision trees. It also allow you to iterate very quickly and explore differrent models. Finetuning the model will have a longer iteration cycle.
\nTo get started I would also recommend checking out openl3. An elegant way to get audio embeddings. Use content_type=\"env\" to access a model trained on environmental sound.