\n
![\"image\"](\"https://user-images.githubusercontent.com/57013219/170329125-532a0640-cb54-4a81-9d8a-fadd4721d6ae.png\")
First of all, thanks to Kaggle and Cornell Lab of Ornithology for organizing this competition.
\nI was lucky enough to team-up with UEMU, it was a fruitful team-up with lots of ideas coming from the both sides.
\nWe've worked hard until the end and thanks to that we managed to secure 3rd place, congratz to all of the winners!
Considering the situation, it seems necessary to mention that we didn't use BirdNet model.
\nExternal data we used:
\nfreefield1010, aicrowd2020 and nocall part of BirdCLEF 2021 soundscapes to mix 2022 samples with background noise and impove robustness.
Now for our approach, the key points to build strong and reliable pipeline are the following:
\nLet's get down to our best finding during competition
\nWhen we merged together, we looked on OOFs predictions produced by our models and found out that SED w/ focal-loss performs very differently compared to mentioned CNN trained w/ BCELoss depending on number of training samples
\nFrom our observations, SED models w/ focal-loss tend to make more conservative predictions, and due to the loss design they don't miss small classes:
\nHere in blue you can see SED model w/ focal-loss, and in pink CNN model trained w/ BCE loss
\n
The same can be said about advantage of models which used BCE loss during training over models with focal-loss for the large classes
\n![\"image\"](\"https://user-images.githubusercontent.com/57013219/170329065-9a9d4da1-1660-46d4-b25e-9419f451f63d.png\")
Therefore, we divided the birds into two groups according to the number of data and manual inspection of distribution plots of target-data as above, and used different models for them.
\nIt appears that it's optimal to include birds with number of training samples >= 10 to the Group1, and all other birds into Group2.
\nGroup1: 14birds ['jabwar', 'yefcan', 'skylar', 'akiapo', 'apapan', 'barpet', 'elepai', 'iiwi', 'houfin', 'omao', 'warwhe1', 'aniani', 'hawama', 'hawcre'],
\nfor them we ended up using CNN + SED models which were trained using BCE loss.
Group2: 7birds, ['crehon', 'ercfra', 'hawgoo', 'hawhaw', 'hawpet1', 'maupar', 'puaioh'],
\nfor this group we chose to use SED w/ focal-loss.
For the details of architecture of CNN model, please refer to 2nd place BirdCLEF 2021 solution
\nIt's worth to note, that we didn't use temporal mix-up mentioned in the solution.
Since the CNN model was used only for inference on large enough classes, it allowed us to build reliable validation and monitor metrics for Group1 birds only, for these models we used BCE loss to select best models on validation, however with mix-up augmentation model converged on the last epoch, - this fact allowed us to include some models which were trained on full data in the final ensemble.
\nThe ones that definitely helped
\nThe SED model hasn't changed much from the previous 4th solution.
\nThe main difference is the addition of some augmentations.
As a result, we think that the score has improved by 0.06 or more in Public LB.
\nFor the Group1 model and the Group2 model, we changed the loss function, and the other settings were not changed.
We couldn't find a good CV strategy, so most of the settings are decided by watching at public LB.
\nThe default threshold for every of 21 birds was decided by watching LB (in the end we used 0.05 for Group1 birds, but 0.04 gives >0.82 in private LB).
We manually set the threshold for \"skylar\" bird as 0.35, since our models trained with BCE loss predict it reliably.
The probability distribution of OOFs prediction for non-targetdata when using focal-loss differs depending on the bird (see below figure), so we set the threshold for each bird from Group2 depending on the distribution.
\nWe adopted the value of 91 percentile of the distribution for these birds.
![\"image\"](\"https://user-images.githubusercontent.com/57013219/170404672-c95e0539-21cd-4378-a4cd-75a8c756bdf4.png\")
For our best public LB submission we used 8 CNN models, 8 SED models for Group1 birds & 12 SED models for Group2 birds (single model here is either model trained on one fold or on the whole data)
\nWe've also applied time-smoothing post-processing to Group1 birds as in 2nd place BirdCLEF 2021 solution, but it didn't help on private LB, though boosted +0.01~ on public LB
\n| Name | \nPublic LB | \nPrivate LB | \n
|---|---|---|
| CNN model (no augs, single fold) | \n0.7715 | \n0.7278 | \n
| CNN model (augs, single fold) | \n0.7761 | \n0.7359 | \n
| CNN ensemble (CNN only w/ BCE loss) | \n0.8327 | \n0.7898 | \n
| SED model (4fold average w/ BCEFocal2wayloss) | \n0.8339 | \n0.7823 | \n
| Combine UEMU's SED w/ focal-loss & slime's CNN w/ BCE Loss using bird split mentioned above | \n0.8532 | \n0.8052 | \n
| Same as above, but add more CNNs w/ BCE loss and SED w/ BCE loss to group1 birds (best public LB, sub1) | \n0.8750 | \n0.8126 | \n
| Safe submission (lower thresholds, sub2) | \n0.8556 | \n0.8071 | \n
| Best private LB (add some models which didn't work on public LB) | \n0.8707 | \n0.8274 | \n
We also had around 20 subs which score > 0.82 on private LB and > 0.87 on public LB, but we didn't select them since we chose two subs in the following manner, - one is the best public LB and the other one with much lower thresholds to prevent the shake-up, this sub also happend to be placed 3rd, we're happy :)
\nAsk your questions! :)
\nUPD: we further checked the performance of some nets
\n| Name | \nPublic LB | \nPrivate LB | \n
|---|---|---|
| ensemble SED only w/ BCE loss (models fine-tuned for Group1) | \n0.8139 | \n0.7563 | \n
| ensemble SED only w/ focal-loss | \n0.8507 | \n0.8135 | \n
| ensemble CNN only w/ BCE loss (models fine-tuned for Group1) | \n0.8360 | \n0.7678 | \n
The conclusion is that we need some way to determine proper thresholds for predictions in order to see the true perfomance of the models.
\nWe're sure that bird split can bring 0.02-0.03 perfomance boost w/ proper thresholds.
\nUPD: 06.08.2022
\nWe published the inference kernel: https://www.kaggle.com/code/asaliquid1011/birdclef2022-3rd-place-inference
\nMy part can be found in the following github repository: https://github.com/dazzle-me/birdclef-2022-3rd-place-solution
First of all, thanks to Kaggle and Cornell Lab of Ornithology for organizing this competition.
\nI was lucky enough to team-up with UEMU, it was a fruitful team-up with lots of ideas coming from the both sides.
\nWe've worked hard until the end and thanks to that we managed to secure 3rd place, congratz to all of the winners!
Considering the situation, it seems necessary to mention that we didn't use BirdNet model.
\nExternal data we used:
\nfreefield1010, aicrowd2020 and nocall part of BirdCLEF 2021 soundscapes to mix 2022 samples with background noise and impove robustness.
Now for our approach, the key points to build strong and reliable pipeline are the following:
\nLet's get down to our best finding during competition
\nWhen we merged together, we looked on OOFs predictions produced by our models and found out that SED w/ focal-loss performs very differently compared to mentioned CNN trained w/ BCELoss depending on number of training samples
\nFrom our observations, SED models w/ focal-loss tend to make more conservative predictions, and due to the loss design they don't miss small classes:
\nHere in blue you can see SED model w/ focal-loss, and in pink CNN model trained w/ BCE loss
\n
The same can be said about advantage of models which used BCE loss during training over models with focal-loss for the large classes
\n
Therefore, we divided the birds into two groups according to the number of data and manual inspection of distribution plots of target-data as above, and used different models for them.
\nIt appears that it's optimal to include birds with number of training samples >= 10 to the Group1, and all other birds into Group2.
\nGroup1: 14birds ['jabwar', 'yefcan', 'skylar', 'akiapo', 'apapan', 'barpet', 'elepai', 'iiwi', 'houfin', 'omao', 'warwhe1', 'aniani', 'hawama', 'hawcre'],
\nfor them we ended up using CNN + SED models which were trained using BCE loss.
Group2: 7birds, ['crehon', 'ercfra', 'hawgoo', 'hawhaw', 'hawpet1', 'maupar', 'puaioh'],
\nfor this group we chose to use SED w/ focal-loss.
For the details of architecture of CNN model, please refer to 2nd place BirdCLEF 2021 solution
\nIt's worth to note, that we didn't use temporal mix-up mentioned in the solution.
Since the CNN model was used only for inference on large enough classes, it allowed us to build reliable validation and monitor metrics for Group1 birds only, for these models we used BCE loss to select best models on validation, however with mix-up augmentation model converged on the last epoch, - this fact allowed us to include some models which were trained on full data in the final ensemble.
\nThe ones that definitely helped
\nThe SED model hasn't changed much from the previous 4th solution.
\nThe main difference is the addition of some augmentations.
As a result, we think that the score has improved by 0.06 or more in Public LB.
\nFor the Group1 model and the Group2 model, we changed the loss function, and the other settings were not changed.
We couldn't find a good CV strategy, so most of the settings are decided by watching at public LB.
\nThe default threshold for every of 21 birds was decided by watching LB (in the end we used 0.05 for Group1 birds, but 0.04 gives >0.82 in private LB).
We manually set the threshold for \"skylar\" bird as 0.35, since our models trained with BCE loss predict it reliably.
The probability distribution of OOFs prediction for non-targetdata when using focal-loss differs depending on the bird (see below figure), so we set the threshold for each bird from Group2 depending on the distribution.
\nWe adopted the value of 91 percentile of the distribution for these birds.
For our best public LB submission we used 8 CNN models, 8 SED models for Group1 birds & 12 SED models for Group2 birds (single model here is either model trained on one fold or on the whole data)
\nWe've also applied time-smoothing post-processing to Group1 birds as in 2nd place BirdCLEF 2021 solution, but it didn't help on private LB, though boosted +0.01~ on public LB
\n| Name | \nPublic LB | \nPrivate LB | \n
|---|---|---|
| CNN model (no augs, single fold) | \n0.7715 | \n0.7278 | \n
| CNN model (augs, single fold) | \n0.7761 | \n0.7359 | \n
| CNN ensemble (CNN only w/ BCE loss) | \n0.8327 | \n0.7898 | \n
| SED model (4fold average w/ BCEFocal2wayloss) | \n0.8339 | \n0.7823 | \n
| Combine UEMU's SED w/ focal-loss & slime's CNN w/ BCE Loss using bird split mentioned above | \n0.8532 | \n0.8052 | \n
| Same as above, but add more CNNs w/ BCE loss and SED w/ BCE loss to group1 birds (best public LB, sub1) | \n0.8750 | \n0.8126 | \n
| Safe submission (lower thresholds, sub2) | \n0.8556 | \n0.8071 | \n
| Best private LB (add some models which didn't work on public LB) | \n0.8707 | \n0.8274 | \n
We also had around 20 subs which score > 0.82 on private LB and > 0.87 on public LB, but we didn't select them since we chose two subs in the following manner, - one is the best public LB and the other one with much lower thresholds to prevent the shake-up, this sub also happend to be placed 3rd, we're happy :)
\nAsk your questions! :)
\nUPD: we further checked the performance of some nets
\n| Name | \nPublic LB | \nPrivate LB | \n
|---|---|---|
| ensemble SED only w/ BCE loss (models fine-tuned for Group1) | \n0.8139 | \n0.7563 | \n
| ensemble SED only w/ focal-loss | \n0.8507 | \n0.8135 | \n
| ensemble CNN only w/ BCE loss (models fine-tuned for Group1) | \n0.8360 | \n0.7678 | \n
The conclusion is that we need some way to determine proper thresholds for predictions in order to see the true perfomance of the models.
\nWe're sure that bird split can bring 0.02-0.03 perfomance boost w/ proper thresholds.
\nUPD: 06.08.2022
\nWe published the inference kernel: https://www.kaggle.com/code/asaliquid1011/birdclef2022-3rd-place-inference
\nMy part can be found in the following github repository: https://github.com/dazzle-me/birdclef-2022-3rd-place-solution
Congrats @martynoveduard and @asaliquid1011 on 3rd place and becoming master. Nice solution without BirdNet, thanks for sharing!
", "rawMarkdown": "Congrats @martynoveduard and @asaliquid1011 on 3rd place and becoming master. Nice solution without BirdNet, thanks for sharing!", "votes": 3 }, { "id": 1802185, "postDate": "2022-05-26T14:35:26.523Z", "content": "Congrats on best score without BirdNet and an elegant solution. Happy that our last year's solution was helpful.
", "rawMarkdown": "Congrats on best score without BirdNet and an elegant solution. Happy that our last year's solution was helpful.", "votes": 4 }, { "id": 1804449, "postDate": "2022-05-29T02:13:11.080Z", "content": "Where is the best place to learn about SED models?
\nThis is great! Congrats and thanks for sharing.
", "rawMarkdown": "Where is the best place to learn about SED models? \n\nThis is great! Congrats and thanks for sharing.", "votes": 1, "replies": [ { "id": 1805208, "postDate": "2022-05-29T23:22:51.100Z", "content": "Thanks.
\nThis discussion will help you.
\nhttps://www.kaggle.com/competitions/rfcx-species-audio-detection/discussion/211007
Congratulations! Really creative solution !
", "rawMarkdown": "Congratulations! Really creative solution !", "votes": 1 }, { "id": 1802305, "postDate": "2022-05-26T16:13:50.467Z", "content": "Amazing idea spliting the birds by number of samples and using different loss functions. I didn't get how you combined the predictions on the ensemble though. Was it a simple union of the events (bird detections) that crossed each threshold in each model? Congrats @martynoveduard and well deserved prize!
", "rawMarkdown": "Amazing idea spliting the birds by number of samples and using different loss functions. I didn't get how you combined the predictions on the ensemble though. Was it a simple union of the events (bird detections) that crossed each threshold in each model? Congrats @martynoveduard and well deserved prize!", "votes": 1, "replies": [ { "id": 1802408, "postDate": "2022-05-26T17:35:29.727Z", "content": "Thanks, actually bird split was @asaliquid1011's idea
\nWe trained all our models using 152 classes,
\nbut some models which belong to Group1 made predictions only for birds included in Group1
\nand other models made predictions for birds from Group2
Then for each group we simply average the probabilities obtained from the models and apply thresholds as described in the solution (0.05 for birds from Group1 except for \"skylar\", for it we set 0.35, for birds from Group2 thresholds were selected based on OOFs distribution)
\nHere are raw values obtained for SED w/ focal-loss
\n\n{'crehon': 0.0610, \n'ercfra': 0.0303, \n'hawhaw': 0.0386, \n'hawpet1': 0.0491, \n'maupar': 0.0724, \n'puaioh': 0.0340}\n
This is teamwork. Congratulations!
", "rawMarkdown": "This is teamwork. Congratulations!", "votes": 1 }, { "id": 1804384, "postDate": "2022-05-28T21:13:28.403Z", "content": "Lovely write-up and some really interesting methods here. Thanks!
", "rawMarkdown": "Lovely write-up and some really interesting methods here. Thanks!", "votes": 2 }, { "id": 1801866, "postDate": "2022-05-26T08:14:57.847Z", "content": "You understands the characteristics of CNN and SED! And a great solution.
\nCongrats.
great work
\nabout the first plot what do you mean by non target is it the prediction as i know the targets just 0 and 1 not proba actully i did not anderstand the plot ?
This plot show the distribution of the predicted probabilities of the out of fold data when the data splited by 4 folds.
\nIn the graph of hawhaw, 'target ' show the probability distribution of hawhaw for 4 data labeled hawhaw. 'nontarget' show the distribution of hawhaw for about 6000 data labeled other than hawhaw.
\nThe purpose is to observe the trends of TP and FP for each target and model.
@asaliquid1011
\ngreat
\nso in the first plot the best is \"SED\" model with cyan color because it's propa_ distribution \"for TP\" has low variance and it's mean is large and because it treated all samples the same way correct me if I was wrong
\nthank you😄
Congrats @martynoveduard and @asaliquid1011 on 3rd place and becoming master. Nice solution without BirdNet, thanks for sharing!
", "votes": 3, "replies": [] }, { "id": 1802185, "author_name": "Psi", "author_url": "", "post_date": "2022-05-26T14:35:26.523000", "content": "Congrats on best score without BirdNet and an elegant solution. Happy that our last year's solution was helpful.
", "votes": 4, "replies": [] }, { "id": 1804449, "author_name": "Michael Mortenson", "author_url": "", "post_date": "2022-05-29T02:13:11.080000", "content": "Where is the best place to learn about SED models?
\nThis is great! Congrats and thanks for sharing.
", "votes": 1, "replies": [ { "id": 1805208, "author_name": "UEMU", "author_url": "", "post_date": "2022-05-29T23:22:51.100000", "content": "Thanks.
\nThis discussion will help you.
\nhttps://www.kaggle.com/competitions/rfcx-species-audio-detection/discussion/211007
Congratulations! Really creative solution !
", "votes": 1, "replies": [] }, { "id": 1802305, "author_name": "HinePo", "author_url": "", "post_date": "2022-05-26T16:13:50.467000", "content": "Amazing idea spliting the birds by number of samples and using different loss functions. I didn't get how you combined the predictions on the ensemble though. Was it a simple union of the events (bird detections) that crossed each threshold in each model? Congrats @martynoveduard and well deserved prize!
", "votes": 1, "replies": [ { "id": 1802408, "author_name": "slime", "author_url": "", "post_date": "2022-05-26T17:35:29.727000", "content": "Thanks, actually bird split was @asaliquid1011's idea
\nWe trained all our models using 152 classes,
\nbut some models which belong to Group1 made predictions only for birds included in Group1
\nand other models made predictions for birds from Group2
Then for each group we simply average the probabilities obtained from the models and apply thresholds as described in the solution (0.05 for birds from Group1 except for \"skylar\", for it we set 0.35, for birds from Group2 thresholds were selected based on OOFs distribution)
\nHere are raw values obtained for SED w/ focal-loss
\n\n{'crehon': 0.0610, \n'ercfra': 0.0303, \n'hawhaw': 0.0386, \n'hawpet1': 0.0491, \n'maupar': 0.0724, \n'puaioh': 0.0340}\n
This is teamwork. Congratulations!
", "votes": 1, "replies": [] }, { "id": 1804384, "author_name": "Tom Denton", "author_url": "", "post_date": "2022-05-28T21:13:28.403000", "content": "Lovely write-up and some really interesting methods here. Thanks!
", "votes": 2, "replies": [] }, { "id": 1801866, "author_name": "shinmura0", "author_url": "", "post_date": "2022-05-26T08:14:57.847000", "content": "You understands the characteristics of CNN and SED! And a great solution.
\nCongrats.
great work
\nabout the first plot what do you mean by non target is it the prediction as i know the targets just 0 and 1 not proba actully i did not anderstand the plot ?
This plot show the distribution of the predicted probabilities of the out of fold data when the data splited by 4 folds.
\nIn the graph of hawhaw, 'target ' show the probability distribution of hawhaw for 4 data labeled hawhaw. 'nontarget' show the distribution of hawhaw for about 6000 data labeled other than hawhaw.
\nThe purpose is to observe the trends of TP and FP for each target and model.
@asaliquid1011
\ngreat
\nso in the first plot the best is \"SED\" model with cyan color because it's propa_ distribution \"for TP\" has low variance and it's mean is large and because it treated all samples the same way correct me if I was wrong
\nthank you😄