First of all, thanks to Kaggle and Cornell Lab of Ornithology for hosting this interesting competition. And I would like to emphasize the thanks to the bird sound recordists who provided their data through xeno-canto.
\nAdditional datasets can be found here.
\nKnowledge distillation of pre-computed predictions in Kaggle Models (bird-vocalization-classifier) was the hallmark of my solution. This model cannot complete inference in less than 2h, but it is very powerful. cmAP_5 on my validation dataset was 0.9479. So I tried to extract useful information from this model.
\nThe Kaggle models pre-computed predictions were created here.
\nAccording to this paper, they argued that efficient distillation required 1. consistent input, 2. aggressive mixup, and 3. a large number of epochs. So I did 2 and 3 because it was challenging to integrate the Kaggle model into my training pipeline. It certainly looked effective.
\nI chose only approaches that improve both CV and LB. In this competition, I just couldn't believe in one or the other.
I also encountered a situation where the training time was significantly longer when pretrained with past competition data. Thanks to everyone who suggested solutions.
\nReducing the inference time is the part I was having trouble with. Thanks to @leonshangguan for sharing his effective approach, I was able to use 4 models.
\n| Name | \nPublic LB | \nPrivate LB | \n
|---|---|---|
| BaseModel | \n0.80603 | \n0.70782 | \n
| BaseModel + Knowledge Distillation | \n0.82073 | \n0.72752 | \n
| BaseModel + Knowledge Distillation + Adding xeno-canto | \n0.82905 | \n0.74038 | \n
| BaseModel + Knowledge Distillation + Adding xeno-canto + Pretraining | \n0.8312 | \n0.74424 | \n
| BaseModel + Knowledge Distillation + Adding xeno-canto + Pretraining + Ensemble (4 models) | \n0.84019 | \n0.75688 | \n
My solution builds on contributions from participants in this and past competitions, birding enthusiasts, and the work of the machine learning community. thank you very much 🙏
\nTraining Code: https://github.com/AtsunoriFujita/BirdCLEF-2023-Identify-bird-calls-in-soundscapes
First of all, thanks to Kaggle and Cornell Lab of Ornithology for hosting this interesting competition. And I would like to emphasize the thanks to the bird sound recordists who provided their data through xeno-canto.
\nAdditional datasets can be found here.
\nKnowledge distillation of pre-computed predictions in Kaggle Models (bird-vocalization-classifier) was the hallmark of my solution. This model cannot complete inference in less than 2h, but it is very powerful. cmAP_5 on my validation dataset was 0.9479. So I tried to extract useful information from this model.
\nThe Kaggle models pre-computed predictions were created here.
\nAccording to this paper, they argued that efficient distillation required 1. consistent input, 2. aggressive mixup, and 3. a large number of epochs. So I did 2 and 3 because it was challenging to integrate the Kaggle model into my training pipeline. It certainly looked effective.
\nI chose only approaches that improve both CV and LB. In this competition, I just couldn't believe in one or the other.
I also encountered a situation where the training time was significantly longer when pretrained with past competition data. Thanks to everyone who suggested solutions.
\nReducing the inference time is the part I was having trouble with. Thanks to @leonshangguan for sharing his effective approach, I was able to use 4 models.
\n| Name | \nPublic LB | \nPrivate LB | \n
|---|---|---|
| BaseModel | \n0.80603 | \n0.70782 | \n
| BaseModel + Knowledge Distillation | \n0.82073 | \n0.72752 | \n
| BaseModel + Knowledge Distillation + Adding xeno-canto | \n0.82905 | \n0.74038 | \n
| BaseModel + Knowledge Distillation + Adding xeno-canto + Pretraining | \n0.8312 | \n0.74424 | \n
| BaseModel + Knowledge Distillation + Adding xeno-canto + Pretraining + Ensemble (4 models) | \n0.84019 | \n0.75688 | \n
My solution builds on contributions from participants in this and past competitions, birding enthusiasts, and the work of the machine learning community. thank you very much 🙏
\nTraining Code: https://github.com/AtsunoriFujita/BirdCLEF-2023-Identify-bird-calls-in-soundscapes
Congrats for your solo gold!💯
", "rawMarkdown": "Congrats for your solo gold!💯", "votes": null }, { "id": "2273354", "postDate": "05/25/2023 06:02:54", "content": "Congratulations on your solo gold! Great Find on how to make use of already existing model
", "rawMarkdown": "Congratulations on your solo gold! Great Find on how to make use of already existing model", "votes": null }, { "id": "2273478", "postDate": "05/25/2023 07:21:01", "content": "Congrats on a great solo finish @atsunorifujita 🎉 your distillation technique is indeed powerful.
", "rawMarkdown": "Congrats on a great solo finish @atsunorifujita 🎉 your distillation technique is indeed powerful.", "votes": null }, { "id": "2273540", "postDate": "05/25/2023 08:12:17", "content": "Awesome! Congratulations @atsunorifujita 🔥
", "rawMarkdown": "Awesome! Congratulations @atsunorifujita 🔥", "votes": null }, { "id": "2273557", "postDate": "05/25/2023 08:23:43", "content": "Thank you for sharing your approach, I am a beginner in Deep Learning, do you have any tips as to how I can approach complex problems like this.
", "rawMarkdown": "Thank you for sharing your approach, I am a beginner in Deep Learning, do you have any tips as to how I can approach complex problems like this.", "votes": null }, { "id": "2273650", "postDate": "05/25/2023 10:10:03", "content": "Congratulations, and thanks for sharing! Just to see if I got it correctly, your training steps were:
\nAny idea how much you gained in inference time from using PyTorch's JIT? I tried PyTorch 2's compilation on single models or the full ensemble, but it didn't help at all (probably also because of graph-breaking Python control statements).
", "rawMarkdown": "Congratulations, and thanks for sharing! Just to see if I got it correctly, your training steps were:\n1. Take a backbone that is pretrained on ImageNet\n2. Train a model with this backbone on BirdCLEF 2021+2022+ff1010, with Google's bird vocalization classifier as a teacher, lr 1e-5\n3. Replace the last layer, train the model further on 2023+ff1010+xeno-canto, again with the same teacher, lr 2.5e-4\n\nAny idea how much you gained in inference time from using PyTorch's JIT? I tried PyTorch 2's compilation on single models or the full ensemble, but it didn't help at all (probably also because of graph-breaking Python control statements).", "votes": null }, { "id": "2273810", "postDate": "05/25/2023 12:33:27", "content": "Thank you and congratulations on your solo gold 🎉
", "rawMarkdown": "Thank you and congratulations on your solo gold 🎉", "votes": null }, { "id": "2273817", "postDate": "05/25/2023 12:35:08", "content": "Thank you and congratulations 🙌
", "rawMarkdown": "Thank you and congratulations 🙌", "votes": null }, { "id": "2273822", "postDate": "05/25/2023 12:37:23", "content": "Thank you! I hope you are enjoying your work 👍
", "rawMarkdown": "Thank you! I hope you are enjoying your work 👍", "votes": null }, { "id": "2273823", "postDate": "05/25/2023 12:37:49", "content": "Thank you 👍
", "rawMarkdown": "Thank you 👍", "votes": null }, { "id": "2273827", "postDate": "05/25/2023 12:41:13", "content": "Thank you, I think we just have to come up with ideas and experiment. That's why we always have to keep learning.
", "rawMarkdown": "Thank you, I think we just have to come up with ideas and experiment. That's why we always have to keep learning.", "votes": null }, { "id": "2273841", "postDate": "05/25/2023 12:58:25", "content": "Thank you!
\nMy training steps are as you said. To be precise, ff1010 in 2. is used as background noise.
\nPyTorch JIT reduced the inference time by about 20%.\nI also tried ONNX, but didn't use it because of the complexity of the inference pipeline. The PyTorch JIT was able to include MelSpectrogram and PCEN in the model and converted it, simplifying the inference code.
", "rawMarkdown": "Thank you! \n\nMy training steps are as you said. To be precise, ff1010 in 2. is used as background noise.\n\nPyTorch JIT reduced the inference time by about 20%.\nI also tried ONNX, but didn't use it because of the complexity of the inference pipeline. The PyTorch JIT was able to include MelSpectrogram and PCEN in the model and converted it, simplifying the inference code.", "votes": null }, { "id": "2275050", "postDate": "05/26/2023 13:04:18", "content": "Congratulations, and thanks for the helpful post! Could you also make your training code available on github and post a link?
", "rawMarkdown": "Congratulations, and thanks for the helpful post! Could you also make your training code available on github and post a link?", "votes": null }, { "id": "2276835", "postDate": "05/27/2023 08:37:42", "content": "Absolutely! I'm preparing for it.
", "rawMarkdown": "Absolutely! I'm preparing for it.", "votes": null }, { "id": "2281991", "postDate": "05/31/2023 09:36:20", "content": "Thank you very much in advance. I am interested in distillation, but I cannot make it work
", "rawMarkdown": "Thank you very much in advance. I am interested in distillation, but I cannot make it work", "votes": null }, { "id": "2286237", "postDate": "06/03/2023 10:24:12", "content": "Congrats @atsunorifujita and thanks for sharing this detailed writeup 🙏
", "rawMarkdown": "Congrats @atsunorifujita and thanks for sharing this detailed writeup 🙏", "votes": null }, { "id": "2287980", "postDate": "06/05/2023 05:09:09", "content": "I have published a GitHub repo for training.
", "rawMarkdown": "I have published a GitHub repo for training.", "votes": null }, { "id": "2288056", "postDate": "06/05/2023 06:31:26", "content": "Thank you very much, I will have a look
", "rawMarkdown": "Thank you very much, I will have a look", "votes": null }, { "id": "2288656", "postDate": "06/05/2023 14:56:53", "content": "Congratulations!, I'd like to know how many time you needed to train your folds in a RTX 3090.
\nThank you in advance!
", "rawMarkdown": "Congratulations!, I'd like to know how many time you needed to train your folds in a RTX 3090.\n\nThank you in advance!", "votes": null }, { "id": "2289137", "postDate": "06/06/2023 00:29:35", "content": "Thank you and good question!
\nCongratulations and brilliant discussion. A few questions since I saw the ps recently:
\nHow much memory the RTX 3090?
", "rawMarkdown": "How much memory the RTX 3090?", "votes": null }, { "id": "2290725", "postDate": "06/07/2023 04:15:52", "content": "RTX 3090 has 24GB of VRAM
", "rawMarkdown": "RTX 3090 has 24GB of VRAM", "votes": null }, { "id": "2290759", "postDate": "06/07/2023 04:44:20", "content": "Thank you and nice questions!
\nIs there any reason why Focal Loss cannot outperform BCE?
\nI don't know why. I was just comparing the experimental results. Focal loss may be better for different data.
Why do you prefer librosa over torchaudio?
\nBecause I wanted my model inputs and KaggleModel inputs to be as consistent as possible. Firstly I was using soundfile.
Is there any substantial postprocessing you used in inference or that you plan to use as future work?
\nI can't think of anything at the moment. A simple average of multiple models worked.
What models are appropriate for ONNX?
\nIt was possible to convert to ONNX by separating mel spectrogram and PCEN from my model. However, I got lost motivation to use it because the inference time was not significantly different from PyTorch JIT.
How do you get started with PyTorch jit? Do you have any recommended documentation?
\nIn many cases, it's pretty straightforward (it fails with unsupported layers). The official documentation is helpful, but you can convert it with a few lines of code.
https://pytorch.org/docs/stable/jit.html
", "rawMarkdown": "Thank you and nice questions!\n\n1. Is there any reason why Focal Loss cannot outperform BCE? \nI don't know why. I was just comparing the experimental results. Focal loss may be better for different data.\n\n\n2. Why do you prefer librosa over torchaudio?\nBecause I wanted my model inputs and KaggleModel inputs to be as consistent as possible. Firstly I was using soundfile.\n\n\n3. Is there any substantial postprocessing you used in inference or that you plan to use as future work?\nI can't think of anything at the moment. A simple average of multiple models worked.\n\n\n4. What models are appropriate for ONNX?\nIt was possible to convert to ONNX by separating mel spectrogram and PCEN from my model. However, I got lost motivation to use it because the inference time was not significantly different from PyTorch JIT.\n\n\n5. How do you get started with PyTorch jit? Do you have any recommended documentation?\nIn many cases, it's pretty straightforward (it fails with unsupported layers). The official documentation is helpful, but you can convert it with a few lines of code.\n\nhttps://pytorch.org/docs/stable/jit.html", "votes": null }, { "id": "2292640", "postDate": "06/08/2023 14:11:45", "content": "Thanks! I downloaded your repo and followed your setup instructions, and am now able to run the training. I look forward to studying it!
", "rawMarkdown": "Thanks! I downloaded your repo and followed your setup instructions, and am now able to run the training. I look forward to studying it!", "votes": null } ], "comments": [ { "id": 2273228, "author_name": "leonshangguan", "author_url": "", "post_date": "05/25/2023 04:48:28", "content": "Congrats for your solo gold!💯
", "votes": null, "replies": [ { "id": 2273810, "author_name": "atsunorifujita", "author_url": "", "post_date": "05/25/2023 12:33:27", "content": "Thank you and congratulations on your solo gold 🎉
", "votes": null, "replies": [] } ] }, { "id": 2273354, "author_name": "harshitsheoran", "author_url": "", "post_date": "05/25/2023 06:02:54", "content": "Congratulations on your solo gold! Great Find on how to make use of already existing model
", "votes": null, "replies": [ { "id": 2273817, "author_name": "atsunorifujita", "author_url": "", "post_date": "05/25/2023 12:35:08", "content": "Thank you and congratulations 🙌
", "votes": null, "replies": [] } ] }, { "id": 2273478, "author_name": "nischaydnk", "author_url": "", "post_date": "05/25/2023 07:21:01", "content": "Congrats on a great solo finish @atsunorifujita 🎉 your distillation technique is indeed powerful.
", "votes": null, "replies": [ { "id": 2273822, "author_name": "atsunorifujita", "author_url": "", "post_date": "05/25/2023 12:37:23", "content": "Thank you! I hope you are enjoying your work 👍
", "votes": null, "replies": [] } ] }, { "id": 2273540, "author_name": "scipygaurav", "author_url": "", "post_date": "05/25/2023 08:12:17", "content": "Awesome! Congratulations @atsunorifujita 🔥
", "votes": null, "replies": [ { "id": 2273823, "author_name": "atsunorifujita", "author_url": "", "post_date": "05/25/2023 12:37:49", "content": "Thank you 👍
", "votes": null, "replies": [] } ] }, { "id": 2273557, "author_name": "aadityabansalcodes", "author_url": "", "post_date": "05/25/2023 08:23:43", "content": "Thank you for sharing your approach, I am a beginner in Deep Learning, do you have any tips as to how I can approach complex problems like this.
", "votes": null, "replies": [ { "id": 2273827, "author_name": "atsunorifujita", "author_url": "", "post_date": "05/25/2023 12:41:13", "content": "Thank you, I think we just have to come up with ideas and experiment. That's why we always have to keep learning.
", "votes": null, "replies": [] } ] }, { "id": 2273650, "author_name": "janschl", "author_url": "", "post_date": "05/25/2023 10:10:03", "content": "Congratulations, and thanks for sharing! Just to see if I got it correctly, your training steps were:
\nAny idea how much you gained in inference time from using PyTorch's JIT? I tried PyTorch 2's compilation on single models or the full ensemble, but it didn't help at all (probably also because of graph-breaking Python control statements).
", "votes": null, "replies": [ { "id": 2273841, "author_name": "atsunorifujita", "author_url": "", "post_date": "05/25/2023 12:58:25", "content": "Thank you!
\nMy training steps are as you said. To be precise, ff1010 in 2. is used as background noise.
\nPyTorch JIT reduced the inference time by about 20%.\nI also tried ONNX, but didn't use it because of the complexity of the inference pipeline. The PyTorch JIT was able to include MelSpectrogram and PCEN in the model and converted it, simplifying the inference code.
", "votes": null, "replies": [] } ] }, { "id": 2275050, "author_name": "janhuus", "author_url": "", "post_date": "05/26/2023 13:04:18", "content": "Congratulations, and thanks for the helpful post! Could you also make your training code available on github and post a link?
", "votes": null, "replies": [ { "id": 2276835, "author_name": "atsunorifujita", "author_url": "", "post_date": "05/27/2023 08:37:42", "content": "Absolutely! I'm preparing for it.
", "votes": null, "replies": [ { "id": 2281991, "author_name": "nyleve", "author_url": "", "post_date": "05/31/2023 09:36:20", "content": "Thank you very much in advance. I am interested in distillation, but I cannot make it work
", "votes": null, "replies": [ { "id": 2287980, "author_name": "atsunorifujita", "author_url": "", "post_date": "06/05/2023 05:09:09", "content": "I have published a GitHub repo for training.
", "votes": null, "replies": [ { "id": 2288056, "author_name": "nyleve", "author_url": "", "post_date": "06/05/2023 06:31:26", "content": "Thank you very much, I will have a look
", "votes": null, "replies": [] }, { "id": 2292640, "author_name": "janhuus", "author_url": "", "post_date": "06/08/2023 14:11:45", "content": "Thanks! I downloaded your repo and followed your setup instructions, and am now able to run the training. I look forward to studying it!
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2286237, "author_name": "pardeep19singh", "author_url": "", "post_date": "06/03/2023 10:24:12", "content": "Congrats @atsunorifujita and thanks for sharing this detailed writeup 🙏
", "votes": null, "replies": [] }, { "id": 2288656, "author_name": "pablolarrosa", "author_url": "", "post_date": "06/05/2023 14:56:53", "content": "Congratulations!, I'd like to know how many time you needed to train your folds in a RTX 3090.
\nThank you in advance!
", "votes": null, "replies": [ { "id": 2289137, "author_name": "atsunorifujita", "author_url": "", "post_date": "06/06/2023 00:29:35", "content": "Thank you and good question!
\nHow much memory the RTX 3090?
", "votes": null, "replies": [ { "id": 2290725, "author_name": "atsunorifujita", "author_url": "", "post_date": "06/07/2023 04:15:52", "content": "RTX 3090 has 24GB of VRAM
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2289703, "author_name": "pushkar007", "author_url": "", "post_date": "06/06/2023 10:15:20", "content": "Congratulations and brilliant discussion. A few questions since I saw the ps recently:
\nThank you and nice questions!
\nIs there any reason why Focal Loss cannot outperform BCE?
\nI don't know why. I was just comparing the experimental results. Focal loss may be better for different data.
Why do you prefer librosa over torchaudio?
\nBecause I wanted my model inputs and KaggleModel inputs to be as consistent as possible. Firstly I was using soundfile.
Is there any substantial postprocessing you used in inference or that you plan to use as future work?
\nI can't think of anything at the moment. A simple average of multiple models worked.
What models are appropriate for ONNX?
\nIt was possible to convert to ONNX by separating mel spectrogram and PCEN from my model. However, I got lost motivation to use it because the inference time was not significantly different from PyTorch JIT.
How do you get started with PyTorch jit? Do you have any recommended documentation?
\nIn many cases, it's pretty straightforward (it fails with unsupported layers). The official documentation is helpful, but you can convert it with a few lines of code.
https://pytorch.org/docs/stable/jit.html
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2273202": "First of all, thanks to Kaggle and Cornell Lab of Ornithology for hosting this interesting competition. And I would like to emphasize the thanks to the bird sound recordists who provided their data through xeno-canto.\n\n### Solution Summary\n- Knowledge Distillation is all you need.\n- Adding no-call data, xeno-canto data, and background audios(Zenodo) is effective.\n\n### Datasets\nAdditional datasets can be found [here](https://www.kaggle.com/datasets/atsunorifujita/birdclef-2023-additional).\n- Bird CLEF 2023\n- Bird CLEF 2021, 2022 (for pretraining)\n- ff1010bird_nocall (5,755 files for learning no call)\n- xeno-canto files not included in training dataset CC-BY-NC-SA (896 files) and CC-BY-NC-ND (5,212 files).\n- Zenodo dataset (background noise)\n- esc50 (rain, frog)\n- aicrowd2020_noise_30sec (background noise for pretraining)\n\n### Models\n- My models are based on [the BirdCLEF 2021 2nd place solution](https://www.kaggle.com/competitions/birdclef-2021/discussion/243463) using [timm](https://github.com/huggingface/pytorch-image-models).\n- My solution consists of a total of 4 models, all with eca_nfnet_l0 as the backbone. Each set is slightly different.\n\n1. MelSpectrogram (sample_rate: 32000, mel_bins: 128, fmin: 20, fmax: 16000, window_size: 2048, hop_size: 512, top_db=80.0, NormalizeMelSpec) following [this solution](https://www.kaggle.com/competitions/birdclef-2021/discussion/243293).\n - Public: 0.8312, Private: 0.74424\n2. Balanced sampling with the above settings.\n - Public: 0.83106, Private: 0.74406\n3. [PCEN](https://github.com/daemon/pytorch-pcen) (sample_rate: 32000, mel_bins: 128, fmin: 20, fmax: 16000, window_size: 2048, hop_size: 512).\n - Public: 0.83005, Private: 0.74134\n4. MelSpectrogram (sample_rate: 32000, mel_bins: 64, fmin: 50, fmax: 14000, window_size: 1024, hop_size: 320).\n - Public: 0.83014, Private: 0.74201\n\n### Training\nKnowledge distillation of pre-computed predictions in [Kaggle Models](https://www.kaggle.com/models/google/bird-vocalization-classifier/frameworks/TensorFlow2/variations/bird-vocalization-classifier/versions/1) (bird-vocalization-classifier) was the hallmark of my solution. This model cannot complete inference in less than 2h, but it is very powerful. cmAP_5 on my validation dataset was 0.9479. So I tried to extract useful information from this model.\n\n**The Kaggle models pre-computed predictions were created [here](https://www.kaggle.com/code/atsunorifujita/extract-from-kaggle-models/notebook)**.\n\nAccording to this [paper](https://arxiv.org/abs/2106.05237), they argued that efficient distillation required 1. consistent input, 2. aggressive mixup, and 3. a large number of epochs. So I did 2 and 3 because it was challenging to integrate the Kaggle model into my training pipeline. It certainly looked effective.\n\nI chose only approaches that improve both CV and LB. In this competition, I just couldn't believe in one or the other.\n\n- Using 5 StratifiedKFold. Only 1 fold did not cover all classes, so the remaining 4 were used for training.\n- The evaluation metric is padded_cmap1. The best results were the same with padded_cmap5 in most cases.\n- Use primary_label only\n- no-call is represented by all 0\n- loss: 0.1 * BCEWithLogitsLoss (primary_label) + 0.9 * KLDivLoss (from Kaggle model)\n - Softmax temperature=20 was best (tried 5, 10, 20, 30).\n- I used randomly sampled 20-sec clips for training. Audio that is less than 20 sec is repeated.\n- epoch = 400 (Most models converge at 100-300)\n- early stopping(pretraining=10, training=20)\n- Optimizer: AdamW (lr: pretraining=5e-4, training=2.5e-4, wd: 1e-6)\n- CosineLRScheduler(t_initial=10, warmup_t=1, cycle_limit=40, cycle_decay=1.0, lr_min=1e-7, t_in_epochs=True,)\n- mixup p = 1.0 (It was better than p=0.5)