Congrats to the winners and also to the host that create these annual competitions. π
\nSummary of the solution
\nInference with 12 models all together within the 2 hours CPU inference window.
\n5 different Tensorflow CNN, SOTA techniques trained with previous competition data and finetuned with current data, post ONNX converted. Ensembled together with: 1. The latest BirdNet TFlite version with 124 of the 264 birds. 2. custom classification TF head trained with Birdnet embeddings on competition data. 3. XGB 5 fold trained with Birdnet embeddings on competition data. Also used the non-bird classes from Birdnet to reduce the probabilities with 50% over the birds if the non-birds classes had higher probability in 5 seconds window, nothing of use here but interesting to implement.
\nTraining and data
\nAs starter code I used @awsaf49 https://www.kaggle.com/code/awsaf49/birdclef23-pretraining-is-all-you-need-train. Tensorflow based TPU training with W&B logging and latest SOTA audio techniques, SpecAug like Time Freq masking and Audio Augmentation like Gaussian Noise, Random CropPad, CutMix and MixUp.
\n\n\nMethodology π―
\n
\n In this notebook, we will explore how to identify bird calls using TensorFlow. Specifically, this notebook will cover:
\n How to use tf.data for audio processing tasks and reading .ogg files in TensorFlow
\n How to extract spectrogram features from raw audio on TPU/GPU, which reduces CPU bottleneck significantly, speeding up the process by ~ 4Γon P100 GPU compared to the previous notebook.
\n Unlike the previous tutorial, this notebook will perform spectrogram augmentation such as TimeFreqMask and Normalization on GPU/TPU and perform CutMix and MixUp with audio data on CPU.
\n This notebook demonstrates how pre-training on the BirdCLEF - 2020, 2021, 2022 & Xeno-Canto Extend dataset can improve transfer learning performance. CNN backbones, like EfficientNet, struggle with spectrogram data even with ImageNet pre-trained weights as they are not fimilar with audio data. Pre-training on an audio dataset, like BirdCLEF, can mitigate this issue and can yield a ~ 5% improvement in local validation and ~ 2% improvement in leaderboard. This notebook is compatible with both GPU, TPU, and the newly launched TPU-VM device is automatically selected, so you won't have to do anything to allocate the device.
Credit to @awsaf49
\nInference and submissions
\nTF CNN models trained and used:
\nEfficientnetb1, Eficientnetb0, Efficientnetv1b1ns, Efficientnetv2m, Efficientnetv2s β some trained with all the data some in single fold, many also trained/saved with weighted average over 2-5 checkpoints-weights, during the pretraining phase for better generalization. All models converted to ONNX format in test mode. All models picked based on speed and performance, e.g. could it be onnx converted and in what cpu inference speed while in onnx format.
\nBirdNet:
\nUsed the code and models from https://github.com/kahst/BirdNET-Analyzer and used the latest V.2.3 tflite model for best test time speed. I could find 124 of the 264 competition birds in model and also some other interesting classes within the 3K BirdNet classes.
\nI used the BirdNet to create and use 4 models to the CNN ensemble.
Then all CNNs and BirdNet variants where power averaged and weighted ensembled.
\nBelow is a progress for the competition metric with the different models/solutions. Not the same as the picked submissions but only another ensemble weight and also the best scored, so tested on that post deadline.
Private β Public - Models
\n.71621 β .81447 β Only CNNs
\n.72391 β .81885 β With CNNs + BirdNet classifier and extract the 124 birds.
\n.72250 β .81873 β With CNNs + BirdNet embeddings to classify on a custom cls-head which I had pretrained on all the classes.
\n.72459 β .81996 β With CNNs + BirdNet embeddings to classify on 5-fold XGB which I had pretrained on all the classes.
\n.73132 β .82337 β All above weighted ensembled.
Thatβs it!
\nCongrats to the winners and also to the host that create these annual competitions. π
\nSummary of the solution
\nInference with 12 models all together within the 2 hours CPU inference window.
\n5 different Tensorflow CNN, SOTA techniques trained with previous competition data and finetuned with current data, post ONNX converted. Ensembled together with: 1. The latest BirdNet TFlite version with 124 of the 264 birds. 2. custom classification TF head trained with Birdnet embeddings on competition data. 3. XGB 5 fold trained with Birdnet embeddings on competition data. Also used the non-bird classes from Birdnet to reduce the probabilities with 50% over the birds if the non-birds classes had higher probability in 5 seconds window, nothing of use here but interesting to implement.
\nTraining and data
\nAs starter code I used @awsaf49 https://www.kaggle.com/code/awsaf49/birdclef23-pretraining-is-all-you-need-train. Tensorflow based TPU training with W&B logging and latest SOTA audio techniques, SpecAug like Time Freq masking and Audio Augmentation like Gaussian Noise, Random CropPad, CutMix and MixUp.
\n\n\nMethodology π―
\n
\n In this notebook, we will explore how to identify bird calls using TensorFlow. Specifically, this notebook will cover:
\n How to use tf.data for audio processing tasks and reading .ogg files in TensorFlow
\n How to extract spectrogram features from raw audio on TPU/GPU, which reduces CPU bottleneck significantly, speeding up the process by ~ 4Γon P100 GPU compared to the previous notebook.
\n Unlike the previous tutorial, this notebook will perform spectrogram augmentation such as TimeFreqMask and Normalization on GPU/TPU and perform CutMix and MixUp with audio data on CPU.
\n This notebook demonstrates how pre-training on the BirdCLEF - 2020, 2021, 2022 & Xeno-Canto Extend dataset can improve transfer learning performance. CNN backbones, like EfficientNet, struggle with spectrogram data even with ImageNet pre-trained weights as they are not fimilar with audio data. Pre-training on an audio dataset, like BirdCLEF, can mitigate this issue and can yield a ~ 5% improvement in local validation and ~ 2% improvement in leaderboard. This notebook is compatible with both GPU, TPU, and the newly launched TPU-VM device is automatically selected, so you won't have to do anything to allocate the device.
Credit to @awsaf49
\nInference and submissions
\nTF CNN models trained and used:
\nEfficientnetb1, Eficientnetb0, Efficientnetv1b1ns, Efficientnetv2m, Efficientnetv2s β some trained with all the data some in single fold, many also trained/saved with weighted average over 2-5 checkpoints-weights, during the pretraining phase for better generalization. All models converted to ONNX format in test mode. All models picked based on speed and performance, e.g. could it be onnx converted and in what cpu inference speed while in onnx format.
\nBirdNet:
\nUsed the code and models from https://github.com/kahst/BirdNET-Analyzer and used the latest V.2.3 tflite model for best test time speed. I could find 124 of the 264 competition birds in model and also some other interesting classes within the 3K BirdNet classes.
\nI used the BirdNet to create and use 4 models to the CNN ensemble.
Then all CNNs and BirdNet variants where power averaged and weighted ensembled.
\nBelow is a progress for the competition metric with the different models/solutions. Not the same as the picked submissions but only another ensemble weight and also the best scored, so tested on that post deadline.
Private β Public - Models
\n.71621 β .81447 β Only CNNs
\n.72391 β .81885 β With CNNs + BirdNet classifier and extract the 124 birds.
\n.72250 β .81873 β With CNNs + BirdNet embeddings to classify on a custom cls-head which I had pretrained on all the classes.
\n.72459 β .81996 β With CNNs + BirdNet embeddings to classify on 5-fold XGB which I had pretrained on all the classes.
\n.73132 β .82337 β All above weighted ensembled.
Thatβs it!
\n@kirderf
\nThank you so much for sharing your 37th-place solution! It's great to see the collaborative spirit in the data science community
Congrats. I could barely ensemble 2 TF CNN models (using onnx). If you don't mind asking what other trick did you use to ensemble 12 models? Converted them to TFlite?
", "rawMarkdown": "Congrats. I could barely ensemble 2 TF CNN models (using onnx). If you don't mind asking what other trick did you use to ensemble 12 models? Converted them to TFlite?", "votes": null }, { "id": "2275081", "postDate": "05/26/2023 13:27:54", "content": "It's 5 CNN Onnx rest is BirdNet TFlite but creates 7 models out of 1 BirdNet tflite output as I use the embeddings to the custom cls heads and the 5 XGB heads are fast. For the CNN I noticed that FSR made it less fast and different architectures are faster like Effv2 works perfect with ONNX in test time and for example Convnexts less faster and Nfnets I didn't manage to convert at all, but maybe it's different in Pytorch as this was in Tensorflow.
", "rawMarkdown": "It's 5 CNN Onnx rest is BirdNet TFlite but creates 7 models out of 1 BirdNet tflite output as I use the embeddings to the custom cls heads and the 5 XGB heads are fast. For the CNN I noticed that FSR made it less fast and different architectures are faster like Effv2 works perfect with ONNX in test time and for example Convnexts less faster and Nfnets I didn't manage to convert at all, but maybe it's different in Pytorch as this was in Tensorflow.", "votes": null } ], "comments": [ { "id": 2274068, "author_name": "bilalwaseer", "author_url": "", "post_date": "05/25/2023 15:36:14", "content": "@kirderf
\nThank you so much for sharing your 37th-place solution! It's great to see the collaborative spirit in the data science community
Congrats. I could barely ensemble 2 TF CNN models (using onnx). If you don't mind asking what other trick did you use to ensemble 12 models? Converted them to TFlite?
", "votes": null, "replies": [ { "id": 2275081, "author_name": "kirderf", "author_url": "", "post_date": "05/26/2023 13:27:54", "content": "It's 5 CNN Onnx rest is BirdNet TFlite but creates 7 models out of 1 BirdNet tflite output as I use the embeddings to the custom cls heads and the 5 XGB heads are fast. For the CNN I noticed that FSR made it less fast and different architectures are faster like Effv2 works perfect with ONNX in test time and for example Convnexts less faster and Nfnets I didn't manage to convert at all, but maybe it's different in Pytorch as this was in Tensorflow.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2274015": "Congrats to the winners and also to the host that create these annual competitions. π\n\n**Summary of the solution**\n\nInference with 12 models all together within the 2 hours CPU inference window.\n\n5 different Tensorflow CNN, SOTA techniques trained with previous competition data and finetuned with current data, post ONNX converted. Ensembled together with: 1. The latest BirdNet TFlite version with 124 of the 264 birds. 2. custom classification TF head trained with Birdnet embeddings on competition data. 3. XGB 5 fold trained with Birdnet embeddings on competition data. Also used the non-bird classes from Birdnet to reduce the probabilities with 50% over the birds if the non-birds classes had higher probability in 5 seconds window, nothing of use here but interesting to implement.\n\n**Training and data**\n\nAs starter code I used @awsaf49 https://www.kaggle.com/code/awsaf49/birdclef23-pretraining-is-all-you-need-train. Tensorflow based TPU training with W&B logging and latest SOTA audio techniques, SpecAug like Time Freq masking and Audio Augmentation like Gaussian Noise, Random CropPad, CutMix and MixUp.\n\n>Methodology π―\nIn this notebook, we will explore how to identify bird calls using TensorFlow. Specifically, this notebook will cover:\nHow to use tf.data for audio processing tasks and reading .ogg files in TensorFlow\nHow to extract spectrogram features from raw audio on TPU/GPU, which reduces CPU bottleneck significantly, speeding up the process by ~ 4Γon P100 GPU compared to the previous notebook.\nUnlike the previous tutorial, this notebook will perform spectrogram augmentation such as TimeFreqMask and Normalization on GPU/TPU and perform CutMix and MixUp with audio data on CPU.\nThis notebook demonstrates how pre-training on the BirdCLEF - 2020, 2021, 2022 & Xeno-Canto Extend dataset can improve transfer learning performance. CNN backbones, like EfficientNet, struggle with spectrogram data even with ImageNet pre-trained weights as they are not fimilar with audio data. Pre-training on an audio dataset, like BirdCLEF, can mitigate this issue and can yield a ~ 5% improvement in local validation and ~ 2% improvement in leaderboard. This notebook is compatible with both GPU, TPU, and the newly launched TPU-VM device is automatically selected, so you won't have to do anything to allocate the device.\n\nCredit to @awsaf49 \n\n**Inference and submissions**\n\nTF CNN models trained and used: \nEfficientnetb1, Eficientnetb0, Efficientnetv1b1ns, Efficientnetv2m, Efficientnetv2s β some trained with all the data some in single fold, many also trained/saved with weighted average over 2-5 checkpoints-weights, during the pretraining phase for better generalization. All models converted to ONNX format in test mode. All models picked based on speed and performance, e.g. could it be onnx converted and in what cpu inference speed while in onnx format. \nBirdNet:\nUsed the code and models from https://github.com/kahst/BirdNET-Analyzer and used the latest V.2.3 tflite model for best test time speed. I could find 124 of the 264 competition birds in model and also some other interesting classes within the 3K BirdNet classes.\nI used the BirdNet to create and use 4 models to the CNN ensemble.\n1.\tClassify and extract the 124 birds and also ~100 non-bird classes for further ensemble and testing.\n2.\tUsed the same modelβs embeddings to classify on a custom cls-head which I had pretrained on all the classes. Describe in the BirdNet code but I used another deeper architecture.\n3.\tUsed the same modelβs embeddings to classify on 5-fold XGB models which I had pretrained on all the classes with multi:softprob and aucpr metric.\n4.\tI concatenated the ensembled classes together with the non-bird classes and extracted which 5 second windows that had highest probabilities within the non-bird classes and reduced the probability by 50% in the 264 bird classes as it was either noisy or other sounds that might would reduce the like hood of a birdcall. Now, this was of no use in this metric but still interesting testing for other purpose and metrics π\n\nThen all CNNs and BirdNet variants where power averaged and weighted ensembled.\nBelow is a progress for the competition metric with the different models/solutions. Not the same as the picked submissions but only another ensemble weight and also the best scored, so tested on that post deadline.\n\nPrivate β Public - Models\n\n.71621 β .81447 β Only CNNs\n.72391 β .81885 β With CNNs + BirdNet classifier and extract the 124 birds.\n.72250 β .81873 β With CNNs + BirdNet embeddings to classify on a custom cls-head which I had pretrained on all the classes.\n.72459 β .81996 β With CNNs + BirdNet embeddings to classify on 5-fold XGB which I had pretrained on all the classes.\n.73132 β .82337 β All above weighted ensembled.\n\n\n**Thatβs it!**\n\n------------------------------------------------", "2274068": "kirderf \nThank you so much for sharing your 37th-place solution! It's great to see the collaborative spirit in the data science community", "2275043": "Congrats. I could barely ensemble 2 TF CNN models (using onnx). If you don't mind asking what other trick did you use to ensemble 12 models? Converted them to TFlite?", "2275081": "It's 5 CNN Onnx rest is BirdNet TFlite but creates 7 models out of 1 BirdNet tflite output as I use the embeddings to the custom cls heads and the 5 XGB heads are fast. For the CNN I noticed that FSR made it less fast and different architectures are faster like Effv2 works perfect with ONNX in test time and for example Convnexts less faster and Nfnets I didn't manage to convert at all, but maybe it's different in Pytorch as this was in Tensorflow." }, "source": "meta" }