First of all, thanks to the organizers for this competition and congratulations to all winning teams!
\nThe start in this year’s edition was not easy. Using models and training methods from last year didn’t work that well at first. Almost all attempts that improved local CV didn’t improve or even decreased public LB score and I was not able to get past 0.64 AUC for single models or 0.69 for ensembles.
\nAlmost thinking about giving up, I restarted from scratch using the public notebook from @salmanahmedtamu following methods discussed by @lihaoweicvch. Thanks for sharing!
\nWith this it was possible to get single model performance of 0.65/0.66 AUC.
\nThe original notebook uses:
From this baseline I started to experiment with different backbones, hyperparameters, augmentation methods and image sizes. One major drawback of the model was its relative long submission time (> 1h). So besides improving the score, one objective was to reduce inference time to fit more models in an ensemble. For this I switched to an EfficientNet B0 backbone (tf_efficientnet_b0_ns) and reduced image sizes. It turned out, that results were pretty unstable regarding LB score (0.62-0.66) and sensitive to different combinations of Mel parameters and input sizes. But with further adjustments I was able to create models with inference time below 12 minutes, still keeping a score around 0.65 AUC.
\nMain changes to the original notebook include:
\nCreating checkpoint soups follows the idea of model soups. But here, weights of the same model from different checkpoints from epochs 13-50 are averaged if they show an improvement in local CV score on one of the tracked metrics (LRAP, cMAP, F1, AUC). This led to more stable and sometimes even better LB scores.
\nWith all above mentioned modifications, I was now able to create an ensemble of 6 models achieving 0.70 AUC on public LB. Not great but good enough to start playing around with pseudo labels created from the unlabeled dataset.
Using pseudo labels from the test domain and their handling during training was the key element to move up to the top 10 in the leaderboard. Pseudo labels were created by applying the model ensemble with the best public LB score on the unlabeled data to create predictions for each 5s interval of each file.
\nAfter that, during next stage training, randomly selected 5s audio segments from the unlabeled test set are added to the training samples with a 25 % to 45 % chance. Before mixing the two audio signals, amplitudes of both waveforms are multiplied by a random factor.
\nThe target vector of the training sample (1.0 at positions of primary and secondary species, rest all zeros) is combined with the pseudo label (vector with predicted probabilities) to form the new target vector by taking the maximum of both.
This method of using the pseudo labeled test data combines several advantages:
\nAfter integrating pseudo labels, LB score significantly increased. The new ensemble was then used again to create a new set of pseudo labels. This cycle was repeated 3 times to iteratively improve model/ensemble performance. Progress on LB score (publ.|priv.) is shown in the following table:
\n| Stage | \nPseudo labels | \nSingle model (ID 4) | \nEnsemble | \n
|---|---|---|---|
| 0 | \n- | \n0.65735 | 0.59270 | \n0.70065 | 0.61738 | \n
| 1 | \nFrom stage 0 ensemble | \n0.69165 | 0.66119 | \n0.71090 | 0.67084 | \n
| 2 | \nFrom stage 1 ensemble | \n0.69936 | 0.67445 | \n0.72528 | 0.69035 * | \n
| 3 | \nFrom stage 2 ens. (normalized) | \n0.71154 | 0.67683 | \n0.71716 | 0.69527 | \n
\nAfter the 2nd iteration pseudo labels needed some normalization (scaling to range 0…1 ) to allow stable model training because label minimum values got too large. Also, the stage 3 ensemble was not selected for final ranking because public LB score unfortunately didn’t show the expected improvement.
\nPerformance and parameters of models from the 2nd place ensemble (2nd stage in previous table) are shown in the following table:
| Params./Model ID | \n1 | \n2 | \n3 | \n4 | \n5 | \n6 | \n
|---|---|---|---|---|---|---|
| seed | \n42 | \n42 | \n42 | \n42 | \n70 | \n42 | \n
| n_folds | \n5 | \n5 | \n5 | \n5 | \n10 | \n5 | \n
| fold | \n4 | \n1 | \n4 | \n4 | \n0 | \n4 | \n
| dataset | \nbc24 | \nbc24 | \nbc24 | \nbc24 | \nbc24+ | \nbc24 | \n
| n_mels | \n128 | \n128 | \n128 | \n64 | \n64 | \n64 | \n
| hop_length | \n512 | \n512 | \n1024 | \n1024 | \n1024 | \n1024 | \n
| image_height | \n256 | \n256 | \n128 | \n64 | \n64 | \n64 | \n
| image_width | \n256 | \n256 | \n128 | \n128 | \n128 | \n64 | \n
| pseudoLabelChance | \n35 % | \n40 % | \n45 % | \n30 % | \n30 % | \n25 % | \n
| ampExpMin | \n-0.5 | \n-1.0 | \n-0.5 | \n-0.5 | \n-0.5 | \n-0.5 | \n
| ampExpMax | \n0.1 | \n0.2 | \n0.1 | \n0.1 | \n0.1 | \n0.1 | \n
| Inference time | \n~ 50 min. | \n~ 50 min. | \n~ 17 min. | \n~ 12 min. | \n~ 12 min. | \n~ 11 min. | \n
| Public LB score | \n0.73270 | \n0.71975 | \n0.71104 | \n0.69936 | \n0.69124 | \n0.69309 | \n
| Private LB score | \n0.68521 | \n0.68533 | \n0.68116 | \n0.67445 | \n0.64543 | \n0.65862 | \n
\nModel diversity in the ensemble was realized by different Mel parameters, data subsets, image sizes, chances to add pseudo labels and amplitude factors to change the volume relation between training and pseudo label data. Parameters ampExpMin and ampExpMax provide the range for the random amplitude factor multiplied to training and pseudo label samples to change their volume in the mix: ampFactor = 10**(random.uniform(ampExpMin,ampExpMax))
\nModel 5 is the only one using external data. For this, additional files for the 182 species of the competition were downloaded from Xeno-Canto and the first 5 seconds part of each file was added to the training set (padded with zeros if too short).
Models were ensembled by simply taking the mean of predictions (probabilities from sigmoid outputs) of each single model. As a last step, for each file, predictions of a given window, were summed with those of the two neighboring windows with factor 0.5. This post-processing method was used by @theoviel and his team in the 3rd place solution of the Cornell Birdcall Identification competition .
\nToo many things ;) Unfortunately, I found a good approach to use pseudo labels rather late in the competition. Some post deadline submissions show, that many things that didn’t work at the beginning are actually quite beneficial to improve private LB score if pseudo labels are included. Hope to continue on those experiments maybe at the next edition…
\nAll 6 models (Score may vary because submission time > 2h)
\nBest 2 models (Stable and better LB score with submission time ~1h10min)
First of all, thanks to the organizers for this competition and congratulations to all winning teams!
\nThe start in this year’s edition was not easy. Using models and training methods from last year didn’t work that well at first. Almost all attempts that improved local CV didn’t improve or even decreased public LB score and I was not able to get past 0.64 AUC for single models or 0.69 for ensembles.
\nAlmost thinking about giving up, I restarted from scratch using the public notebook from @salmanahmedtamu following methods discussed by @lihaoweicvch. Thanks for sharing!
\nWith this it was possible to get single model performance of 0.65/0.66 AUC.
\nThe original notebook uses:
From this baseline I started to experiment with different backbones, hyperparameters, augmentation methods and image sizes. One major drawback of the model was its relative long submission time (> 1h). So besides improving the score, one objective was to reduce inference time to fit more models in an ensemble. For this I switched to an EfficientNet B0 backbone (tf_efficientnet_b0_ns) and reduced image sizes. It turned out, that results were pretty unstable regarding LB score (0.62-0.66) and sensitive to different combinations of Mel parameters and input sizes. But with further adjustments I was able to create models with inference time below 12 minutes, still keeping a score around 0.65 AUC.
\nMain changes to the original notebook include:
\nCreating checkpoint soups follows the idea of model soups. But here, weights of the same model from different checkpoints from epochs 13-50 are averaged if they show an improvement in local CV score on one of the tracked metrics (LRAP, cMAP, F1, AUC). This led to more stable and sometimes even better LB scores.
\nWith all above mentioned modifications, I was now able to create an ensemble of 6 models achieving 0.70 AUC on public LB. Not great but good enough to start playing around with pseudo labels created from the unlabeled dataset.
Using pseudo labels from the test domain and their handling during training was the key element to move up to the top 10 in the leaderboard. Pseudo labels were created by applying the model ensemble with the best public LB score on the unlabeled data to create predictions for each 5s interval of each file.
\nAfter that, during next stage training, randomly selected 5s audio segments from the unlabeled test set are added to the training samples with a 25 % to 45 % chance. Before mixing the two audio signals, amplitudes of both waveforms are multiplied by a random factor.
\nThe target vector of the training sample (1.0 at positions of primary and secondary species, rest all zeros) is combined with the pseudo label (vector with predicted probabilities) to form the new target vector by taking the maximum of both.
This method of using the pseudo labeled test data combines several advantages:
\nAfter integrating pseudo labels, LB score significantly increased. The new ensemble was then used again to create a new set of pseudo labels. This cycle was repeated 3 times to iteratively improve model/ensemble performance. Progress on LB score (publ.|priv.) is shown in the following table:
\n| Stage | \nPseudo labels | \nSingle model (ID 4) | \nEnsemble | \n
|---|---|---|---|
| 0 | \n- | \n0.65735 | 0.59270 | \n0.70065 | 0.61738 | \n
| 1 | \nFrom stage 0 ensemble | \n0.69165 | 0.66119 | \n0.71090 | 0.67084 | \n
| 2 | \nFrom stage 1 ensemble | \n0.69936 | 0.67445 | \n0.72528 | 0.69035 * | \n
| 3 | \nFrom stage 2 ens. (normalized) | \n0.71154 | 0.67683 | \n0.71716 | 0.69527 | \n
\nAfter the 2nd iteration pseudo labels needed some normalization (scaling to range 0…1 ) to allow stable model training because label minimum values got too large. Also, the stage 3 ensemble was not selected for final ranking because public LB score unfortunately didn’t show the expected improvement.
\nPerformance and parameters of models from the 2nd place ensemble (2nd stage in previous table) are shown in the following table:
| Params./Model ID | \n1 | \n2 | \n3 | \n4 | \n5 | \n6 | \n
|---|---|---|---|---|---|---|
| seed | \n42 | \n42 | \n42 | \n42 | \n70 | \n42 | \n
| n_folds | \n5 | \n5 | \n5 | \n5 | \n10 | \n5 | \n
| fold | \n4 | \n1 | \n4 | \n4 | \n0 | \n4 | \n
| dataset | \nbc24 | \nbc24 | \nbc24 | \nbc24 | \nbc24+ | \nbc24 | \n
| n_mels | \n128 | \n128 | \n128 | \n64 | \n64 | \n64 | \n
| hop_length | \n512 | \n512 | \n1024 | \n1024 | \n1024 | \n1024 | \n
| image_height | \n256 | \n256 | \n128 | \n64 | \n64 | \n64 | \n
| image_width | \n256 | \n256 | \n128 | \n128 | \n128 | \n64 | \n
| pseudoLabelChance | \n35 % | \n40 % | \n45 % | \n30 % | \n30 % | \n25 % | \n
| ampExpMin | \n-0.5 | \n-1.0 | \n-0.5 | \n-0.5 | \n-0.5 | \n-0.5 | \n
| ampExpMax | \n0.1 | \n0.2 | \n0.1 | \n0.1 | \n0.1 | \n0.1 | \n
| Inference time | \n~ 50 min. | \n~ 50 min. | \n~ 17 min. | \n~ 12 min. | \n~ 12 min. | \n~ 11 min. | \n
| Public LB score | \n0.73270 | \n0.71975 | \n0.71104 | \n0.69936 | \n0.69124 | \n0.69309 | \n
| Private LB score | \n0.68521 | \n0.68533 | \n0.68116 | \n0.67445 | \n0.64543 | \n0.65862 | \n
\nModel diversity in the ensemble was realized by different Mel parameters, data subsets, image sizes, chances to add pseudo labels and amplitude factors to change the volume relation between training and pseudo label data. Parameters ampExpMin and ampExpMax provide the range for the random amplitude factor multiplied to training and pseudo label samples to change their volume in the mix: ampFactor = 10**(random.uniform(ampExpMin,ampExpMax))
\nModel 5 is the only one using external data. For this, additional files for the 182 species of the competition were downloaded from Xeno-Canto and the first 5 seconds part of each file was added to the training set (padded with zeros if too short).
Models were ensembled by simply taking the mean of predictions (probabilities from sigmoid outputs) of each single model. As a last step, for each file, predictions of a given window, were summed with those of the two neighboring windows with factor 0.5. This post-processing method was used by @theoviel and his team in the 3rd place solution of the Cornell Birdcall Identification competition .
\nToo many things ;) Unfortunately, I found a good approach to use pseudo labels rather late in the competition. Some post deadline submissions show, that many things that didn’t work at the beginning are actually quite beneficial to improve private LB score if pseudo labels are included. Hope to continue on those experiments maybe at the next edition…
\nAll 6 models (Score may vary because submission time > 2h)
\nBest 2 models (Stable and better LB score with submission time ~1h10min)
Well done on the second place and thanks for sharing your insights!
", "rawMarkdown": "Well done on the second place and thanks for sharing your insights!", "votes": null } ], "comments": [ { "id": 2876206, "author_name": "yassinealouini", "author_url": "", "post_date": "06/17/2024 15:48:37", "content": "Well done on the second place and thanks for sharing your insights!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2872160": "First of all, thanks to the organizers for this competition and congratulations to all winning teams!\n\n### Summary\n- Only first 5 seconds of each recording is used for training\n- EfficientNet B0 backbone\n- Performance boost by using pseudo labels from target domain\n- Ensemble of 6 models\n- Model diversity through different Mel parameters, data subsets, image sizes and probabilities to add pseudo labels\n\n### First experiments\n\nThe start in this year’s edition was not easy. Using models and training methods from last year didn’t work that well at first. Almost all attempts that improved local CV didn’t improve or even decreased public LB score and I was not able to get past 0.64 AUC for single models or 0.69 for ensembles.\n\nAlmost thinking about giving up, I restarted from scratch using the public [notebook] (https://www.kaggle.com/code/salmanahmedtamu/training-0-65-0-66) from @salmanahmedtamu following [methods discussed] (https://www.kaggle.com/competitions/birdclef-2024/discussion/497539) by @lihaoweicvch. Thanks for sharing!\n\nWith this it was possible to get single model performance of 0.65/0.66 AUC. \nThe original notebook uses:\n- Only first 5s of training files and no extra files or classes from other sources\n- Model input: resized 3 channel Mel spec images of size 256x256\n- Backbone: eca_nfnet_l0\n- Mel parameters:\n - n_fft = 2048\n - hop_length = 512\n - n_mels = 128\n - f_min = 20\n - f_max = 16000\n- Training parameters:\n - CosineAnnealingLR scheduler with 5 warmup epochs\n - Peak learning rate 1e-4\n - 100 epochs with early stopping if AUC is not improving for 7 epochs\n - Batch size 64\n - Average of BCE and FocalLoss\n - GEM pooling\n- Augmentations:\n - HorizontalFlip\n - CoarseDropout\n - Mixup of Mel spectrogram images within training batches\n\nFrom this baseline I started to experiment with different backbones, hyperparameters, augmentation methods and image sizes. One major drawback of the model was its relative long submission time (> 1h). So besides improving the score, one objective was to reduce inference time to fit more models in an ensemble. For this I switched to an EfficientNet B0 backbone (tf_efficientnet_b0_ns) and reduced image sizes. It turned out, that results were pretty unstable regarding LB score (0.62-0.66) and sensitive to different combinations of Mel parameters and input sizes. But with further adjustments I was able to create models with inference time below 12 minutes, still keeping a score around 0.65 AUC. \n\nMain changes to the original notebook include:\n- Backbone: tf_efficientnet_b0_ns\n- 5 dropout layers before fc layer (inspired by [BirdCLEF2023 4th place] (https://www.kaggle.com/competitions/birdclef-2023/discussion/412753) and [BirdCLEF2021 2nd place] (https://www.kaggle.com/competitions/birdclef-2021/discussion/243463))\n- Higher learning rate (1e-3), less warmup epochs (3) and less epochs (50)\n- Different Mel parameters (n_mels, hop_length)\n- Additional augmentation: local and global time/frequency stretching performed on Mel spec images via resizing parts and entire image\n- Creating checkpoint soup instead of using early stopping\n\nCreating checkpoint soups follows the idea of [model soups] (https://arxiv.org/abs/2203.05482). But here, weights of the same model from different checkpoints from epochs 13-50 are averaged if they show an improvement in local CV score on one of the tracked metrics (LRAP, cMAP, F1, AUC). This led to more stable and sometimes even better LB scores.\nWith all above mentioned modifications, I was now able to create an ensemble of 6 models achieving 0.70 AUC on public LB. Not great but good enough to start playing around with pseudo labels created from the unlabeled dataset.\n\n### Performance boost with pseudo labels\n\nUsing pseudo labels from the test domain and their handling during training was the key element to move up to the top 10 in the leaderboard. Pseudo labels were created by applying the model ensemble with the best public LB score on the unlabeled data to create predictions for each 5s interval of each file.\nAfter that, during next stage training, randomly selected 5s audio segments from the unlabeled test set are added to the training samples with a 25 % to 45 % chance. Before mixing the two audio signals, amplitudes of both waveforms are multiplied by a random factor. \nThe target vector of the training sample (1.0 at positions of primary and secondary species, rest all zeros) is combined with the pseudo label (vector with predicted probabilities) to form the new target vector by taking the maximum of both. \n\nThis method of using the pseudo labeled test data combines several advantages:\n1. **Noise augmentation:** By mixing training samples with samples from the target domain the model learns how species sound within the environmental background noise of the test site habitat. This helps to address the domain shift between Xeno-Canto recordings and test soundscapes.\n2. **Additional training data:** The model gets more training samples representing the noise characteristics and species distribution of the target domain.\n3. **Knowledge distillation:** Since pseudo labels are derived from predictions of a stronger model (or ensemble of models in this case), its knowledge is transferred during training to the smaller model\n\nAfter integrating pseudo labels, LB score significantly increased. The new ensemble was then used again to create a new set of pseudo labels. This cycle was repeated 3 times to iteratively improve model/ensemble performance. Progress on LB score (publ.|priv.) is shown in the following table:\n\n| Stage | Pseudo labels | Single model (ID 4) | Ensemble |\n| --- | --- | --- | --- |\n| 0 | - \t\t | 0.65735 \\| 0.59270 | 0.70065 \\| 0.61738 |\n| 1 | From stage 0 ensemble | 0.69165 \\| 0.66119 | 0.71090 \\| 0.67084 |\n| 2 | From stage 1 ensemble | 0.69936 \\| 0.67445 | **0.72528** \\| 0.69035 * |\n| 3 | From stage 2 ens. (normalized) | **0.71154 \\| 0.67683** | 0.71716 \\| **0.69527** |\n\n