{ "id": 183436, "title": "13th place digest", "url": "/competitions/birdsong-recognition/writeups/tmz-13th-place-digest", "author_name": "", "post_date": "2020-09-16T18:11:43.683Z", "votes": 27, "comment_count": 11, "views": 0, "content": "

It was my first audio competition and I really enjoyed it! Thanks to organizers and Kaggle for it.
\nThanks to my teammates @tikutiku @zfturbo the great collaboration, it was a pleasure and I've learnt a lot again.

\n

Here is a digest of our solution.

\n

Main Pipeline (see image below):

\n\n

Time Augmentations:

\n\n

Spec/Image augmentations:

\n\n

CNN backbones:

\n\n

What did not work well (it worked but was disapointing):

\n\n

Data:

\n\n

Training procedure:

\n\n

Final ensemble:
\nEnsemble is a combination of models (8 to 10) with union strategy and with per-model threshold tuned on public LB (it overfitted for sure).
\nIdea of union was to capture most of the TP (better for metric used in this competition), drawback is that we capture FP too.
\nInference was fast, we cached resampled audio in memory, and (almost) full GPU pipeline helped. It ran in around 1h30 so we still had room for more models.

\n

Post-processing:
\nWe have a \"nobird\" majority vote to try to remove FP (force nocall) as some models have been trained with 265 classes instead of 264 (class 265 = nobird)

\n

Both our final submissions reached similar score on private LB.

\n

\"\"

\n

Final words to conclude: Congratulations to top teams! and thanks to @hidehisaarai1213, @hengck23 for their sharings.

", "messages": [ { "id": "1013357", "postDate": "09/16/2020 16:53:08", "content": "

It was my first audio competition and I really enjoyed it! Thanks to organizers and Kaggle for it.
\nThanks to my teammates @tikutiku @zfturbo the great collaboration, it was a pleasure and I've learnt a lot again.

\n

Here is a digest of our solution.

\n

Main Pipeline (see image below):

\n\n

Time Augmentations:

\n\n

Spec/Image augmentations:

\n\n

CNN backbones:

\n\n

What did not work well (it worked but was disapointing):

\n\n

Data:

\n\n

Training procedure:

\n\n

Final ensemble:
\nEnsemble is a combination of models (8 to 10) with union strategy and with per-model threshold tuned on public LB (it overfitted for sure).
\nIdea of union was to capture most of the TP (better for metric used in this competition), drawback is that we capture FP too.
\nInference was fast, we cached resampled audio in memory, and (almost) full GPU pipeline helped. It ran in around 1h30 so we still had room for more models.

\n

Post-processing:
\nWe have a \"nobird\" majority vote to try to remove FP (force nocall) as some models have been trained with 265 classes instead of 264 (class 265 = nobird)

\n

Both our final submissions reached similar score on private LB.

\n

\"\"

\n

Final words to conclude: Congratulations to top teams! and thanks to @hidehisaarai1213, @hengck23 for their sharings.

", "rawMarkdown": "It was my first audio competition and I really enjoyed it! Thanks to organizers and Kaggle for it.\nThanks to my teammates @tikutiku @zfturbo the great collaboration, it was a pleasure and I've learnt a lot again.\n\nHere is a digest of our solution.\n\n**Main Pipeline** (see image below):\n- MEL-Spectrogram (torchaudio) with differents settings to capture most birds frequency shapes.\n- Most processing/augmentation occurs in GPU to speed up training and inference.\n\n**Time Augmentations**:\n- Pink Noise/White noise\n- Time roll\n- Volume gain\n- PitchShift\n- Additional bandpass filters,lowcut (1kHz-2.5kHz), highcut (10kHz-15kHz)\n- Background/Ambient noise Mixup\n- Other birds Mixup (2 to 3 birds)\n\n**Spec/Image augmentations**:\n- Frequency/time masking\n- Color jitter\n\n**CNN backbones**:\n- EfficientNet B1 and B2\n- SEReseXt26\n- ResneSt50\n- Optional GRU layer to learn about time sequences (see [TALNet Sound Event Classifier](https://github.com/srvk/TALNet))\n- Different pooling to apply SED concept with either Attention block or max pooling\nWe also had one model based on 1D signal only (DensetNet1D)\n\nWhat did not work well (it worked but was disapointing):\n- Wavegram + MEL-Spectrogram model (was bad with soundscape)\n- Create large image with 3x2 grid spectrogram (was bad on inference)\n\n**Data**:\n- Train audio provided with duration outliers removed\n- Additional xeno-canto data (from Vopani dataset, thanks @rohanrao for your clean dump)\n- Distractors (NoBird/Nocall/Ambient) 10s slices (from freefield1010)\nWe've built our own validation data by mixing test_audio/birds/noise/nocall to try to correlate LB. It correlated a bit but was not enough to give trust in it. Too bad as it was key for this competition.\n\n**Training procedure**:\n- Stage1: Train a few models with 5s slices picked randomly, then save birds probabilities on CV OOF, ensemble all OOF models results to generate \"*hot*\" slices with high probabilities.\n It allowed to reach public LB=0.582\n- Stage2: Train more models with only such 5s *hot slices*.\n It allowed to reach public LB=0.596\nWhile reading other's solutions, we should have tried another stage with \"very hot\" slices to have a super clean train dataset.\n\n**Final ensemble**:\nEnsemble is a combination of models (8 to 10) with union strategy and with per-model threshold tuned on public LB (it overfitted for sure).\nIdea of union was to capture most of the TP (better for metric used in this competition), drawback is that we capture FP too.\nInference was fast, we cached resampled audio in memory, and (almost) full GPU pipeline helped. It ran in around 1h30 so we still had room for more models.\n\n**Post-processing**:\nWe have a \"nobird\" majority vote to try to remove FP (force nocall) as some models have been trained with 265 classes instead of 264 (class 265 = nobird)\n\nBoth our final submissions reached similar score on private LB.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2F4e2db4b7460eb1a51737fbf81bd8ed50%2Fpipeline.png?generation=1600274872405000&alt=media)\n\nFinal words to conclude: Congratulations to top teams! and thanks to @hidehisaarai1213, @hengck23 for their sharings.", "votes": null }, { "id": "1013387", "postDate": "09/16/2020 17:14:01", "content": "

Great…. Just one step away 👍

", "rawMarkdown": "Great.... Just one step away 👍", "votes": null }, { "id": "1013388", "postDate": "09/16/2020 17:14:44", "content": "

Thanks for sharing now and during discussions during the competition too.
\nIf possible, can you elaborate a bit more on this point please…

\n
\n

ensemble all OOF models results to generate \"hot\" slices with high probabilities.

\n
", "rawMarkdown": "Thanks for sharing now and during discussions during the competition too.\nIf possible, can you elaborate a bit more on this point please...\n>ensemble all OOF models results to generate \"hot\" slices with high probabilities.", "votes": null }, { "id": "1013394", "postDate": "09/16/2020 17:16:54", "content": "

Thanks, we had our best public LB submission with 0.647 on private LB (last gold) but we've selected the one with 0.646 that had more diversity in ensemble.

", "rawMarkdown": "Thanks, we had our best public LB submission with 0.647 on private LB (last gold) but we've selected the one with 0.646 that had more diversity in ensemble.", "votes": null }, { "id": "1013427", "postDate": "09/16/2020 17:29:47", "content": "

@watzisname On model training with cross-validation you can keep OOF (out-of-fold) predictions for each validation fold (nothing new here). You can store OOFs based on a sliding window (e.g. width=5s, step=1s) over each validation audio file. Do it for several models (with good LB) and you have, for each time slice of each audio, the probabilities of the ground truth. Average them and keep the best slices for each audio (e.g. probs > 0.9 or 0.7). That's way you've generated a subset with cleaned labels to you can use in a next step. That is what we called hot time slices.

", "rawMarkdown": "watzisname On model training with cross-validation you can keep OOF (out-of-fold) predictions for each validation fold (nothing new here). You can store OOFs based on a sliding window (e.g. width=5s, step=1s) over each validation audio file. Do it for several models (with good LB) and you have, for each time slice of each audio, the probabilities of the ground truth. Average them and keep the best slices for each audio (e.g. probs > 0.9 or 0.7). That's way you've generated a subset with cleaned labels to you can use in a next step. That is what we called hot time slices.", "votes": null }, { "id": "1013435", "postDate": "09/16/2020 17:33:30", "content": "

ok, since you saved the hot slices for only the OOF - that means for Stage 2 you have a limited amount of data (for 5 fold, only 1/5th of data). Is stage 2 effectively fine-tuning in that case ?

", "rawMarkdown": "ok, since you saved the hot slices for only the OOF - that means for Stage 2 you have a limited amount of data (for 5 fold, only 1/5th of data). Is stage 2 effectively fine-tuning in that case ?", "votes": null }, { "id": "1013441", "postDate": "09/16/2020 17:36:05", "content": "

For CV5 you have 5 OOF so at the end you've covered the full dataset. But that's true that stage2 dataset is smaller than stage1. For us, it reduced from 40k audio files to 32k audio files because we had 8k files with bad predictions.

", "rawMarkdown": "For CV5 you have 5 OOF so at the end you've covered the full dataset. But that's true that stage2 dataset is smaller than stage1. For us, it reduced from 40k audio files to 32k audio files because we had 8k files with bad predictions.", "votes": null }, { "id": "1013445", "postDate": "09/16/2020 17:37:36", "content": "

Thanks, understood - thats a great idea indeed.

", "rawMarkdown": "Thanks, understood - thats a great idea indeed.", "votes": null }, { "id": "1013451", "postDate": "09/16/2020 17:42:25", "content": "

Good labels, good models 😏. There are threads in the forum from @hengck23 about attempts/approaches to find such good time slices.

", "rawMarkdown": "Good labels, good models 😏. There are threads in the forum from @hengck23 about attempts/approaches to find such good time slices.", "votes": null }, { "id": "1013732", "postDate": "09/16/2020 23:03:36", "content": "

Thanks for sharing, finding hot slice sounds very nice!

", "rawMarkdown": "Thanks for sharing, finding hot slice sounds very nice!", "votes": null }, { "id": "1013881", "postDate": "09/17/2020 03:29:12", "content": "

@mpware

\n

hope for best…. there is always second chance…. good luck!

", "rawMarkdown": "mpware \n\nhope for best.... there is always second chance.... good luck!", "votes": null }, { "id": "1025198", "postDate": "09/24/2020 11:58:11", "content": "

Thanks for sharing and congrats!

\n
\n

Wavegram + MEL-Spectrogram model (was bad with soundscape)
\n Create large image with 3x2 grid spectrogram (was bad on inference)

\n
\n

About these two points, could you talk more details like how to fuse waveform and spec, and what is 3x2 grid spectrogram.

\n

Since they look like very useful, though they didn't work this time.

", "rawMarkdown": "Thanks for sharing and congrats!\n\n> Wavegram + MEL-Spectrogram model (was bad with soundscape)\n> Create large image with 3x2 grid spectrogram (was bad on inference)\n\nAbout these two points, could you talk more details like how to fuse waveform and spec, and what is `3x2 grid spectrogram`. \n\nSince they look like very useful, though they didn't work this time.", "votes": null } ], "comments": [ { "id": 1013387, "author_name": "gopidurgaprasad", "author_url": "", "post_date": "09/16/2020 17:14:01", "content": "

Great…. Just one step away 👍

", "votes": null, "replies": [ { "id": 1013394, "author_name": "mpware", "author_url": "", "post_date": "09/16/2020 17:16:54", "content": "

Thanks, we had our best public LB submission with 0.647 on private LB (last gold) but we've selected the one with 0.646 that had more diversity in ensemble.

", "votes": null, "replies": [] }, { "id": 1013881, "author_name": "gopidurgaprasad", "author_url": "", "post_date": "09/17/2020 03:29:12", "content": "

@mpware

\n

hope for best…. there is always second chance…. good luck!

", "votes": null, "replies": [] } ] }, { "id": 1013388, "author_name": "watzisname", "author_url": "", "post_date": "09/16/2020 17:14:44", "content": "

Thanks for sharing now and during discussions during the competition too.
\nIf possible, can you elaborate a bit more on this point please…

\n
\n

ensemble all OOF models results to generate \"hot\" slices with high probabilities.

\n
", "votes": null, "replies": [ { "id": 1013427, "author_name": "mpware", "author_url": "", "post_date": "09/16/2020 17:29:47", "content": "

@watzisname On model training with cross-validation you can keep OOF (out-of-fold) predictions for each validation fold (nothing new here). You can store OOFs based on a sliding window (e.g. width=5s, step=1s) over each validation audio file. Do it for several models (with good LB) and you have, for each time slice of each audio, the probabilities of the ground truth. Average them and keep the best slices for each audio (e.g. probs > 0.9 or 0.7). That's way you've generated a subset with cleaned labels to you can use in a next step. That is what we called hot time slices.

", "votes": null, "replies": [] }, { "id": 1013435, "author_name": "watzisname", "author_url": "", "post_date": "09/16/2020 17:33:30", "content": "

ok, since you saved the hot slices for only the OOF - that means for Stage 2 you have a limited amount of data (for 5 fold, only 1/5th of data). Is stage 2 effectively fine-tuning in that case ?

", "votes": null, "replies": [] }, { "id": 1013441, "author_name": "mpware", "author_url": "", "post_date": "09/16/2020 17:36:05", "content": "

For CV5 you have 5 OOF so at the end you've covered the full dataset. But that's true that stage2 dataset is smaller than stage1. For us, it reduced from 40k audio files to 32k audio files because we had 8k files with bad predictions.

", "votes": null, "replies": [] }, { "id": 1013445, "author_name": "watzisname", "author_url": "", "post_date": "09/16/2020 17:37:36", "content": "

Thanks, understood - thats a great idea indeed.

", "votes": null, "replies": [] }, { "id": 1013451, "author_name": "mpware", "author_url": "", "post_date": "09/16/2020 17:42:25", "content": "

Good labels, good models 😏. There are threads in the forum from @hengck23 about attempts/approaches to find such good time slices.

", "votes": null, "replies": [] } ] }, { "id": 1013732, "author_name": "daisukelab", "author_url": "", "post_date": "09/16/2020 23:03:36", "content": "

Thanks for sharing, finding hot slice sounds very nice!

", "votes": null, "replies": [] }, { "id": 1025198, "author_name": "karlyukang", "author_url": "", "post_date": "09/24/2020 11:58:11", "content": "

Thanks for sharing and congrats!

\n
\n

Wavegram + MEL-Spectrogram model (was bad with soundscape)
\n Create large image with 3x2 grid spectrogram (was bad on inference)

\n
\n

About these two points, could you talk more details like how to fuse waveform and spec, and what is 3x2 grid spectrogram.

\n

Since they look like very useful, though they didn't work this time.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1013357": "It was my first audio competition and I really enjoyed it! Thanks to organizers and Kaggle for it.\nThanks to my teammates @tikutiku @zfturbo the great collaboration, it was a pleasure and I've learnt a lot again.\n\nHere is a digest of our solution.\n\n**Main Pipeline** (see image below):\n- MEL-Spectrogram (torchaudio) with differents settings to capture most birds frequency shapes.\n- Most processing/augmentation occurs in GPU to speed up training and inference.\n\n**Time Augmentations**:\n- Pink Noise/White noise\n- Time roll\n- Volume gain\n- PitchShift\n- Additional bandpass filters,lowcut (1kHz-2.5kHz), highcut (10kHz-15kHz)\n- Background/Ambient noise Mixup\n- Other birds Mixup (2 to 3 birds)\n\n**Spec/Image augmentations**:\n- Frequency/time masking\n- Color jitter\n\n**CNN backbones**:\n- EfficientNet B1 and B2\n- SEReseXt26\n- ResneSt50\n- Optional GRU layer to learn about time sequences (see [TALNet Sound Event Classifier](https://github.com/srvk/TALNet))\n- Different pooling to apply SED concept with either Attention block or max pooling\nWe also had one model based on 1D signal only (DensetNet1D)\n\nWhat did not work well (it worked but was disapointing):\n- Wavegram + MEL-Spectrogram model (was bad with soundscape)\n- Create large image with 3x2 grid spectrogram (was bad on inference)\n\n**Data**:\n- Train audio provided with duration outliers removed\n- Additional xeno-canto data (from Vopani dataset, thanks @rohanrao for your clean dump)\n- Distractors (NoBird/Nocall/Ambient) 10s slices (from freefield1010)\nWe've built our own validation data by mixing test_audio/birds/noise/nocall to try to correlate LB. It correlated a bit but was not enough to give trust in it. Too bad as it was key for this competition.\n\n**Training procedure**:\n- Stage1: Train a few models with 5s slices picked randomly, then save birds probabilities on CV OOF, ensemble all OOF models results to generate \"*hot*\" slices with high probabilities.\n It allowed to reach public LB=0.582\n- Stage2: Train more models with only such 5s *hot slices*.\n It allowed to reach public LB=0.596\nWhile reading other's solutions, we should have tried another stage with \"very hot\" slices to have a super clean train dataset.\n\n**Final ensemble**:\nEnsemble is a combination of models (8 to 10) with union strategy and with per-model threshold tuned on public LB (it overfitted for sure).\nIdea of union was to capture most of the TP (better for metric used in this competition), drawback is that we capture FP too.\nInference was fast, we cached resampled audio in memory, and (almost) full GPU pipeline helped. It ran in around 1h30 so we still had room for more models.\n\n**Post-processing**:\nWe have a \"nobird\" majority vote to try to remove FP (force nocall) as some models have been trained with 265 classes instead of 264 (class 265 = nobird)\n\nBoth our final submissions reached similar score on private LB.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2F4e2db4b7460eb1a51737fbf81bd8ed50%2Fpipeline.png?generation=1600274872405000&alt=media)\n\nFinal words to conclude: Congratulations to top teams! and thanks to @hidehisaarai1213, @hengck23 for their sharings.", "1013387": "Great.... Just one step away 👍", "1013388": "Thanks for sharing now and during discussions during the competition too.\nIf possible, can you elaborate a bit more on this point please...\n>ensemble all OOF models results to generate \"hot\" slices with high probabilities.", "1013394": "Thanks, we had our best public LB submission with 0.647 on private LB (last gold) but we've selected the one with 0.646 that had more diversity in ensemble.", "1013427": "watzisname On model training with cross-validation you can keep OOF (out-of-fold) predictions for each validation fold (nothing new here). You can store OOFs based on a sliding window (e.g. width=5s, step=1s) over each validation audio file. Do it for several models (with good LB) and you have, for each time slice of each audio, the probabilities of the ground truth. Average them and keep the best slices for each audio (e.g. probs > 0.9 or 0.7). That's way you've generated a subset with cleaned labels to you can use in a next step. That is what we called hot time slices.", "1013435": "ok, since you saved the hot slices for only the OOF - that means for Stage 2 you have a limited amount of data (for 5 fold, only 1/5th of data). Is stage 2 effectively fine-tuning in that case ?", "1013441": "For CV5 you have 5 OOF so at the end you've covered the full dataset. But that's true that stage2 dataset is smaller than stage1. For us, it reduced from 40k audio files to 32k audio files because we had 8k files with bad predictions.", "1013445": "Thanks, understood - thats a great idea indeed.", "1013451": "Good labels, good models 😏. There are threads in the forum from @hengck23 about attempts/approaches to find such good time slices.", "1013732": "Thanks for sharing, finding hot slice sounds very nice!", "1013881": "mpware \n\nhope for best.... there is always second chance.... good luck!", "1025198": "Thanks for sharing and congrats!\n\n> Wavegram + MEL-Spectrogram model (was bad with soundscape)\n> Create large image with 3x2 grid spectrogram (was bad on inference)\n\nAbout these two points, could you talk more details like how to fuse waveform and spec, and what is `3x2 grid spectrogram`. \n\nSince they look like very useful, though they didn't work this time." }, "source": "meta" }