{ "id": 220436, "title": "21st place solution - FP co-teaching with loss improvements", "url": "/competitions/rfcx-species-audio-detection/writeups/vasiliy-kotov-21st-place-solution-fp-co-teaching-w", "author_name": "", "post_date": "2021-02-18T21:32:20.757Z", "votes": 22, "comment_count": 4, "views": 0, "content": "

Congratulations to the top finishers!

\n

The whole my solution is here: https://github.com/MPGek/mpgek-rfcx-species-audio-detection-public.
\nI left the best configs that were used in the final submission.

\n

Summary

\n

Summary of my solution:

\n\n

Spectrogram and random crop

\n

For the training, I used a random crop with the centered sample in 10 seconds and a random crop for 6 seconds.
\nFor validation and prediction, I used 6 seconds crop with a stride in 2 seconds with maximizing outputs.
\nFor the mel spectrogram, I used the following parameters:

\n\n

So one sample in 6 seconds produced an image with size 720 x 380 pixels

\n

Augmentations

\n

Augmentation that improves LB:

\n\n

Example of the Frequency band filtering (top - original sample, bottom - sample after filtering):
\n\"\"
\nExample of the Frequency band mixup (top - original sample, bottom - sample after mixup):
\n\"\"
\nAugmentation that doesn't improve LB:

\n\n

Network topology

\n

I got the best results on EfficientNetB2, B4, B7 (noisy students weights) with a simple FC to 24 classes after the adaptive average pool.

\n

I tried to use different head but all of them gave the same or worse result:

\n\n

TP training

\n

Based on the post where described that every sound file can present unlabeled samples I have to work noisy samples.

\n

I split all samples into confident samples and noisy samples:

\n\n

For the confident samples, I used simple BCE. For the noisy samples, I used LSoft with beta 0.7 (https://arxiv.org/pdf/1901.01189.pdf).
\nLSoft:

\n
def forward(self, input: torch.Tensor, target: torch.Tensor):\n        with torch.no_grad():\n            pred = torch.sigmoid(input)\n            target_update = self.beta * target + (1 - self.beta) * pred\n        loss = F.binary_cross_entropy_with_logits(input, target_update, reduction=self.reduction)\n        return loss\n
\n

In the loss function, I flatten all outputs (even batch dim) to a linear array. And split items into 2 arrays where targets were 1 and where targets were 0.

\n

With LSoft I got on EfficientNetB7 0.912-0.915 LB.
\nWithout LSoft - BCE for all samples I got only about 0.895-0.900.

\n

FP training

\n

For the FP training, I used a dataset with undersampling of the FP samples. Each epoch had all TP samples and the same count of the FP samples.
\nI used batch sampling to provide a balanced batch of TP/FP samples - after each TP I added FP with the same species id.

\n

In the loss function, I calculate loss only for those sigmoid outputs that present in train_tp.csv or train_fp.csv. So all noisy samples are ignored.

\n

FP co-teaching training

\n

I have tried to find a way how to use FOCI or SELFIE to work with noisy data, but all of them use historical predictions of each sample. With my random crop and frequency band mixup it's almost impossible. Even shift for 0.5-1 seconds can add a new species to the sample. So historical data will be incorrect.

\n

I tried co-teaching training because it doesn't require historical data.
\nPaper: https://arxiv.org/pdf/1804.06872.pdf
\nCode sample: https://github.com/bhanML/Co-teaching

\n

When I implemented co-teaching training I have only 5 days before the deadline.
\nThe first experiments with co-teaching gave 0.830 LB for the TP and 0.880 LB for the FP. So it looked like a bad experiment.

\n

I tried to improve loss function by adding high loss samples with changed targets (by default co-teaching should ignore high loss samples as wrong samples).

\n

The final loss function consists of:

\n\n

When I invented this loss I had only 3 days before the deadline.
\nThis loss has a good potential for future experiments. I trained only 2 experiments with this loss, both with the same hyperparameters. The first one had a bug so it produces extremely high metrics and used wrong epochs to submission - however, even with the bug it produces a good LB score comparing to the original FP training.

\n

Folds and best epoch metric

\n

To chose the best epoch in all training I used the BCE which calculated only on confident samples.
\nIn some experiments I used 5 stratified KFold, in some, I used 7 folds with stratified shuffle split with test size 0.3.

\n

Ensembles

\n

The TP training with EfficientNetB7 gave me only 0.912-0.915 on the public LB.
\nThe FP training with EfficientNetB2-B4 gave only 0.887-0.893 LB.
\nThe ensemble of the TP and FP gave 0.929 LB.

\n

The FP co-teaching training on simple EfficientNetB2 gave me 0.925 LB (a quite good improvement from the original FP with 0.893)

\n

The final ensemble consists of all best experiments (0.941 public LB and 0.944 private LB):

\n", "messages": [ { "id": "1208460", "postDate": "02/18/2021 09:46:55", "content": "

Congratulations to the top finishers!

\n

The whole my solution is here: https://github.com/MPGek/mpgek-rfcx-species-audio-detection-public.
\nI left the best configs that were used in the final submission.

\n

Summary

\n

Summary of my solution:

\n\n

Spectrogram and random crop

\n

For the training, I used a random crop with the centered sample in 10 seconds and a random crop for 6 seconds.
\nFor validation and prediction, I used 6 seconds crop with a stride in 2 seconds with maximizing outputs.
\nFor the mel spectrogram, I used the following parameters:

\n\n

So one sample in 6 seconds produced an image with size 720 x 380 pixels

\n

Augmentations

\n

Augmentation that improves LB:

\n\n

Example of the Frequency band filtering (top - original sample, bottom - sample after filtering):
\n\"\"
\nExample of the Frequency band mixup (top - original sample, bottom - sample after mixup):
\n\"\"
\nAugmentation that doesn't improve LB:

\n\n

Network topology

\n

I got the best results on EfficientNetB2, B4, B7 (noisy students weights) with a simple FC to 24 classes after the adaptive average pool.

\n

I tried to use different head but all of them gave the same or worse result:

\n\n

TP training

\n

Based on the post where described that every sound file can present unlabeled samples I have to work noisy samples.

\n

I split all samples into confident samples and noisy samples:

\n\n

For the confident samples, I used simple BCE. For the noisy samples, I used LSoft with beta 0.7 (https://arxiv.org/pdf/1901.01189.pdf).
\nLSoft:

\n
def forward(self, input: torch.Tensor, target: torch.Tensor):\n        with torch.no_grad():\n            pred = torch.sigmoid(input)\n            target_update = self.beta * target + (1 - self.beta) * pred\n        loss = F.binary_cross_entropy_with_logits(input, target_update, reduction=self.reduction)\n        return loss\n
\n

In the loss function, I flatten all outputs (even batch dim) to a linear array. And split items into 2 arrays where targets were 1 and where targets were 0.

\n

With LSoft I got on EfficientNetB7 0.912-0.915 LB.
\nWithout LSoft - BCE for all samples I got only about 0.895-0.900.

\n

FP training

\n

For the FP training, I used a dataset with undersampling of the FP samples. Each epoch had all TP samples and the same count of the FP samples.
\nI used batch sampling to provide a balanced batch of TP/FP samples - after each TP I added FP with the same species id.

\n

In the loss function, I calculate loss only for those sigmoid outputs that present in train_tp.csv or train_fp.csv. So all noisy samples are ignored.

\n

FP co-teaching training

\n

I have tried to find a way how to use FOCI or SELFIE to work with noisy data, but all of them use historical predictions of each sample. With my random crop and frequency band mixup it's almost impossible. Even shift for 0.5-1 seconds can add a new species to the sample. So historical data will be incorrect.

\n

I tried co-teaching training because it doesn't require historical data.
\nPaper: https://arxiv.org/pdf/1804.06872.pdf
\nCode sample: https://github.com/bhanML/Co-teaching

\n

When I implemented co-teaching training I have only 5 days before the deadline.
\nThe first experiments with co-teaching gave 0.830 LB for the TP and 0.880 LB for the FP. So it looked like a bad experiment.

\n

I tried to improve loss function by adding high loss samples with changed targets (by default co-teaching should ignore high loss samples as wrong samples).

\n

The final loss function consists of:

\n\n

When I invented this loss I had only 3 days before the deadline.
\nThis loss has a good potential for future experiments. I trained only 2 experiments with this loss, both with the same hyperparameters. The first one had a bug so it produces extremely high metrics and used wrong epochs to submission - however, even with the bug it produces a good LB score comparing to the original FP training.

\n

Folds and best epoch metric

\n

To chose the best epoch in all training I used the BCE which calculated only on confident samples.
\nIn some experiments I used 5 stratified KFold, in some, I used 7 folds with stratified shuffle split with test size 0.3.

\n

Ensembles

\n

The TP training with EfficientNetB7 gave me only 0.912-0.915 on the public LB.
\nThe FP training with EfficientNetB2-B4 gave only 0.887-0.893 LB.
\nThe ensemble of the TP and FP gave 0.929 LB.

\n

The FP co-teaching training on simple EfficientNetB2 gave me 0.925 LB (a quite good improvement from the original FP with 0.893)

\n

The final ensemble consists of all best experiments (0.941 public LB and 0.944 private LB):

\n", "rawMarkdown": "Congratulations to the top finishers!\n\nThe whole my solution is here: https://github.com/MPGek/mpgek-rfcx-species-audio-detection-public.\nI left the best configs that were used in the final submission.\n\n## Summary\nSummary of my solution:\n- General augmentations with frequency band augmentations:\n - Frequency band filtering\n - Frequency band mixup\n- TP training with a combined loss of the BCE for confident samples and LSoft 0.7 for noisy samples\n- FP training with a BCE for confident samples and ignored losses for samples with noisy samples\n- FP co-teaching training with loss as described in the co-teaching paper, but with the extra loss for the high loss samples\n- Ensemble of TP, FP, FP co-teaching results.\n\n## Spectrogram and random crop\nFor the training, I used a random crop with the centered sample in 10 seconds and a random crop for 6 seconds.\nFor validation and prediction, I used 6 seconds crop with a stride in 2 seconds with maximizing outputs.\nFor the mel spectrogram, I used the following parameters:\n- mels count: 380\n- FTT: 4096\n- window length: 1536\n- hop length: 400\n- fmin: 50\n- fmax: 15000\n\nSo one sample in 6 seconds produced an image with size 720 x 380 pixels\n\n## Augmentations\n**Augmentation that improves LB:**\n- Gaussian noise\n- Random crop + resize with size reduction in 40 and 20 pixels\n- Frequency band filtering - based on f_min and f_max of the sample I set 0 values to all mels that lower or higher than f_min and f_max with a sigmoid transition to remove sharp edges\n - Frequency band mixup - for some samples I used frequency band filtering and then I mixed it with different samples with a band that higher than f_max and with a band that lower than f_min. So I managed to get a single image with the 3 mixed samples.\n\n**Example of the Frequency band filtering (top - original sample, bottom - sample after filtering):**\n![](https://github.com/MPGek/mpgek-rfcx-species-audio-detection-public/raw/main/img/Band%20filtering.png)\n**Example of the Frequency band mixup (top - original sample, bottom - sample after mixup):**\n![](https://raw.githubusercontent.com/MPGek/mpgek-rfcx-species-audio-detection-public/main/img/Band%20mixup.png)\n**Augmentation that doesn't improve LB:**\n- SpecAugment\n- Sum mixup\n\n## Network topology\n\nI got the best results on EfficientNetB2, B4, B7 (noisy students weights) with a simple FC to 24 classes after the adaptive average pool.\n\nI tried to use different head but all of them gave the same or worse result:\n- the hyper column\n- the hyper column with auxiliary losses after each pooling\n- extra spatial and channel attentions blocks - CBAM\n- dense convolutions\n\n## TP training\nBased on the post where described that every sound file can present unlabeled samples I have to work noisy samples.\n\nI split all samples into confident samples and noisy samples:\n- confident samples - all sigmoid outputs for classes which present in the train_tp.csv file (1 in targets tensor)\n- noisy samples - all sigmoid outputs for classes which not described in the train_tp.csv file (0 in target tensor)\n\nFor the confident samples, I used simple BCE. For the noisy samples, I used LSoft with beta 0.7 (https://arxiv.org/pdf/1901.01189.pdf). \nLSoft:\n```\ndef forward(self, input: torch.Tensor, target: torch.Tensor):\n with torch.no_grad():\n pred = torch.sigmoid(input)\n target_update = self.beta * target + (1 - self.beta) * pred\n loss = F.binary_cross_entropy_with_logits(input, target_update, reduction=self.reduction)\n return loss\n```\n\nIn the loss function, I flatten all outputs (even batch dim) to a linear array. And split items into 2 arrays where targets were 1 and where targets were 0.\n\nWith LSoft I got on EfficientNetB7 0.912-0.915 LB.\nWithout LSoft - BCE for all samples I got only about 0.895-0.900.\n\n## FP training\nFor the FP training, I used a dataset with undersampling of the FP samples. Each epoch had all TP samples and the same count of the FP samples.\nI used batch sampling to provide a balanced batch of TP/FP samples - after each TP I added FP with the same species id.\n\nIn the loss function, I calculate loss only for those sigmoid outputs that present in train_tp.csv or train_fp.csv. So all noisy samples are ignored.\n\n## FP co-teaching training\nI have tried to find a way how to use FOCI or SELFIE to work with noisy data, but all of them use historical predictions of each sample. With my random crop and frequency band mixup it's almost impossible. Even shift for 0.5-1 seconds can add a new species to the sample. So historical data will be incorrect.\n\nI tried co-teaching training because it doesn't require historical data.\nPaper: https://arxiv.org/pdf/1804.06872.pdf\nCode sample: https://github.com/bhanML/Co-teaching\n\nWhen I implemented co-teaching training I have only 5 days before the deadline.\nThe first experiments with co-teaching gave 0.830 LB for the TP and 0.880 LB for the FP. So it looked like a bad experiment.\n\nI tried to improve loss function by adding high loss samples with changed targets (by default co-teaching should ignore high loss samples as wrong samples).\n\nThe final loss function consists of:\n- 50% lowest loss samples (all confident samples mandatory added to this part of loss with scale factor 2)\n- 45% ignored losses\n- 5% highest loss samples with the changed target (1 for predictions with sigmoid >= 0.5 and 0 for sigmoid < 0.5)\nThe loss implementation is here: https://github.com/MPGek/mpgek-rfcx-species-audio-detection-public/blob/main/model/forward_passes_coteaching.py\n\nWhen I invented this loss I had only 3 days before the deadline.\nThis loss has a good potential for future experiments. I trained only 2 experiments with this loss, both with the same hyperparameters. The first one had a bug so it produces extremely high metrics and used wrong epochs to submission - however, even with the bug it produces a good LB score comparing to the original FP training.\n\n## Folds and best epoch metric\nTo chose the best epoch in all training I used the BCE which calculated only on confident samples.\nIn some experiments I used 5 stratified KFold, in some, I used 7 folds with stratified shuffle split with test size 0.3.\n\n## Ensembles\nThe TP training with EfficientNetB7 gave me only 0.912-0.915 on the public LB.\nThe FP training with EfficientNetB2-B4 gave only 0.887-0.893 LB.\nThe ensemble of the TP and FP gave 0.929 LB.\n\nThe FP co-teaching training on simple EfficientNetB2 gave me 0.925 LB (a quite good improvement from the original FP with 0.893)\n\n**The final ensemble consists of all best experiments (0.941 public LB and 0.944 private LB):**\n- TP EfficientNetB7 with 0.915\n- FP EfficientNetB2-B4 with 0.893\n- FP co-teaching EfficientNetB2 with 0.925", "votes": null }, { "id": "1208693", "postDate": "02/18/2021 12:47:12", "content": "

Good job, congrats on results and thanks for sharing solution and code @xmpgek

", "rawMarkdown": "Good job, congrats on results and thanks for sharing solution and code @xmpgek", "votes": null }, { "id": "1209457", "postDate": "02/18/2021 22:56:41", "content": "

Thanks for sharing. And congrats on the good result. Also, thanks for mentioning yet another teacher/student paper I didn't know. I'll read it for sure.

", "rawMarkdown": "Thanks for sharing. And congrats on the good result. Also, thanks for mentioning yet another teacher/student paper I didn't know. I'll read it for sure.", "votes": null }, { "id": "1219713", "postDate": "02/27/2021 06:57:26", "content": "

Thanks, I have starred!

", "rawMarkdown": "Thanks, I have starred!", "votes": null }, { "id": "1255080", "postDate": "03/28/2021 11:58:36", "content": "

I want to ask something about your code. In RainforestModel, the self.is_base_resnet is always set to False, why is that?

", "rawMarkdown": "I want to ask something about your code. In `RainforestModel`, the `self.is_base_resnet` is always set to `False`, why is that?", "votes": null } ], "comments": [ { "id": 1208693, "author_name": "duykhanh99", "author_url": "", "post_date": "02/18/2021 12:47:12", "content": "

Good job, congrats on results and thanks for sharing solution and code @xmpgek

", "votes": null, "replies": [] }, { "id": 1209457, "author_name": "cpmpml", "author_url": "", "post_date": "02/18/2021 22:56:41", "content": "

Thanks for sharing. And congrats on the good result. Also, thanks for mentioning yet another teacher/student paper I didn't know. I'll read it for sure.

", "votes": null, "replies": [] }, { "id": 1219713, "author_name": "wubinbai", "author_url": "", "post_date": "02/27/2021 06:57:26", "content": "

Thanks, I have starred!

", "votes": null, "replies": [] }, { "id": 1255080, "author_name": "faraksuli", "author_url": "", "post_date": "03/28/2021 11:58:36", "content": "

I want to ask something about your code. In RainforestModel, the self.is_base_resnet is always set to False, why is that?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1208460": "Congratulations to the top finishers!\n\nThe whole my solution is here: https://github.com/MPGek/mpgek-rfcx-species-audio-detection-public.\nI left the best configs that were used in the final submission.\n\n## Summary\nSummary of my solution:\n- General augmentations with frequency band augmentations:\n - Frequency band filtering\n - Frequency band mixup\n- TP training with a combined loss of the BCE for confident samples and LSoft 0.7 for noisy samples\n- FP training with a BCE for confident samples and ignored losses for samples with noisy samples\n- FP co-teaching training with loss as described in the co-teaching paper, but with the extra loss for the high loss samples\n- Ensemble of TP, FP, FP co-teaching results.\n\n## Spectrogram and random crop\nFor the training, I used a random crop with the centered sample in 10 seconds and a random crop for 6 seconds.\nFor validation and prediction, I used 6 seconds crop with a stride in 2 seconds with maximizing outputs.\nFor the mel spectrogram, I used the following parameters:\n- mels count: 380\n- FTT: 4096\n- window length: 1536\n- hop length: 400\n- fmin: 50\n- fmax: 15000\n\nSo one sample in 6 seconds produced an image with size 720 x 380 pixels\n\n## Augmentations\n**Augmentation that improves LB:**\n- Gaussian noise\n- Random crop + resize with size reduction in 40 and 20 pixels\n- Frequency band filtering - based on f_min and f_max of the sample I set 0 values to all mels that lower or higher than f_min and f_max with a sigmoid transition to remove sharp edges\n - Frequency band mixup - for some samples I used frequency band filtering and then I mixed it with different samples with a band that higher than f_max and with a band that lower than f_min. So I managed to get a single image with the 3 mixed samples.\n\n**Example of the Frequency band filtering (top - original sample, bottom - sample after filtering):**\n![](https://github.com/MPGek/mpgek-rfcx-species-audio-detection-public/raw/main/img/Band%20filtering.png)\n**Example of the Frequency band mixup (top - original sample, bottom - sample after mixup):**\n![](https://raw.githubusercontent.com/MPGek/mpgek-rfcx-species-audio-detection-public/main/img/Band%20mixup.png)\n**Augmentation that doesn't improve LB:**\n- SpecAugment\n- Sum mixup\n\n## Network topology\n\nI got the best results on EfficientNetB2, B4, B7 (noisy students weights) with a simple FC to 24 classes after the adaptive average pool.\n\nI tried to use different head but all of them gave the same or worse result:\n- the hyper column\n- the hyper column with auxiliary losses after each pooling\n- extra spatial and channel attentions blocks - CBAM\n- dense convolutions\n\n## TP training\nBased on the post where described that every sound file can present unlabeled samples I have to work noisy samples.\n\nI split all samples into confident samples and noisy samples:\n- confident samples - all sigmoid outputs for classes which present in the train_tp.csv file (1 in targets tensor)\n- noisy samples - all sigmoid outputs for classes which not described in the train_tp.csv file (0 in target tensor)\n\nFor the confident samples, I used simple BCE. For the noisy samples, I used LSoft with beta 0.7 (https://arxiv.org/pdf/1901.01189.pdf). \nLSoft:\n```\ndef forward(self, input: torch.Tensor, target: torch.Tensor):\n with torch.no_grad():\n pred = torch.sigmoid(input)\n target_update = self.beta * target + (1 - self.beta) * pred\n loss = F.binary_cross_entropy_with_logits(input, target_update, reduction=self.reduction)\n return loss\n```\n\nIn the loss function, I flatten all outputs (even batch dim) to a linear array. And split items into 2 arrays where targets were 1 and where targets were 0.\n\nWith LSoft I got on EfficientNetB7 0.912-0.915 LB.\nWithout LSoft - BCE for all samples I got only about 0.895-0.900.\n\n## FP training\nFor the FP training, I used a dataset with undersampling of the FP samples. Each epoch had all TP samples and the same count of the FP samples.\nI used batch sampling to provide a balanced batch of TP/FP samples - after each TP I added FP with the same species id.\n\nIn the loss function, I calculate loss only for those sigmoid outputs that present in train_tp.csv or train_fp.csv. So all noisy samples are ignored.\n\n## FP co-teaching training\nI have tried to find a way how to use FOCI or SELFIE to work with noisy data, but all of them use historical predictions of each sample. With my random crop and frequency band mixup it's almost impossible. Even shift for 0.5-1 seconds can add a new species to the sample. So historical data will be incorrect.\n\nI tried co-teaching training because it doesn't require historical data.\nPaper: https://arxiv.org/pdf/1804.06872.pdf\nCode sample: https://github.com/bhanML/Co-teaching\n\nWhen I implemented co-teaching training I have only 5 days before the deadline.\nThe first experiments with co-teaching gave 0.830 LB for the TP and 0.880 LB for the FP. So it looked like a bad experiment.\n\nI tried to improve loss function by adding high loss samples with changed targets (by default co-teaching should ignore high loss samples as wrong samples).\n\nThe final loss function consists of:\n- 50% lowest loss samples (all confident samples mandatory added to this part of loss with scale factor 2)\n- 45% ignored losses\n- 5% highest loss samples with the changed target (1 for predictions with sigmoid >= 0.5 and 0 for sigmoid < 0.5)\nThe loss implementation is here: https://github.com/MPGek/mpgek-rfcx-species-audio-detection-public/blob/main/model/forward_passes_coteaching.py\n\nWhen I invented this loss I had only 3 days before the deadline.\nThis loss has a good potential for future experiments. I trained only 2 experiments with this loss, both with the same hyperparameters. The first one had a bug so it produces extremely high metrics and used wrong epochs to submission - however, even with the bug it produces a good LB score comparing to the original FP training.\n\n## Folds and best epoch metric\nTo chose the best epoch in all training I used the BCE which calculated only on confident samples.\nIn some experiments I used 5 stratified KFold, in some, I used 7 folds with stratified shuffle split with test size 0.3.\n\n## Ensembles\nThe TP training with EfficientNetB7 gave me only 0.912-0.915 on the public LB.\nThe FP training with EfficientNetB2-B4 gave only 0.887-0.893 LB.\nThe ensemble of the TP and FP gave 0.929 LB.\n\nThe FP co-teaching training on simple EfficientNetB2 gave me 0.925 LB (a quite good improvement from the original FP with 0.893)\n\n**The final ensemble consists of all best experiments (0.941 public LB and 0.944 private LB):**\n- TP EfficientNetB7 with 0.915\n- FP EfficientNetB2-B4 with 0.893\n- FP co-teaching EfficientNetB2 with 0.925", "1208693": "Good job, congrats on results and thanks for sharing solution and code @xmpgek", "1209457": "Thanks for sharing. And congrats on the good result. Also, thanks for mentioning yet another teacher/student paper I didn't know. I'll read it for sure.", "1219713": "Thanks, I have starred!", "1255080": "I want to ask something about your code. In `RainforestModel`, the `self.is_base_resnet` is always set to `False`, why is that?" }, "source": "meta" }