![\"solution\"](\"https://user-images.githubusercontent.com/69585273/170442235-ef15296d-33dd-475c-80fc-9e3482ca1278.png\")
Thank you to every competitor and host that held this competition.
\nThis is my solution and what I tried.
As I have posted in this discussion, I believe that three aspects of this competition needed to be considered.
\nGet high CV score
\nThis is a given, but a better model should be used to improve cross-validation scores.
\nI have the impression that many people used the improved PANNS model used in the Notebook published by @hidehisaarai1213 in the 2021 Bird Competition. I also used this model and reached CV:0.789 (f1 Score), Public:0.739, and Private:0.711.
\nThen, in the middle of the competition, I switched to use another model (that I'll explain below.) and improved to CV:0.805(f1 Score), Public:0.7634, Private:0.7210 (best with a single model).
Measures against Domain Shift
\nThere were several models with high CV but low LB. This is probably due to the existence of a domain shift between training data and test data. As a countermeasure, Augmentation, which inserts noise, was effective. In particular, Gaussian Noise and Pink Noise worked effectively.
\nModels trained with these augmentations got relatively low CV scores, but the Public scores improved.
Selecting the Appropriate Threshold
\nIn this competition, since there were no Test Soundscapes (data collected under the same conditions as the test data), the threshold had to be set manually. This appropriate threshold was usually far from the value that was optimal during the Validation phase of the model, so tuning was an important step. Since the Public Score was the only available factor to choose threshold, we had no choice but to overfit the Public Score (despite this, LB did not shake as much as we expected).
As figure shows, the sounddata was converted into a 20-second segmented mel-spectrogram, and then Augmentation (Normalize, Pink Noise) was added.
\nAugmentation (model1: Normalize&PinkNoise, model2: Normalize&GaussianNoise&RandomVolume) was added to the input, and the features were extracted using ResNet101d(model1) and ResNeXt50d_32x4d(model2).
\nThe output is a batch size × channel × frequency × time (four-dimensional).
\nThen, the features are compressed in the frequency and time directions using GeM pooling, and the result was returned to a two-dimensional tensor of batch size × channel.
\nThen fc layer is applied to obtain a 152-dimensional output.
\nI also applied MixUp as augmentation with a probability of 0.25. I believe that the robustness of the model has been improved, albeit only by a small margin.
\nAs a loss function, I used Focal loss based on BCEWithLogitsLoss (https://www.kaggle.com/competitions/rfcx-species-audio-detection/discussion/213075 ).
\nI also used SAM Optimizer based on Adam as the Optimizer (https://github.com/davda54/sam ).
\nBy changing the Optimizer to SAM, the Public Score for the Sound Event Detection model improved by about 0.02 when the experiment was conducted under all conditions except for the Optimizer.
Instead of passing the model-by-model average of the output predictions through the threshold, the logical OR or logical product of the true/false values after passing through the threshold was
\nsubmitted as the final prediction result. By applying an ensemble that takes Logical OR, I obtained an output result with a Private score of 0.7292, from the predictions whose private score are 0.7176 and 0.7008 (I we believe they worked effectively)
Thank you to every competitor and host that held this competition.
\nThis is my solution and what I tried.
As I have posted in this discussion, I believe that three aspects of this competition needed to be considered.
\nGet high CV score
\nThis is a given, but a better model should be used to improve cross-validation scores.
\nI have the impression that many people used the improved PANNS model used in the Notebook published by @hidehisaarai1213 in the 2021 Bird Competition. I also used this model and reached CV:0.789 (f1 Score), Public:0.739, and Private:0.711.
\nThen, in the middle of the competition, I switched to use another model (that I'll explain below.) and improved to CV:0.805(f1 Score), Public:0.7634, Private:0.7210 (best with a single model).
Measures against Domain Shift
\nThere were several models with high CV but low LB. This is probably due to the existence of a domain shift between training data and test data. As a countermeasure, Augmentation, which inserts noise, was effective. In particular, Gaussian Noise and Pink Noise worked effectively.
\nModels trained with these augmentations got relatively low CV scores, but the Public scores improved.
Selecting the Appropriate Threshold
\nIn this competition, since there were no Test Soundscapes (data collected under the same conditions as the test data), the threshold had to be set manually. This appropriate threshold was usually far from the value that was optimal during the Validation phase of the model, so tuning was an important step. Since the Public Score was the only available factor to choose threshold, we had no choice but to overfit the Public Score (despite this, LB did not shake as much as we expected).
As figure shows, the sounddata was converted into a 20-second segmented mel-spectrogram, and then Augmentation (Normalize, Pink Noise) was added.
\nAugmentation (model1: Normalize&PinkNoise, model2: Normalize&GaussianNoise&RandomVolume) was added to the input, and the features were extracted using ResNet101d(model1) and ResNeXt50d_32x4d(model2).
\nThe output is a batch size × channel × frequency × time (four-dimensional).
\nThen, the features are compressed in the frequency and time directions using GeM pooling, and the result was returned to a two-dimensional tensor of batch size × channel.
\nThen fc layer is applied to obtain a 152-dimensional output.
\nI also applied MixUp as augmentation with a probability of 0.25. I believe that the robustness of the model has been improved, albeit only by a small margin.
\nAs a loss function, I used Focal loss based on BCEWithLogitsLoss (https://www.kaggle.com/competitions/rfcx-species-audio-detection/discussion/213075 ).
\nI also used SAM Optimizer based on Adam as the Optimizer (https://github.com/davda54/sam ).
\nBy changing the Optimizer to SAM, the Public Score for the Sound Event Detection model improved by about 0.02 when the experiment was conducted under all conditions except for the Optimizer.
Instead of passing the model-by-model average of the output predictions through the threshold, the logical OR or logical product of the true/false values after passing through the threshold was
\nsubmitted as the final prediction result. By applying an ensemble that takes Logical OR, I obtained an output result with a Private score of 0.7292, from the predictions whose private score are 0.7176 and 0.7008 (I we believe they worked effectively)