I am sorry for asking very basic question, while others are publishing great solutions now.
\nDuring this competition, I sometimes tried to train SED or custom pooling (eg. freq axis only) models for about 30 epochs on ColabPro. However, all of them ended up just all zero f1 score. Did anyone have similar result ?
Because of this, I used normal CNN models. By using some tricks (rating-based weight loss, background noise from train_soundscape, fine-tuning from non-background noise models) I got bronze model, but I do want to know the key to train SED model for my future work🙏
", "messages": [ { "id": "1335082", "postDate": "06/04/2021 02:36:07", "content": "I am sorry for asking very basic question, while others are publishing great solutions now.
\nDuring this competition, I sometimes tried to train SED or custom pooling (eg. freq axis only) models for about 30 epochs on ColabPro. However, all of them ended up just all zero f1 score. Did anyone have similar result ?
Because of this, I used normal CNN models. By using some tricks (rating-based weight loss, background noise from train_soundscape, fine-tuning from non-background noise models) I got bronze model, but I do want to know the key to train SED model for my future work🙏
", "rawMarkdown": "I am sorry for asking very basic question, while others are publishing great solutions now.\nDuring this competition, I sometimes tried to train SED or custom pooling (eg. freq axis only) models for about 30 epochs on ColabPro. However, all of them ended up just all zero f1 score. Did anyone have similar result ?\n\nBecause of this, I used normal CNN models. By using some tricks (rating-based weight loss, background noise from train_soundscape, fine-tuning from non-background noise models) I got bronze model, but I do want to know the key to train SED model for my future work🙏", "votes": null }, { "id": "1335113", "postDate": "06/04/2021 02:59:00", "content": "In my experiments, the SED model was sensitive to learning rate and batch size.
\nAlso, some architectures (rexnet150, skresnet-50, etc.) produced only nobird (probably due to lack of tuning) in train_soundscape despite good CV on train_short_audio.
@tattaka Thank you for kind reply 🙌
\nI learned a lot from your 4th place solution as well.
\nI finally got it and will try to re-train my model after changing some hyper parameters.
Before you start building your network architecture, first thing you need to do is to verify your input data into the network if an input (x) corresponds to a label (y). In case of dense prediction, make sure that the ground-truth label (y) is properly encoded to label indexes (or one-hot encoding). If not, the training won’t work.
\nThen, if your dataset in your problem domain is similar to ImageNet dataset, use a pre-trained model on this dataset. The most widely used pre-trained models are VGG net, ResNet, DenseNet or Xception etc. There are many layer architectures, for instance, VGG (19 and 16 layers), ResNet (152, 101, 50 layers or less), DenseNet (201, 169 and 121 layers). Note: Do not try searching hyper-parameters by using more layers nets (e.g. VGG-19, ResNet-152 or DenseNet-201 layers net because it is computationally expensive), use less layers nets instead (e.g. VGG-16, ResNet-50 or DenseNet-121 layers). Pick one pre-trained model that you think it gives the best performance with your hyper-parameters (say ResNet-50 layers). After you obtained the optimal hyper parameters, just select the same but more layers net (say ResNet-101 or ResNet-152 layers) to increase the accuracy.
\nAlways use normalization layers in your network. InstanceNormalization gives slightly performance improvements if they use a small batch-size. Or you may also try GroupNormalization.
\nUse SpatialDropout after a features concatenation if you have two or more convolution layers (say Li) operate on the same input (say F). Always shuffle your training data, both before training and during training, in case you don’t take benefit from temporal data. This may help improving your network performance. Choose a right optimizer. There are many popular adaptive optimizers such as Adam, Adagrad, Adadelta, or RMSprop etc. SGD+momentum is widely used in various problem domains.
\nRead More here: Custom pooling model
In my experiments, the SED model was sensitive to learning rate and batch size.
\nAlso, some architectures (rexnet150, skresnet-50, etc.) produced only nobird (probably due to lack of tuning) in train_soundscape despite good CV on train_short_audio.
@tattaka Thank you for kind reply 🙌
\nI learned a lot from your 4th place solution as well.
\nI finally got it and will try to re-train my model after changing some hyper parameters.
Before you start building your network architecture, first thing you need to do is to verify your input data into the network if an input (x) corresponds to a label (y). In case of dense prediction, make sure that the ground-truth label (y) is properly encoded to label indexes (or one-hot encoding). If not, the training won’t work.
\nThen, if your dataset in your problem domain is similar to ImageNet dataset, use a pre-trained model on this dataset. The most widely used pre-trained models are VGG net, ResNet, DenseNet or Xception etc. There are many layer architectures, for instance, VGG (19 and 16 layers), ResNet (152, 101, 50 layers or less), DenseNet (201, 169 and 121 layers). Note: Do not try searching hyper-parameters by using more layers nets (e.g. VGG-19, ResNet-152 or DenseNet-201 layers net because it is computationally expensive), use less layers nets instead (e.g. VGG-16, ResNet-50 or DenseNet-121 layers). Pick one pre-trained model that you think it gives the best performance with your hyper-parameters (say ResNet-50 layers). After you obtained the optimal hyper parameters, just select the same but more layers net (say ResNet-101 or ResNet-152 layers) to increase the accuracy.
\nAlways use normalization layers in your network. InstanceNormalization gives slightly performance improvements if they use a small batch-size. Or you may also try GroupNormalization.
\nUse SpatialDropout after a features concatenation if you have two or more convolution layers (say Li) operate on the same input (say F). Always shuffle your training data, both before training and during training, in case you don’t take benefit from temporal data. This may help improving your network performance. Choose a right optimizer. There are many popular adaptive optimizers such as Adam, Adagrad, Adadelta, or RMSprop etc. SGD+momentum is widely used in various problem domains.
\nRead More here: Custom pooling model