At first, I would like to thank my team members @vladimirsydor, @zekamrozek, @yakuben and @uulott for their work and organizers and the Kaggle community for the amazing experience we had during this competition.
\nBlend of 6 2D CNN + 3 1D CNN: 1x2D EfficentNet_b0 + 3 x _b3 + 2 x _b5 + 2 x 1D ResNet with transformer + 1D basic CNN
\nBandpass was crucial for this data, we experimented with it a lot and tried different setups, those parameters showed better results then others we tried:
\n fmin=20,
\n fmax=1024 or fmax=500 or fmax=600 depending on the model,
\n hop_length=4,
\n bins_per_octave=12,
\n filter_scale=0.5,
\n pad=10.
For images, we used only CQT and didn't experiment with CWT.
\nAugmentations that worked for us:
\nnew_wave = (wave1 + wave2) / 2 and new_target = max(target1, target2);Models
\nFor most of the competition, we were training EfficentNets, until around 1-2 last weeks we started experimenting with 1D models.
We found out, that increasing the input image would increase the performance even for EfficentNet_B0, so we experimented with it more. For us best result was when we used stride=(1,2) instead of default (2,2) with an unchanged shape of input. That allowed model to \"stretch\" the input itself across the frequency axis and give an effect of feeding higher resolution images to the network.
Our 2D models suffered from spoiled gradients in BatchNorm layers on first epochs, so to overcome that we used 2 procedures. The first idea that worked was to substitute BatchNorm for InstanceNorm throughout the whole model. 2nd idea that worked was that we would freeze all encoder layers but BatchNorm, and train only them plus Linear head for the first 2 epochs.
\nAs was mentioned, we started working late on 1D models, mainly simple stacking Conv layers, but after merging with @uulott, he brought a really promising 1D ResNet model that performed on similar level as our best 2D ones (~0.879 LB).
\nFinally, all of our models benefited when we started finetuning them on a small learning rate for multiple epochs with augmentations turned off. That boosted us past 0.88 LB and was keeping us in silver as we kept going up LB the more models we fine-tune and blend with.
\nEnsemble
\nFor ensembling, we used straightforward hyperopt blending. Adding Any 1D model boosted all of our blends.
The final blend consisted of 6 2D-CNNs and 3 1D-CNNs:
\nEfficentNet-B5, successful augs, freeze 2 epochs, normalize after CQT;
Same B5, but fine-tuned;
fine-tuned EfficentNet-B3, all successful augs, freeze 2 epochs, normalize after CQT;
fine-tuned EfficentNet-B3, initial stride=(1,2) GaussNoise aug, freeze 2 epochs, normalize after CQT;
fine-tuned EfficentNet-B3, initial stride=(1,2) GaussNoise aug, freeze 2 epochs, normalize after CQT;
fine-tuned EfficentNet-B0 , InstanceNorm instead of BatchNorm, normalize after CQT;
1D ResNet + Transformer head, filter order 8 and CosineAnnealing;
1D ResNet + Transformer head, filter order 5 and ReduceLROnPlateau;
fine-tuned 6-layer CNN1d with MLP head, MixUp and GaussNoise, linear warm-up and cosine decay.
P.S. Other things we tried, but it didn't make it to the final blend:
\nAugmentations that we stopped using :
\nAt first, I would like to thank my team members @vladimirsydor, @zekamrozek, @yakuben and @uulott for their work and organizers and the Kaggle community for the amazing experience we had during this competition.
\nBlend of 6 2D CNN + 3 1D CNN: 1x2D EfficentNet_b0 + 3 x _b3 + 2 x _b5 + 2 x 1D ResNet with transformer + 1D basic CNN
\nBandpass was crucial for this data, we experimented with it a lot and tried different setups, those parameters showed better results then others we tried:
\n fmin=20,
\n fmax=1024 or fmax=500 or fmax=600 depending on the model,
\n hop_length=4,
\n bins_per_octave=12,
\n filter_scale=0.5,
\n pad=10.
For images, we used only CQT and didn't experiment with CWT.
\nAugmentations that worked for us:
\nnew_wave = (wave1 + wave2) / 2 and new_target = max(target1, target2);Models
\nFor most of the competition, we were training EfficentNets, until around 1-2 last weeks we started experimenting with 1D models.
We found out, that increasing the input image would increase the performance even for EfficentNet_B0, so we experimented with it more. For us best result was when we used stride=(1,2) instead of default (2,2) with an unchanged shape of input. That allowed model to \"stretch\" the input itself across the frequency axis and give an effect of feeding higher resolution images to the network.
Our 2D models suffered from spoiled gradients in BatchNorm layers on first epochs, so to overcome that we used 2 procedures. The first idea that worked was to substitute BatchNorm for InstanceNorm throughout the whole model. 2nd idea that worked was that we would freeze all encoder layers but BatchNorm, and train only them plus Linear head for the first 2 epochs.
\nAs was mentioned, we started working late on 1D models, mainly simple stacking Conv layers, but after merging with @uulott, he brought a really promising 1D ResNet model that performed on similar level as our best 2D ones (~0.879 LB).
\nFinally, all of our models benefited when we started finetuning them on a small learning rate for multiple epochs with augmentations turned off. That boosted us past 0.88 LB and was keeping us in silver as we kept going up LB the more models we fine-tune and blend with.
\nEnsemble
\nFor ensembling, we used straightforward hyperopt blending. Adding Any 1D model boosted all of our blends.
The final blend consisted of 6 2D-CNNs and 3 1D-CNNs:
\nEfficentNet-B5, successful augs, freeze 2 epochs, normalize after CQT;
Same B5, but fine-tuned;
fine-tuned EfficentNet-B3, all successful augs, freeze 2 epochs, normalize after CQT;
fine-tuned EfficentNet-B3, initial stride=(1,2) GaussNoise aug, freeze 2 epochs, normalize after CQT;
fine-tuned EfficentNet-B3, initial stride=(1,2) GaussNoise aug, freeze 2 epochs, normalize after CQT;
fine-tuned EfficentNet-B0 , InstanceNorm instead of BatchNorm, normalize after CQT;
1D ResNet + Transformer head, filter order 8 and CosineAnnealing;
1D ResNet + Transformer head, filter order 5 and ReduceLROnPlateau;
fine-tuned 6-layer CNN1d with MLP head, MixUp and GaussNoise, linear warm-up and cosine decay.
P.S. Other things we tried, but it didn't make it to the final blend:
\nAugmentations that we stopped using :
\nFun to know that low learning rate worked for you… I got stuck for 2 weeks @ CV 8750 only because I was using lr < 1e-4. After trying lower epochs and lr 1e-3 I got a big jump to 8795.
", "rawMarkdown": "Fun to know that low learning rate worked for you… I got stuck for 2 weeks @ CV 8750 only because I was using lr < 1e-4. After trying lower epochs and lr 1e-3 I got a big jump to 8795.", "votes": null }, { "id": "1559900", "postDate": "10/27/2021 08:07:12", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
Fun to know that low learning rate worked for you… I got stuck for 2 weeks @ CV 8750 only because I was using lr < 1e-4. After trying lower epochs and lr 1e-3 I got a big jump to 8795.
", "votes": null, "replies": [] }, { "id": 1559900, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 08:07:12", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris