![\"mask\"](\"https://i.imgur.com/SXWVA5I.png\")
I want to thank Kaggle and Host for this amazing challenge. Topic was great, and teaming with @onodera and @titericz was great too.
\nWe led the competition for a while then all of a sudden teams started to pass us. In hindsight we know it is thanks to using 1D models and/or resnets. Unfortunately for us we kept working with efficientnet v2 as we were having good results so far. We have no excuse given both ideas were shared in the forum soon enough to be leveraged. To be honest we tried resnet34 but it did not match effnetv2 results for us. I wonder why it is different for other teams. We also tried 1D models briefly, but, same, they were low quality compared to effnets. We should have submitted a blend obviously.
\nHere is a snapshot of our solution.
\nData processing
\nData was multiplied by 1e19 (1e21 later) to ensure that CQT and CWT could be performed correctly using FP16.
\nCQT and CWT (results are very similar). Best model (public 0.8826 private 0.8805) used these setting for CQT:
\nsr = 2048\nhop_length = 5\nfmin = 22\nfmax = 22*16\nbins_per_octave = 8\nn_octaves = 4\nn_bins = n_octaves * bins_per_octave\nfscale = 1\ncqt = CQT1992v2(sr=sr, hop_length=hop_length, fmin=fmin, fmax=fmax,\n n_bins=n_bins, bins_per_octave=bins_per_octave * fscale, \n window=('kaiser', 14), filter_scale=1/fscale,\n )\nWe used fscale = 2 (i.e. filter_scale = 0.5) in few models, it improves a bit but we could not train them as much as our best model. We also varied hop length to get different image sizes. We then concatenated the 3 CQT images on the frequency dim, getting images of size 96xL where L depends on hop length (L = 820 for the above setting).
\nSignal To Noise EDA
\nWe found the best frequency range by doing a signal ratio analysis. We computed the average of CQT images for positive samples (POS) , and divided it by the average of CQT images for negative samples (NEG). This gives us what we called a mask. Here is the mask for above settings
\n
We see where the waves are on average.
\nBy taking mask average over detectors and time we see that we can crop images left and right safely (y axis labels should be multiplied by 3, sorry):
\n![\"time\"](\"https://i.imgur.com/BuBZGJo.png\")
Cropping sides gets rid of the CQT border artifacts.
\nBy taking the mask mean over detectors and time we get signal to noise ratio per frequency (y axis labels should be multiplied by 3, sorry):
\n![\"freq\"](\"https://i.imgur.com/QTTfca3.png\")
x is log scale, but doing the math we see that snr is 1 outside 22Hz-352Hz
\nWe tuned bins per octave so that the image height is a multiple of 16 (for speed), and 32 bins overall was a very good tradeoff.
\nInput images
\nWe used nnAudio CQT (or CWT) as the first model layer. Then 3 outputs are stacked on frequency axis. We then divided the image by NEG (average of negative sample images) to get rid of noise on average. This was quite better than whitening. We finally applied a log scaling, a bit similar to amplitude to db in audio signal processing. An example of resulting image is shown below (ignore frequency axis labels):
\n![\"image\"](\"https://i.imgur.com/ERThCPY.png\")
Depending on the model we added the mask as an additional channel. We then scaled data to have multiple of 16 dimensions. For instance resizing 96x820 to 1x96x768
\nModel
\nWe mostly used efficientnet_v2s_s and efficientnet_v2_m from timm package. Models were trained using pytorch.cuda.amp. By using batch size and image dimensions that are multiple of 16 we use of RT cores and speed up training significantly on GPU V100.
\nWe used BCEWithLogitsLoss, Adam and OneCycleLR most of the competition. We later found that retraining models was helping. Our best model went through 4 training cycles. Using cosine annealing from the start would probably have been a good choice.
\nAugmentations
\nKazuki shared most of our augmentations here: https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/275335
\nWe also used a random time shift in addition to these.
\nPseudo Labeling
\nPseudo labeling gave us a 0.0002 boost on LB when we used it. We used non rounded predictions from teacher model for test dataset and concatenated to training fold. It improved CV by 0.0008 on average but LB by only 0.0002. However, our best model was not a pseudo labeling model.
\nEnsembling
\nWe kept oof predictions for all our models and trained second level models on them. We used logistic regression on logits, scipy optimize on ranked predictions, and XGBoost on ranked predictions. They yield similar results, and averaging them did not help really.
\nEnsembling moved us from 8826 for best single model to 8832 on public LB.
\nWhat did not work
\nThat's it. We clearly missed 1D model, but for the rest we did everything we could. And I learned from my team mates! Teaming is good!.
", "messages": [ { "id": "1529466", "postDate": "09/30/2021 12:34:46", "content": "I want to thank Kaggle and Host for this amazing challenge. Topic was great, and teaming with @onodera and @titericz was great too.
\nWe led the competition for a while then all of a sudden teams started to pass us. In hindsight we know it is thanks to using 1D models and/or resnets. Unfortunately for us we kept working with efficientnet v2 as we were having good results so far. We have no excuse given both ideas were shared in the forum soon enough to be leveraged. To be honest we tried resnet34 but it did not match effnetv2 results for us. I wonder why it is different for other teams. We also tried 1D models briefly, but, same, they were low quality compared to effnets. We should have submitted a blend obviously.
\nHere is a snapshot of our solution.
\nData processing
\nData was multiplied by 1e19 (1e21 later) to ensure that CQT and CWT could be performed correctly using FP16.
\nCQT and CWT (results are very similar). Best model (public 0.8826 private 0.8805) used these setting for CQT:
\nsr = 2048\nhop_length = 5\nfmin = 22\nfmax = 22*16\nbins_per_octave = 8\nn_octaves = 4\nn_bins = n_octaves * bins_per_octave\nfscale = 1\ncqt = CQT1992v2(sr=sr, hop_length=hop_length, fmin=fmin, fmax=fmax,\n n_bins=n_bins, bins_per_octave=bins_per_octave * fscale, \n window=('kaiser', 14), filter_scale=1/fscale,\n )\nWe used fscale = 2 (i.e. filter_scale = 0.5) in few models, it improves a bit but we could not train them as much as our best model. We also varied hop length to get different image sizes. We then concatenated the 3 CQT images on the frequency dim, getting images of size 96xL where L depends on hop length (L = 820 for the above setting).
\nSignal To Noise EDA
\nWe found the best frequency range by doing a signal ratio analysis. We computed the average of CQT images for positive samples (POS) , and divided it by the average of CQT images for negative samples (NEG). This gives us what we called a mask. Here is the mask for above settings
\n
We see where the waves are on average.
\nBy taking mask average over detectors and time we see that we can crop images left and right safely (y axis labels should be multiplied by 3, sorry):
\n
Cropping sides gets rid of the CQT border artifacts.
\nBy taking the mask mean over detectors and time we get signal to noise ratio per frequency (y axis labels should be multiplied by 3, sorry):
\n
x is log scale, but doing the math we see that snr is 1 outside 22Hz-352Hz
\nWe tuned bins per octave so that the image height is a multiple of 16 (for speed), and 32 bins overall was a very good tradeoff.
\nInput images
\nWe used nnAudio CQT (or CWT) as the first model layer. Then 3 outputs are stacked on frequency axis. We then divided the image by NEG (average of negative sample images) to get rid of noise on average. This was quite better than whitening. We finally applied a log scaling, a bit similar to amplitude to db in audio signal processing. An example of resulting image is shown below (ignore frequency axis labels):
\n
Depending on the model we added the mask as an additional channel. We then scaled data to have multiple of 16 dimensions. For instance resizing 96x820 to 1x96x768
\nModel
\nWe mostly used efficientnet_v2s_s and efficientnet_v2_m from timm package. Models were trained using pytorch.cuda.amp. By using batch size and image dimensions that are multiple of 16 we use of RT cores and speed up training significantly on GPU V100.
\nWe used BCEWithLogitsLoss, Adam and OneCycleLR most of the competition. We later found that retraining models was helping. Our best model went through 4 training cycles. Using cosine annealing from the start would probably have been a good choice.
\nAugmentations
\nKazuki shared most of our augmentations here: https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/275335
\nWe also used a random time shift in addition to these.
\nPseudo Labeling
\nPseudo labeling gave us a 0.0002 boost on LB when we used it. We used non rounded predictions from teacher model for test dataset and concatenated to training fold. It improved CV by 0.0008 on average but LB by only 0.0002. However, our best model was not a pseudo labeling model.
\nEnsembling
\nWe kept oof predictions for all our models and trained second level models on them. We used logistic regression on logits, scipy optimize on ranked predictions, and XGBoost on ranked predictions. They yield similar results, and averaging them did not help really.
\nEnsembling moved us from 8826 for best single model to 8832 on public LB.
\nWhat did not work
\nThat's it. We clearly missed 1D model, but for the rest we did everything we could. And I learned from my team mates! Teaming is good!.
", "rawMarkdown": "I want to thank Kaggle and Host for this amazing challenge. Topic was great, and teaming with @onodera and @titericz was great too.\n\nWe led the competition for a while then all of a sudden teams started to pass us. In hindsight we know it is thanks to using 1D models and/or resnets. Unfortunately for us we kept working with efficientnet v2 as we were having good results so far. We have no excuse given both ideas were shared in the forum soon enough to be leveraged. To be honest we tried resnet34 but it did not match effnetv2 results for us. I wonder why it is different for other teams. We also tried 1D models briefly, but, same, they were low quality compared to effnets. We should have submitted a blend obviously. \n\nHere is a snapshot of our solution.\n\n**Data processing**\n\nData was multiplied by 1e19 (1e21 later) to ensure that CQT and CWT could be performed correctly using FP16.\n\nCQT and CWT (results are very similar). Best model (public 0.8826 private 0.8805) used these setting for CQT:\n\n```\nsr = 2048\nhop_length = 5\nfmin = 22\nfmax = 22*16\nbins_per_octave = 8\nn_octaves = 4\nn_bins = n_octaves * bins_per_octave\nfscale = 1\ncqt = CQT1992v2(sr=sr, hop_length=hop_length, fmin=fmin, fmax=fmax,\n n_bins=n_bins, bins_per_octave=bins_per_octave * fscale, \n window=('kaiser', 14), filter_scale=1/fscale,\n )\n```\n\nWe used fscale = 2 (i.e. filter_scale = 0.5) in few models, it improves a bit but we could not train them as much as our best model. We also varied hop length to get different image sizes. We then concatenated the 3 CQT images on the frequency dim, getting images of size 96xL where L depends on hop length (L = 820 for the above setting). \n\n**Signal To Noise EDA**\n\nWe found the best frequency range by doing a signal ratio analysis. We computed the average of CQT images for positive samples (POS) , and divided it by the average of CQT images for negative samples (NEG). This gives us what we called a mask. Here is the mask for above settings\n\n\n\nWe see where the waves are on average.\n\nBy taking mask average over detectors and time we see that we can crop images left and right safely (y axis labels should be multiplied by 3, sorry):\n\n \n\nCropping sides gets rid of the CQT border artifacts.\n\nBy taking the mask mean over detectors and time we get signal to noise ratio per frequency (y axis labels should be multiplied by 3, sorry):\n\n\n\nx is log scale, but doing the math we see that snr is 1 outside 22Hz-352Hz\n\nWe tuned bins per octave so that the image height is a multiple of 16 (for speed), and 32 bins overall was a very good tradeoff.\n\n**Input images**\n\nWe used nnAudio CQT (or CWT) as the first model layer. Then 3 outputs are stacked on frequency axis. We then divided the image by NEG (average of negative sample images) to get rid of noise on average. This was quite better than whitening. We finally applied a log scaling, a bit similar to amplitude to db in audio signal processing. An example of resulting image is shown below (ignore frequency axis labels):\n\n\n\nDepending on the model we added the mask as an additional channel. We then scaled data to have multiple of 16 dimensions. For instance resizing 96x820 to 1x96x768\n\n**Model**\n\nWe mostly used efficientnet_v2s_s and efficientnet_v2_m from timm package. Models were trained using pytorch.cuda.amp. By using batch size and image dimensions that are multiple of 16 we use of RT cores and speed up training significantly on GPU V100.\n\nWe used BCEWithLogitsLoss, Adam and OneCycleLR most of the competition. We later found that retraining models was helping. Our best model went through 4 training cycles. Using cosine annealing from the start would probably have been a good choice.\n\n**Augmentations**\n\nKazuki shared most of our augmentations here: https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/275335\n\nWe also used a random time shift in addition to these.\n\n**Pseudo Labeling**\n\nPseudo labeling gave us a 0.0002 boost on LB when we used it. We used non rounded predictions from teacher model for test dataset and concatenated to training fold. It improved CV by 0.0008 on average but LB by only 0.0002. However, our best model was not a pseudo labeling model.\n\n**Ensembling**\n\nWe kept oof predictions for all our models and trained second level models on them. We used logistic regression on logits, scipy optimize on ranked predictions, and XGBoost on ranked predictions. They yield similar results, and averaging them did not help really. \n\nEnsembling moved us from 8826 for best single model to 8832 on public LB.\n\n**What did not work**\n\n- Giba tried hard to generate additional positive samples. Issue was to calibrate these to be of the same distribution as training data. We did not find the right way in time.\n- Resnet34. As written above, we could not meet effnetv2s performance. And speedup was only 2x anyway.\n- We tried many other things that did not work well enough, like triplet models (a backbone on each detector then a common head), or using detectors as RGB channels.\n\nThat's it. We clearly missed 1D model, but for the rest we did everything we could. And I learned from my team mates! Teaming is good!.", "votes": null }, { "id": "1529482", "postDate": "09/30/2021 12:45:39", "content": "\n\nWe found the best frequency range by doing a signal ratio analysis. We computed the average of CQT images for positive samples (POS) , and divided it by the average of CQT images for negative samples (NEG). This gives us what we called a mask. Here is the mask for above settings
\n
So simple and genius that it's frustrating it did not come to me to attempt this.
\n\n\nWe then divided the image by NEG (average of negative sample images) to get rid of noise on average. This was quite better than whitening.
\n
omg nnnrrrrrrrrggh!
\nAs always, an elegant solution, well deserved.
\nPS:
\n\n\nWe kept oof predictions for all our models and trained second level models on them. We used logistic regression on logits, scipy optimize on ranked predictions, and XGBoost on ranked predictions. They yield similar results, and averaging them did not help really.
\n
@cpmpml if you haven't deleted your preds, might I convince you to humor me and attempt to run SGD for your second level stacker? Interested in what the CV / late private LB of that would be…
", "rawMarkdown": "> We found the best frequency range by doing a signal ratio analysis. We computed the average of CQT images for positive samples (POS) , and divided it by the average of CQT images for negative samples (NEG). This gives us what we called a mask. Here is the mask for above settings\n\nSo simple and genius that it's frustrating it did not come to me to attempt this.\n\n> We then divided the image by NEG (average of negative sample images) to get rid of noise on average. This was quite better than whitening.\n\nomg nnnrrrrrrrrggh!\n\nAs always, an elegant solution, well deserved.\n\nPS:\n\n> We kept oof predictions for all our models and trained second level models on them. We used logistic regression on logits, scipy optimize on ranked predictions, and XGBoost on ranked predictions. They yield similar results, and averaging them did not help really. \n\n@cpmpml if you haven't deleted your preds, might I convince you to humor me and attempt to run SGD for your second level stacker? Interested in what the CV / late private LB of that would be...", "votes": null }, { "id": "1529490", "postDate": "09/30/2021 12:52:50", "content": "Thanks! We thought that signal processing was they, but we totally missed the 1D avenue still.
", "rawMarkdown": "Thanks! We thought that signal processing was they, but we totally missed the 1D avenue still.", "votes": null }, { "id": "1529515", "postDate": "09/30/2021 13:09:25", "content": "\"To be honest we tried resnet34 but it did not match effnetv2 results for us. I wonder why it is different for other teams. \"
\nResnet34 is not a strong classifier. the performance actually comes from the learnable front-end for my case. If the backend is not strong, the model is forced to learn a better front end. just some experiment results from my side:
\nfixed cqt + resnet34 : cv 0.872
\nsame fixed cqt + effb2 : cv 0.874
\nsame fixed cqt + effb7 : cv 0.876
learnable cqt + resnet34 : cv 0.878
\nlearnable cqt + effb2 : cv 0.877
you can consider fixed cqt as non optimised cqt in some sense
\nI am surprised that the mask cqt images you showed have the GW waves extending beyond 1000Hz
", "rawMarkdown": "\"To be honest we tried resnet34 but it did not match effnetv2 results for us. I wonder why it is different for other teams. \"\n\nResnet34 is not a strong classifier. the performance actually comes from the learnable front-end for my case. If the backend is not strong, the model is forced to learn a better front end. just some experiment results from my side:\n\nfixed cqt + resnet34 : cv 0.872 \nsame fixed cqt + effb2 : cv 0.874\nsame fixed cqt + effb7 : cv 0.876\n\nlearnable cqt + resnet34 : cv 0.878\nlearnable cqt + effb2 : cv 0.877\n\nyou can consider fixed cqt as non optimised cqt in some sense\n\nI am surprised that the mask cqt images you showed have the GW waves extending beyond 1000Hz", "votes": null }, { "id": "1529527", "postDate": "09/30/2021 13:15:18", "content": "The frequency labels are not right because CQT output is on a log scale. The mask does not extend above 350 Hz as shown by the snr curve.
", "rawMarkdown": "The frequency labels are not right because CQT output is on a log scale. The mask does not extend above 350 Hz as shown by the snr curve.", "votes": null }, { "id": "1529536", "postDate": "09/30/2021 13:20:25", "content": "\"We then divided the image by NEG (average of negative sample images) to get rid of noise on average. This was quite better than whitening.\"
\njust an idea … use some different sets(random set or clustering …. even from the test set) of NEG template as TTA (and training)?
", "rawMarkdown": "\"We then divided the image by NEG (average of negative sample images) to get rid of noise on average. This was quite better than whitening.\"\n\njust an idea ... use some different sets(random set or clustering .... even from the test set) of NEG template as TTA (and training)?", "votes": null }, { "id": "1529594", "postDate": "09/30/2021 14:03:55", "content": "Neg estimate didn't vary much when subsampling, even 10k samples. Anyway, goal is to improve snr, hence NEG average is the best noise estimate we can get.. But we made NEG and MASK trainable last few days of comp, and this improved models significantly.
\nI wish we had tried trainable CQT too as you did.
", "rawMarkdown": "Neg estimate didn't vary much when subsampling, even 10k samples. Anyway, goal is to improve snr, hence NEG average is the best noise estimate we can get.. But we made NEG and MASK trainable last few days of comp, and this improved models significantly.\n\n\nI wish we had tried trainable CQT too as you did.", "votes": null }, { "id": "1529608", "postDate": "09/30/2021 14:14:00", "content": "Thank you for sharing and congrats! Did you manage to get your STFT Transformer from Birdclef to work for this? I tried the same approach (but replacing STFT with CQT), but couldn't get decent results. Wondering if you tried the same?
", "rawMarkdown": "Thank you for sharing and congrats! Did you manage to get your STFT Transformer from Birdclef to work for this? I tried the same approach (but replacing STFT with CQT), but couldn't get decent results. Wondering if you tried the same?", "votes": null }, { "id": "1529644", "postDate": "09/30/2021 14:53:47", "content": "I tried transformers indeed, slow and not as good as effnets here!
", "rawMarkdown": "I tried transformers indeed, slow and not as good as effnets here!", "votes": null }, { "id": "1529654", "postDate": "09/30/2021 15:02:00", "content": "trainable NEG and MASK, that is a very good idea.
\nbasically, make an \"attention mask\", then make it trainable … I can see potential use in the future👍👍👍
", "rawMarkdown": "trainable NEG and MASK, that is a very good idea.\n\nbasically, make an \"attention mask\", then make it trainable ... I can see potential use in the future👍👍👍", "votes": null }, { "id": "1529705", "postDate": "09/30/2021 15:52:03", "content": "We've also tried, and had the same result.
", "rawMarkdown": "We've also tried, and had the same result.", "votes": null }, { "id": "1531478", "postDate": "10/02/2021 03:28:17", "content": "I also tried it, following your Birdclef report, but without much success( probably next time… It is really a great idea.
", "rawMarkdown": "I also tried it, following your Birdclef report, but without much success( probably next time... It is really a great idea.", "votes": null }, { "id": "1532869", "postDate": "10/03/2021 12:58:09", "content": "I missed that. Which SGD you want me to run?
", "rawMarkdown": "I missed that. Which SGD you want me to run?", "votes": null }, { "id": "1549136", "postDate": "10/18/2021 20:20:49", "content": "Thanks a lot for sharing an for such an understandable explanation. A couple of questions:
\n\n\nWe later found that retraining models was helping. Our best model went through 4 training cycles.
\n
You mean 4 cycles of the scheduler, right? How did you figure out that?
\n\n\nBut we made NEG and MASK trainable last few days of comp, and this improved models significantly.
\n
How did you do that? Considering them as a torch nn.Parameter or something like that?
", "rawMarkdown": "Thanks a lot for sharing an for such an understandable explanation. A couple of questions:\n\n> We later found that retraining models was helping. Our best model went through 4 training cycles.\n\nYou mean 4 cycles of the scheduler, right? How did you figure out that?\n\n> But we made NEG and MASK trainable last few days of comp, and this improved models significantly.\n\nHow did you do that? Considering them as a torch nn.Parameter or something like that?", "votes": null }, { "id": "1559928", "postDate": "10/27/2021 08:24:04", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
\n\nWe found the best frequency range by doing a signal ratio analysis. We computed the average of CQT images for positive samples (POS) , and divided it by the average of CQT images for negative samples (NEG). This gives us what we called a mask. Here is the mask for above settings
\n
So simple and genius that it's frustrating it did not come to me to attempt this.
\n\n\nWe then divided the image by NEG (average of negative sample images) to get rid of noise on average. This was quite better than whitening.
\n
omg nnnrrrrrrrrggh!
\nAs always, an elegant solution, well deserved.
\nPS:
\n\n\nWe kept oof predictions for all our models and trained second level models on them. We used logistic regression on logits, scipy optimize on ranked predictions, and XGBoost on ranked predictions. They yield similar results, and averaging them did not help really.
\n
@cpmpml if you haven't deleted your preds, might I convince you to humor me and attempt to run SGD for your second level stacker? Interested in what the CV / late private LB of that would be…
", "votes": null, "replies": [ { "id": 1529490, "author_name": "cpmpml", "author_url": "", "post_date": "09/30/2021 12:52:50", "content": "Thanks! We thought that signal processing was they, but we totally missed the 1D avenue still.
", "votes": null, "replies": [] }, { "id": 1532869, "author_name": "cpmpml", "author_url": "", "post_date": "10/03/2021 12:58:09", "content": "I missed that. Which SGD you want me to run?
", "votes": null, "replies": [] } ] }, { "id": 1529515, "author_name": "hengck23", "author_url": "", "post_date": "09/30/2021 13:09:25", "content": "\"To be honest we tried resnet34 but it did not match effnetv2 results for us. I wonder why it is different for other teams. \"
\nResnet34 is not a strong classifier. the performance actually comes from the learnable front-end for my case. If the backend is not strong, the model is forced to learn a better front end. just some experiment results from my side:
\nfixed cqt + resnet34 : cv 0.872
\nsame fixed cqt + effb2 : cv 0.874
\nsame fixed cqt + effb7 : cv 0.876
learnable cqt + resnet34 : cv 0.878
\nlearnable cqt + effb2 : cv 0.877
you can consider fixed cqt as non optimised cqt in some sense
\nI am surprised that the mask cqt images you showed have the GW waves extending beyond 1000Hz
", "votes": null, "replies": [ { "id": 1529527, "author_name": "cpmpml", "author_url": "", "post_date": "09/30/2021 13:15:18", "content": "The frequency labels are not right because CQT output is on a log scale. The mask does not extend above 350 Hz as shown by the snr curve.
", "votes": null, "replies": [] } ] }, { "id": 1529536, "author_name": "hengck23", "author_url": "", "post_date": "09/30/2021 13:20:25", "content": "\"We then divided the image by NEG (average of negative sample images) to get rid of noise on average. This was quite better than whitening.\"
\njust an idea … use some different sets(random set or clustering …. even from the test set) of NEG template as TTA (and training)?
", "votes": null, "replies": [ { "id": 1529594, "author_name": "cpmpml", "author_url": "", "post_date": "09/30/2021 14:03:55", "content": "Neg estimate didn't vary much when subsampling, even 10k samples. Anyway, goal is to improve snr, hence NEG average is the best noise estimate we can get.. But we made NEG and MASK trainable last few days of comp, and this improved models significantly.
\nI wish we had tried trainable CQT too as you did.
", "votes": null, "replies": [] }, { "id": 1529654, "author_name": "hengck23", "author_url": "", "post_date": "09/30/2021 15:02:00", "content": "trainable NEG and MASK, that is a very good idea.
\nbasically, make an \"attention mask\", then make it trainable … I can see potential use in the future👍👍👍
", "votes": null, "replies": [] } ] }, { "id": 1529608, "author_name": "anjum48", "author_url": "", "post_date": "09/30/2021 14:14:00", "content": "Thank you for sharing and congrats! Did you manage to get your STFT Transformer from Birdclef to work for this? I tried the same approach (but replacing STFT with CQT), but couldn't get decent results. Wondering if you tried the same?
", "votes": null, "replies": [ { "id": 1529644, "author_name": "cpmpml", "author_url": "", "post_date": "09/30/2021 14:53:47", "content": "I tried transformers indeed, slow and not as good as effnets here!
", "votes": null, "replies": [] }, { "id": 1529705, "author_name": "hidehisaarai1213", "author_url": "", "post_date": "09/30/2021 15:52:03", "content": "We've also tried, and had the same result.
", "votes": null, "replies": [] }, { "id": 1531478, "author_name": "iafoss", "author_url": "", "post_date": "10/02/2021 03:28:17", "content": "I also tried it, following your Birdclef report, but without much success( probably next time… It is really a great idea.
", "votes": null, "replies": [] } ] }, { "id": 1549136, "author_name": "delai50", "author_url": "", "post_date": "10/18/2021 20:20:49", "content": "Thanks a lot for sharing an for such an understandable explanation. A couple of questions:
\n\n\nWe later found that retraining models was helping. Our best model went through 4 training cycles.
\n
You mean 4 cycles of the scheduler, right? How did you figure out that?
\n\n\nBut we made NEG and MASK trainable last few days of comp, and this improved models significantly.
\n
How did you do that? Considering them as a torch nn.Parameter or something like that?
", "votes": null, "replies": [] }, { "id": 1559928, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 08:24:04", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris