\n
![\"https://i.ibb.co/6WYFtKN/Selection-933.png\"](\"https://i.ibb.co/6WYFtKN/Selection-933.png\")
\n
![\"https://i.ibb.co/vXb6zqX/Selection-932.png\"](\"https://i.ibb.co/vXb6zqX/Selection-932.png\")
\nlearnable QT code: https://gist.github.com/hengck23/0dd338bed8aad0e98b7477a6f17f9de7
the secret of resnet34? here are some initial writeup:
\n
\n
\nlearnable QT code: https://gist.github.com/hengck23/0dd338bed8aad0e98b7477a6f17f9de7
#----\n self.register_buffer('freq', freq)\n self.register_buffer('length', length)\n self.register_buffer('cqt_window', cqt_window) #this is not trainable !!!!\n if trainable:\n self.cqt_kernel_real = nn.Parameter(cqt_kernel_real)\n self.cqt_kernel_imag = nn.Parameter(cqt_kernel_imag)\n else:\n self.register_buffer('cqt_kernel_real', cqt_kernel_real)\n self.register_buffer('cqt_kernel_imag', cqt_kernel_imag)\njust by plugging a learnable CQT to resnet34, you will easily get CV in the range of 0.880 and LB in the range of 0.881.
\nthe secret of resnet34? here are some initial writeup:
\n
\n
\nlearnable QT code: https://gist.github.com/hengck23/0dd338bed8aad0e98b7477a6f17f9de7
#----\n self.register_buffer('freq', freq)\n self.register_buffer('length', length)\n self.register_buffer('cqt_window', cqt_window) #this is not trainable !!!!\n if trainable:\n self.cqt_kernel_real = nn.Parameter(cqt_kernel_real)\n self.cqt_kernel_imag = nn.Parameter(cqt_kernel_imag)\n else:\n self.register_buffer('cqt_kernel_real', cqt_kernel_real)\n self.register_buffer('cqt_kernel_imag', cqt_kernel_imag)\njust by plugging a learnable CQT to resnet34, you will easily get CV in the range of 0.880 and LB in the range of 0.881.
\n@hengck23 - Just don't be the second person to accidentally share a top solution too early!
", "rawMarkdown": "@hengck23 - Just don't be the second person to accidentally share a top solution too early!", "votes": 3, "replies": [ { "id": 1528730, "postDate": "2021-09-29T23:14:51.543Z", "content": "i don't have solution in kaggle notebook, so it won't happen
", "rawMarkdown": "i don't have solution in kaggle notebook, so it won't happen", "votes": 2 } ] }, { "id": 1531141, "postDate": "2021-10-01T16:47:31.303Z", "content": "visualisation of what is learned:
\n![\"Selection-985\"](\"https://i.ibb.co/GP9Y0sQ/Selection-985.png\")
\n![\"Selection-986\"](\"https://i.ibb.co/tHptbN6/Selection-986.png\")
\n![\"Selection-987\"](\"https://i.ibb.co/cYdD78Y/Selection-987.png\")
\n\n\n",
"votes": 1
},
{
"id": 1528785,
"postDate": "2021-09-30T00:51:12.803Z",
"content": "for us, using Radam (+lookahead) was enough to get trainable cqt to work. no need to freeze it/pre train first
", "rawMarkdown": "for us, using Radam (+lookahead) was enough to get trainable cqt to work. no need to freeze it/pre train first", "votes": 2, "replies": [ { "id": 1528890, "postDate": "2021-09-30T02:46:52.290Z", "content": "@proletheus
\nthanks for the feedback.
I had compared results with and without pretraining. For me, pre-training works better. I think it also depends on my augmentation and regularisation (e.g. dropout).
\nI use Radam (+lookahead) as well.
", "rawMarkdown": "@proletheus \nthanks for the feedback.\n\nI had compared results with and without pretraining. For me, pre-training works better. I think it also depends on my augmentation and regularisation (e.g. dropout).\n\nI use Radam (+lookahead) as well.", "votes": 1 }, { "id": 1528997, "postDate": "2021-09-30T04:42:02.277Z", "content": "i used it because of your bms kernel;) i noticed that the loss would first decrease (0.7-> .5) then increase again with trainabble cqts, so i also tried freezing it first but it would go back to 0.7 loss within a few batches when using just adam
", "rawMarkdown": "i used it because of your bms kernel;) i noticed that the loss would first decrease (0.7-> .5) then increase again with trainabble cqts, so i also tried freezing it first but it would go back to 0.7 loss within a few batches when using just adam" } ] }, { "id": 1528715, "postDate": "2021-09-29T22:49:27.610Z", "content": "I was really expecting it to go all the way to the zero at first 😀
", "rawMarkdown": "I was really expecting it to go all the way to the zero at first 😀", "votes": 2 }, { "id": 1559899, "postDate": "2021-10-27T08:07:07.940Z", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
I used the similar approach as yours.
\nCWT/CQT is a kind of 1D Conv in nature. But it is only one layer and the kernel is too big. So I just replaced the CWT/CQT with a trainable multiple layers 1D Conv.
\nThe detail is here.
\nhttps://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/275353
thanks for the link.
\nI tried rectangle window. In some cases, hann or gaussian-shaped windows work better. It all has to depend on the value of Q (cycles in your kernel size) and whether you need to localize/restrict frequency response to large or small bandwidth
", "rawMarkdown": "thanks for the link.\n\nI tried rectangle window. In some cases, hann or gaussian-shaped windows work better. It all has to depend on the value of Q (cycles in your kernel size) and whether you need to localize/restrict frequency response to large or small bandwidth" } ] }, { "id": 1529044, "postDate": "2021-09-30T05:39:40.733Z", "content": "whitening is only useful if you explore the cross correlation between the 3 signals. I tried many different correlation methods, but got no luck to make it work with this low SNR dataset
", "rawMarkdown": "whitening is only useful if you explore the cross correlation between the 3 signals. I tried many different correlation methods, but got no luck to make it work with this low SNR dataset" }, { "id": 1528880, "postDate": "2021-09-30T02:38:14.483Z", "content": "Ha somehow my ResNet34 is trained the same way…but is no where as good. I guess the devil is really in the details.
\n![\"Imgur\"](\"https://i.imgur.com/AnD86Ji.png\")
@scaomath if you are making post-competition submission, how about starting a slack team and send me an invite email.
\ni will send you my code to repeat results and take a look at your results. Maybe I can offer advice on how to debug your algorithm.
\nafter working on this for many weeks (and falling many attempts), maybe I can pinpoint what goes wrong.
", "rawMarkdown": "@scaomath if you are making post-competition submission, how about starting a slack team and send me an invite email.\n\ni will send you my code to repeat results and take a look at your results. Maybe I can offer advice on how to debug your algorithm.\n\nafter working on this for many weeks (and falling many attempts), maybe I can pinpoint what goes wrong.\n", "votes": 2 }, { "id": 1530006, "postDate": "2021-09-30T20:58:45.403Z", "content": "I just created a post-competition Slack channel and have invited you. Greatly appreciated.
", "rawMarkdown": "I just created a post-competition Slack channel and have invited you. Greatly appreciated." } ] }, { "id": 1528806, "postDate": "2021-09-30T01:13:32.210Z", "content": "I may see what you mean 1dcnn_resnet34?
", "rawMarkdown": "I may see what you mean 1dcnn_resnet34?" }, { "id": 1528796, "postDate": "2021-09-30T01:06:55.690Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1528947, "postDate": "2021-09-30T03:50:23.723Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 1528783, "postDate": "2021-09-30T00:49:22.063Z", "content": "Great Points! Thank you for sharing !!!
", "rawMarkdown": "Great Points! Thank you for sharing !!!" } ], "comments": [ { "id": 1528723, "author_name": "John Mitchell", "author_url": "", "post_date": "2021-09-29T23:02:09.960000", "content": "@hengck23 - Just don't be the second person to accidentally share a top solution too early!
", "votes": 3, "replies": [ { "id": 1528730, "author_name": "hengck23", "author_url": "", "post_date": "2021-09-29T23:14:51.543000", "content": "i don't have solution in kaggle notebook, so it won't happen
", "votes": 2, "replies": [] } ] }, { "id": 1531141, "author_name": "hengck23", "author_url": "", "post_date": "2021-10-01T16:47:31.303000", "content": "visualisation of what is learned:
\n
\n
\n
for us, using Radam (+lookahead) was enough to get trainable cqt to work. no need to freeze it/pre train first
", "votes": 2, "replies": [ { "id": 1528890, "author_name": "hengck23", "author_url": "", "post_date": "2021-09-30T02:46:52.290000", "content": "@proletheus
\nthanks for the feedback.
I had compared results with and without pretraining. For me, pre-training works better. I think it also depends on my augmentation and regularisation (e.g. dropout).
\nI use Radam (+lookahead) as well.
", "votes": 1, "replies": [] }, { "id": 1528997, "author_name": "Proletheus", "author_url": "", "post_date": "2021-09-30T04:42:02.277000", "content": "i used it because of your bms kernel;) i noticed that the loss would first decrease (0.7-> .5) then increase again with trainabble cqts, so i also tried freezing it first but it would go back to 0.7 loss within a few batches when using just adam
", "votes": 0, "replies": [] } ] }, { "id": 1528715, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2021-09-29T22:49:27.610000", "content": "I was really expecting it to go all the way to the zero at first 😀
", "votes": 2, "replies": [] }, { "id": 1559899, "author_name": "ChristopherZerafa", "author_url": "", "post_date": "2021-10-27T08:07:07.940000", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
I used the similar approach as yours.
\nCWT/CQT is a kind of 1D Conv in nature. But it is only one layer and the kernel is too big. So I just replaced the CWT/CQT with a trainable multiple layers 1D Conv.
\nThe detail is here.
\nhttps://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/275353
thanks for the link.
\nI tried rectangle window. In some cases, hann or gaussian-shaped windows work better. It all has to depend on the value of Q (cycles in your kernel size) and whether you need to localize/restrict frequency response to large or small bandwidth
", "votes": 0, "replies": [] } ] }, { "id": 1529044, "author_name": "jobless physicist", "author_url": "", "post_date": "2021-09-30T05:39:40.733000", "content": "whitening is only useful if you explore the cross correlation between the 3 signals. I tried many different correlation methods, but got no luck to make it work with this low SNR dataset
", "votes": 0, "replies": [] }, { "id": 1528880, "author_name": "Shuhao Cao", "author_url": "", "post_date": "2021-09-30T02:38:14.483000", "content": "Ha somehow my ResNet34 is trained the same way…but is no where as good. I guess the devil is really in the details.
\n
@scaomath if you are making post-competition submission, how about starting a slack team and send me an invite email.
\ni will send you my code to repeat results and take a look at your results. Maybe I can offer advice on how to debug your algorithm.
\nafter working on this for many weeks (and falling many attempts), maybe I can pinpoint what goes wrong.
", "votes": 2, "replies": [] }, { "id": 1530006, "author_name": "Shuhao Cao", "author_url": "", "post_date": "2021-09-30T20:58:45.403000", "content": "I just created a post-competition Slack channel and have invited you. Greatly appreciated.
", "votes": 0, "replies": [] } ] }, { "id": 1528806, "author_name": "zhangeng", "author_url": "", "post_date": "2021-09-30T01:13:32.210000", "content": "I may see what you mean 1dcnn_resnet34?
", "votes": 0, "replies": [] }, { "id": 1528796, "author_name": "", "author_url": "", "post_date": "2021-09-30T01:06:55.690000", "content": "", "votes": 0, "replies": [] }, { "id": 1528947, "author_name": "YOg19", "author_url": "", "post_date": "2021-09-30T03:50:23.723000", "content": "Thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 1528783, "author_name": "Chansey Zheng", "author_url": "", "post_date": "2021-09-30T00:49:22.063000", "content": "Great Points! Thank you for sharing !!!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1528710": " \n\nthe secret of resnet34? here are some initial writeup:\n\n\n\n\nlearnable QT code: https://gist.github.com/hengck23/0dd338bed8aad0e98b7477a6f17f9de7\n\n\n```\n\n #----\n self.register_buffer('freq', freq)\n self.register_buffer('length', length)\n self.register_buffer('cqt_window', cqt_window) #this is not trainable !!!!\n\n if trainable:\n self.cqt_kernel_real = nn.Parameter(cqt_kernel_real)\n self.cqt_kernel_imag = nn.Parameter(cqt_kernel_imag)\n else:\n self.register_buffer('cqt_kernel_real', cqt_kernel_real)\n self.register_buffer('cqt_kernel_imag', cqt_kernel_imag)\n\n```\n\n\n\n\njust by plugging a learnable CQT to resnet34, you will easily get CV in the range of 0.880 and LB in the range of 0.881.\n\n- some important points when training:\n\n(1) pre-trained with initial cqt kernels first. set trainable=False\n(2) after one epoch, you can set trainable=True\n(3) for TTA to work, use with shifted-versions of the sample wave as train augmentation\n(4) bandpass is a necessity. Tapering is good to have.\n\n\n\n\n- I cannot get whitening to work, so my solution don't use whitening. But I think it may lead to better solution?\n\n \n- Lastly, I am grateful to Z by HP & NVIDIA for providing a powerful workstation HP Z8 with two Quadro RTX 8000 (48 GB VRAM each) for this competition. For resnet34 and learnable CQT of size 128x256, it takes only** 10 min for one epoch**. This lets me perform many experiments in very short time. With large memory, I can try efficient net b7 with large image size. And for training transformers, I can disable torch.amp (to avoid NAN in attention computation) too.\n", "1528723": "@hengck23 - Just don't be the second person to accidentally share a top solution too early!", "1531141": "visualisation of what is learned:\n\n\n\n\n",
"1528785": "for us, using Radam (+lookahead) was enough to get trainable cqt to work. no need to freeze it/pre train first",
"1528715": "I was really expecting it to go all the way to the zero at first 😀",
"1559899": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris",
"1529641": "I used the similar approach as yours.\nCWT/CQT is a kind of 1D Conv in nature. But it is only one layer and the kernel is too big. So I just replaced the CWT/CQT with a trainable multiple layers 1D Conv. \nThe detail is here.\nhttps://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/275353",
"1529044": "whitening is only useful if you explore the cross correlation between the 3 signals. I tried many different correlation methods, but got no luck to make it work with this low SNR dataset",
"1528880": "Ha somehow my ResNet34 is trained the same way...but is no where as good. I guess the devil is really in the details.\n\n",
"1528806": "I may see what you mean 1dcnn_resnet34?",
"1528796": "",
"1528947": "Thanks for sharing!",
"1528783": "Great Points! Thank you for sharing !!!"
}
}