![\"https://i.ibb.co/7Cm6Fwy/Selection-839.png\"](\"https://i.ibb.co/7Cm6Fwy/Selection-839.png\")
this is a preview only.
\nmore analysis and results coming up soon!
this is a preview only.
\nmore analysis and results coming up soon!
\n\nThe CQT is not used in the CNN
\n
What's used ?
", "rawMarkdown": "> The CQT is not used in the CNN\n\nWhat's used ?", "votes": null }, { "id": "1509910", "postDate": "09/11/2021 19:17:07", "content": "It's a 1d cnn, you're thinking of 2d cnn. In this case, he probably used the raw time series.
", "rawMarkdown": "It's a 1d cnn, you're thinking of 2d cnn. In this case, he probably used the raw time series.", "votes": null }, { "id": "1509912", "postDate": "09/11/2021 19:19:16", "content": "Oops sorry, missed that part !
", "rawMarkdown": "Oops sorry, missed that part !", "votes": null }, { "id": "1510028", "postDate": "09/12/2021 01:09:36", "content": "update1.
\n(selection of max kernel (red box) could be wrong as I did not consider the weights of the next bn layer)
![\"https://i.ibb.co/n3gQzy8/Selection-851.png\"](\"https://i.ibb.co/n3gQzy8/Selection-851.png\")
\n![\"https://i.ibb.co/C70jXPQ/Selection-850.png\"](\"https://i.ibb.co/C70jXPQ/Selection-850.png\")
if you are interested in how to design waveform 1d CNN and compare it with spectrogram 2d CNN, refer to:
\nISMIR 2019 tutorial: waveform-based music processing with deep learning
\nhttps://zenodo.org/record/3529714#.YT1UZnUzbCJ
Learning Multiscale Features Directly From Waveforms
\nhttp://research.baidu.com/Public/uploads/5ac04fa6a3f28.PDF
SampleCNN: End-to-End Deep Convolutional Neural Networks Using Very Small Filters for Music Classification
", "rawMarkdown": "if you are interested in how to design waveform 1d CNN and compare it with spectrogram 2d CNN, refer to:\n\nISMIR 2019 tutorial: waveform-based music processing with deep learning\nhttps://zenodo.org/record/3529714#.YT1UZnUzbCJ\n\nLearning Multiscale Features Directly From Waveforms\nhttp://research.baidu.com/Public/uploads/5ac04fa6a3f28.PDF\n\n\nSampleCNN: End-to-End Deep Convolutional Neural Networks Using Very Small Filters for Music Classification", "votes": null }, { "id": "1510036", "postDate": "09/12/2021 02:11:13", "content": "Interesting. It seems as though you're running your 1d cnn on frequency domain magnitudes as opposed to time domain?
", "rawMarkdown": "Interesting. It seems as though you're running your 1d cnn on frequency domain magnitudes as opposed to time domain?", "votes": null }, { "id": "1510038", "postDate": "09/12/2021 02:16:53", "content": "no. i am running on time domain
", "rawMarkdown": "no. i am running on time domain", "votes": null }, { "id": "1511136", "postDate": "09/13/2021 06:43:39", "content": "interesting, whitening don't work for both 1d cnn and 2d cnn.
\n(maybe 2 sec is too short to estimate noise power)
i transfer the 2d-cnn signal processing for 1d-cnn, the performance of 1d-cnn is greatly increased
\nsimple 8 layer 1dcnn has cv = 0.868902
\n(45 min to train up to 6 epoach)
the same wave signal when transform with CQT(128x256)+eff-b0 gives cv = 0.8716
\nensembling gives cv =0.873843
\nbandpass seems to be a must for 1d-cnn
", "rawMarkdown": "interesting, whitening don't work for both 1d cnn and 2d cnn.\n(maybe 2 sec is too short to estimate noise power)\n\ni transfer the 2d-cnn signal processing for 1d-cnn, the performance of 1d-cnn is greatly increased\n\nsimple 8 layer 1dcnn has cv = 0.868902\n(45 min to train up to 6 epoach)\n\nthe same wave signal when transform with CQT(128x256)+eff-b0 gives cv = 0.8716\n\nensembling gives cv =0.873843\n\n---\n\nbandpass seems to be a must for 1d-cnn", "votes": null }, { "id": "1511166", "postDate": "09/13/2021 07:15:39", "content": "@hengck23 Are you saying you are feeding raw waveform + bandpass to 1D CNNs? Any other pre-processing that you can tell us about?
", "rawMarkdown": "hengck23 Are you saying you are feeding raw waveform + bandpass to 1D CNNs? Any other pre-processing that you can tell us about?", "votes": null }, { "id": "1511309", "postDate": "09/13/2021 10:11:16", "content": "after increasing the parameters of the 1d-cnn:
\n1d-cnn auc cv = 0.870425
\nensemble auc cv = 0.874456
(this is wrong. i forget that the y-axis is log scale)
\n![\"https://i.ibb.co/xhFrMYs/Selection-864.png\"](\"https://i.ibb.co/xhFrMYs/Selection-864.png\")
\n![\"https://i.ibb.co/fCX7YLw/Selection-863.png\"](\"https://i.ibb.co/fCX7YLw/Selection-863.png\")
instead of ensembling, why not train jointly (or partially jointly, e.g. freeze one of the network if you have limited GPU memory)?
\nif the 1d-CNN of the ensemble gets accurate enough, one might want to try distillation
", "rawMarkdown": "after increasing the parameters of the 1d-cnn:\n1d-cnn auc cv = 0.870425\nensemble auc cv = 0.874456\n\n\n~~where did the improvement come from?\ni think we did not detect more GW. But we did reduce the number of FP~~\n\n(this is wrong. i forget that the y-axis is log scale)\n\n\n\n\n\n---\ninstead of ensembling, why not train jointly (or partially jointly, e.g. freeze one of the network if you have limited GPU memory)?\n\nif the 1d-CNN of the ensemble gets accurate enough, one might want to try distillation", "votes": null }, { "id": "1511470", "postDate": "09/13/2021 12:58:52", "content": "\n\ninstead of ensembling, why not train jointly
\n
I did this. But 1d cnn learning schedule is different from 2d, so either separate training then joining needs to be done. Or for more optimal results, careful finetuning of differential learning rates, which I haven't explored.
", "rawMarkdown": "> instead of ensembling, why not train jointly\n\nI did this. But 1d cnn learning schedule is different from 2d, so either separate training then joining needs to be done. Or for more optimal results, careful finetuning of differential learning rates, which I haven't explored.", "votes": null }, { "id": "1511500", "postDate": "09/13/2021 13:27:41", "content": "i just use trained 1d and 2d network as feature extractors.
\nthey are freeze when training a combined network.
but even so, there seems to be overfitting, so maybe it is not easy to train jointly
\nwe can flip left-right and up-down for noise wave form. can we do that for GW samples as well?
", "rawMarkdown": "i just use trained 1d and 2d network as feature extractors.\nthey are freeze when training a combined network.\n\nbut even so, there seems to be overfitting, so maybe it is not easy to train jointly\n\n\nwe can flip left-right and up-down for noise wave form. can we do that for GW samples as well?", "votes": null }, { "id": "1511550", "postDate": "09/13/2021 14:09:41", "content": "I'm not sure that effect of reduction of FP comes from exactly \"ensemble of 1D-net and 2D-net\" but instead because of the nature of ensemble itself
\nWhat I'm saying is that you could achieve same result with ensembling using only 2D-nets, I suppose
\nIt would be interesting to look on the same plots, but with ensemble of 2d-nets only, I should try to do that myself I suppose
By the way, how to you define \"pos\" or \"neg\" in your plots?
\nBecause usually we compute ROC-AUC on the raw probabilities, but judging from the second plot you divided samples into \"pos/neg\" somehow.
By the way, how to you define \"pos\" or \"neg\" in your plots?
\n…
we have the ground truth label in validation for the distribution plots.
\nfor ROC, it is for all train samples (i.e. pos and neg)
@rimuru try something like:
\nhist, edges = np.histogram(z.probs, bins=100)\nplt.plot(edges[:-1], hist, linewidth=1)\nplt.yscale('log')\nplt.show()\nBut split z by train target or oof target.
", "rawMarkdown": "rimuru try something like:\n\n```\nhist, edges = np.histogram(z.probs, bins=100)\nplt.plot(edges[:-1], hist, linewidth=1)\nplt.yscale('log')\nplt.show()\n```\n\nBut split z by train target or oof target.", "votes": null }, { "id": "1511811", "postDate": "09/13/2021 17:52:39", "content": "after more augmentations:
\n\nfor one fold only:
\n2d-cnn ( CQT 128x256 + eff-b0) auc cv = 0.872293
\n1d-cnn (wave input to 8 layer temporal conv) auc cv = 0.873005 (lb =0.8746)
\nensemble: 0.875676 (this is almost the same results as effnet b7)
conclusion:
\n@hengck23 are you using the 1d-cnn on the signal directly or on the spectrogram ?
", "rawMarkdown": "hengck23 are you using the 1d-cnn on the signal directly or on the spectrogram ?", "votes": null }, { "id": "1511860", "postDate": "09/13/2021 18:23:32", "content": "class Net(nn.Module):\n # 1D convolutional neural network. \n def __init__(self):\n super().__init__()\n self.cnn1 = nnn.Conv1d(3, 64, kernel_size=64, padding=32)\n ... etc ...\n\n\n def forward(self, wave):\n x = self.cnn1(wave) \n ... etc ...\n\n\ndef run_check_net():\n batch_size = 5\n wave = torch.randn(batch_size, 3, 4096) \n\n net = Net() \n signal= net(wave)\n![\"https://i.ibb.co/5F3dvjD/Selection-868.png\"](\"https://i.ibb.co/5F3dvjD/Selection-868.png\")
my guess.
\ndo note that performance is limited by data.
\nanything that is in the \"front\" (e.g. data, augmentation, signal processing) set the limit of performance.
\nanything in the \"back\" determines how close you can reach the limit
you can refer to:
\nhttps://arxiv.org/pdf/1912.10211.pdf
\nPanns: Large-scale pretrained audio neural networks for audio pattern recognition
![\"https://i.ibb.co/k9c222Y/Selection-869.png\"](\"https://i.ibb.co/k9c222Y/Selection-869.png\")
but gap between CV and LB is narrow .. with 2d cnn generally we are getting better LB wrt to CV.
\nany thing from this we can tell how well 1dcnn generalize wrt to 2dCNN
i think the only way to is to measure the generalization gap from experiments.
\ni.e. try different parameters and validation sets. we need to measure difference between training and validation performances
Yeah but how do you do transfert learning with 1d convnet ? You take pretrained model (trained on audio recognition tasks ?). My take on deep learning competition is that you always need transfert learning.
\nVery interesting discussion.
", "rawMarkdown": "Yeah but how do you do transfert learning with 1d convnet ? You take pretrained model (trained on audio recognition tasks ?). My take on deep learning competition is that you always need transfert learning.\n\nVery interesting discussion.", "votes": null }, { "id": "1512456", "postDate": "09/14/2021 09:11:54", "content": "you can train from scratch
\n(actually, for 2d cnn , you can also do that for the smaller resnet)
you can always go to github to search for trained model weights on audio, wave or GW data
", "rawMarkdown": "you can train from scratch\n(actually, for 2d cnn , you can also do that for the smaller resnet)\n\nyou can always go to github to search for trained model weights on audio, wave or GW data", "votes": null }, { "id": "1512594", "postDate": "09/14/2021 11:45:08", "content": "@hengck23 thanks, that is very interesting analysis! if you'd like to answer bellow
\nwith k=64, padding=32 the output of 1st conv is: bs, 64, 4097 - do you compute paddings by hand? what follows next? --> FYI Torch 1.9 accepts padding='same'
what is the output of the conv stack blocks in time-dimension ? still remains 4096 or you perform any kind of pooling in between ?
good work!
", "rawMarkdown": "good work!", "votes": null }, { "id": "1513174", "postDate": "09/14/2021 23:07:19", "content": "i find a smart trick to ensure that the learned 1d Conv filters have both \"sine and cosine components\" like the CQT
\n….
sift, flip your input in training and testing (TTA) instead!
", "rawMarkdown": "i find a smart trick to ensure that the learned 1d Conv filters have both \"sine and cosine components\" like the CQT\n....\n\n sift, flip your input in training and testing (TTA) instead!", "votes": null }, { "id": "1514331", "postDate": "09/16/2021 01:28:25", "content": "Is there any tips to train 1dCNN? I tried to follow the GW papers but my model won't learn anything. It get stuck at ROC = 0.5 forever
", "rawMarkdown": "Is there any tips to train 1dCNN? I tried to follow the GW papers but my model won't learn anything. It get stuck at ROC = 0.5 forever", "votes": null }, { "id": "1515889", "postDate": "09/17/2021 16:07:09", "content": "Great work!
", "rawMarkdown": "Great work!", "votes": null }, { "id": "1515990", "postDate": "09/17/2021 18:26:09", "content": "normalization!
", "rawMarkdown": "normalization!", "votes": null }, { "id": "1517001", "postDate": "09/19/2021 04:50:15", "content": "Tnx for answering. Do you mind elaborating a bit more? or maybe pointing me to some good material? Thanks
", "rawMarkdown": "Tnx for answering. Do you mind elaborating a bit more? or maybe pointing me to some good material? Thanks", "votes": null }, { "id": "1559723", "postDate": "10/27/2021 07:09:30", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
\n\nThe CQT is not used in the CNN
\n
What's used ?
", "votes": null, "replies": [ { "id": 1509910, "author_name": "brachester", "author_url": "", "post_date": "09/11/2021 19:17:07", "content": "It's a 1d cnn, you're thinking of 2d cnn. In this case, he probably used the raw time series.
", "votes": null, "replies": [] }, { "id": 1509912, "author_name": "zarif98sjs", "author_url": "", "post_date": "09/11/2021 19:19:16", "content": "Oops sorry, missed that part !
", "votes": null, "replies": [] } ] }, { "id": 1510028, "author_name": "hengck23", "author_url": "", "post_date": "09/12/2021 01:09:36", "content": "update1.
\n(selection of max kernel (red box) could be wrong as I did not consider the weights of the next bn layer)
\n
Interesting. It seems as though you're running your 1d cnn on frequency domain magnitudes as opposed to time domain?
", "votes": null, "replies": [] }, { "id": 1510038, "author_name": "hengck23", "author_url": "", "post_date": "09/12/2021 02:16:53", "content": "no. i am running on time domain
", "votes": null, "replies": [] } ] }, { "id": 1510029, "author_name": "hengck23", "author_url": "", "post_date": "09/12/2021 01:17:44", "content": "if you are interested in how to design waveform 1d CNN and compare it with spectrogram 2d CNN, refer to:
\nISMIR 2019 tutorial: waveform-based music processing with deep learning
\nhttps://zenodo.org/record/3529714#.YT1UZnUzbCJ
Learning Multiscale Features Directly From Waveforms
\nhttp://research.baidu.com/Public/uploads/5ac04fa6a3f28.PDF
SampleCNN: End-to-End Deep Convolutional Neural Networks Using Very Small Filters for Music Classification
", "votes": null, "replies": [] }, { "id": 1511136, "author_name": "hengck23", "author_url": "", "post_date": "09/13/2021 06:43:39", "content": "interesting, whitening don't work for both 1d cnn and 2d cnn.
\n(maybe 2 sec is too short to estimate noise power)
i transfer the 2d-cnn signal processing for 1d-cnn, the performance of 1d-cnn is greatly increased
\nsimple 8 layer 1dcnn has cv = 0.868902
\n(45 min to train up to 6 epoach)
the same wave signal when transform with CQT(128x256)+eff-b0 gives cv = 0.8716
\nensembling gives cv =0.873843
\nbandpass seems to be a must for 1d-cnn
", "votes": null, "replies": [ { "id": 1511166, "author_name": "pheadrus", "author_url": "", "post_date": "09/13/2021 07:15:39", "content": "@hengck23 Are you saying you are feeding raw waveform + bandpass to 1D CNNs? Any other pre-processing that you can tell us about?
", "votes": null, "replies": [] }, { "id": 1511309, "author_name": "hengck23", "author_url": "", "post_date": "09/13/2021 10:11:16", "content": "after increasing the parameters of the 1d-cnn:
\n1d-cnn auc cv = 0.870425
\nensemble auc cv = 0.874456
(this is wrong. i forget that the y-axis is log scale)
\n
\n
instead of ensembling, why not train jointly (or partially jointly, e.g. freeze one of the network if you have limited GPU memory)?
\nif the 1d-CNN of the ensemble gets accurate enough, one might want to try distillation
", "votes": null, "replies": [] }, { "id": 1511470, "author_name": "authman", "author_url": "", "post_date": "09/13/2021 12:58:52", "content": "\n\ninstead of ensembling, why not train jointly
\n
I did this. But 1d cnn learning schedule is different from 2d, so either separate training then joining needs to be done. Or for more optimal results, careful finetuning of differential learning rates, which I haven't explored.
", "votes": null, "replies": [] }, { "id": 1511500, "author_name": "hengck23", "author_url": "", "post_date": "09/13/2021 13:27:41", "content": "i just use trained 1d and 2d network as feature extractors.
\nthey are freeze when training a combined network.
but even so, there seems to be overfitting, so maybe it is not easy to train jointly
\nwe can flip left-right and up-down for noise wave form. can we do that for GW samples as well?
", "votes": null, "replies": [] }, { "id": 1511550, "author_name": "martynoveduard", "author_url": "", "post_date": "09/13/2021 14:09:41", "content": "I'm not sure that effect of reduction of FP comes from exactly \"ensemble of 1D-net and 2D-net\" but instead because of the nature of ensemble itself
\nWhat I'm saying is that you could achieve same result with ensembling using only 2D-nets, I suppose
\nIt would be interesting to look on the same plots, but with ensemble of 2d-nets only, I should try to do that myself I suppose
By the way, how to you define \"pos\" or \"neg\" in your plots?
\nBecause usually we compute ROC-AUC on the raw probabilities, but judging from the second plot you divided samples into \"pos/neg\" somehow.
By the way, how to you define \"pos\" or \"neg\" in your plots?
\n…
we have the ground truth label in validation for the distribution plots.
\nfor ROC, it is for all train samples (i.e. pos and neg)
@rimuru try something like:
\nhist, edges = np.histogram(z.probs, bins=100)\nplt.plot(edges[:-1], hist, linewidth=1)\nplt.yscale('log')\nplt.show()\nBut split z by train target or oof target.
", "votes": null, "replies": [] }, { "id": 1511811, "author_name": "hengck23", "author_url": "", "post_date": "09/13/2021 17:52:39", "content": "after more augmentations:
\n\nfor one fold only:
\n2d-cnn ( CQT 128x256 + eff-b0) auc cv = 0.872293
\n1d-cnn (wave input to 8 layer temporal conv) auc cv = 0.873005 (lb =0.8746)
\nensemble: 0.875676 (this is almost the same results as effnet b7)
conclusion:
\n@hengck23 are you using the 1d-cnn on the signal directly or on the spectrogram ?
", "votes": null, "replies": [] }, { "id": 1511860, "author_name": "hengck23", "author_url": "", "post_date": "09/13/2021 18:23:32", "content": "class Net(nn.Module):\n # 1D convolutional neural network. \n def __init__(self):\n super().__init__()\n self.cnn1 = nnn.Conv1d(3, 64, kernel_size=64, padding=32)\n ... etc ...\n\n\n def forward(self, wave):\n x = self.cnn1(wave) \n ... etc ...\n\n\ndef run_check_net():\n batch_size = 5\n wave = torch.randn(batch_size, 3, 4096) \n\n net = Net() \n signal= net(wave)\n
my guess.
\ndo note that performance is limited by data.
\nanything that is in the \"front\" (e.g. data, augmentation, signal processing) set the limit of performance.
\nanything in the \"back\" determines how close you can reach the limit
you can refer to:
\nhttps://arxiv.org/pdf/1912.10211.pdf
\nPanns: Large-scale pretrained audio neural networks for audio pattern recognition
but gap between CV and LB is narrow .. with 2d cnn generally we are getting better LB wrt to CV.
\nany thing from this we can tell how well 1dcnn generalize wrt to 2dCNN
i think the only way to is to measure the generalization gap from experiments.
\ni.e. try different parameters and validation sets. we need to measure difference between training and validation performances
@hengck23 thanks, that is very interesting analysis! if you'd like to answer bellow
\nwith k=64, padding=32 the output of 1st conv is: bs, 64, 4097 - do you compute paddings by hand? what follows next? --> FYI Torch 1.9 accepts padding='same'
what is the output of the conv stack blocks in time-dimension ? still remains 4096 or you perform any kind of pooling in between ?
Yeah but how do you do transfert learning with 1d convnet ? You take pretrained model (trained on audio recognition tasks ?). My take on deep learning competition is that you always need transfert learning.
\nVery interesting discussion.
", "votes": null, "replies": [ { "id": 1512456, "author_name": "hengck23", "author_url": "", "post_date": "09/14/2021 09:11:54", "content": "you can train from scratch
\n(actually, for 2d cnn , you can also do that for the smaller resnet)
you can always go to github to search for trained model weights on audio, wave or GW data
", "votes": null, "replies": [] } ] }, { "id": 1512963, "author_name": "faaizhashmi", "author_url": "", "post_date": "09/14/2021 17:53:59", "content": "good work!
", "votes": null, "replies": [] }, { "id": 1513174, "author_name": "hengck23", "author_url": "", "post_date": "09/14/2021 23:07:19", "content": "i find a smart trick to ensure that the learned 1d Conv filters have both \"sine and cosine components\" like the CQT
\n….
sift, flip your input in training and testing (TTA) instead!
", "votes": null, "replies": [] }, { "id": 1514331, "author_name": "coldfir3", "author_url": "", "post_date": "09/16/2021 01:28:25", "content": "Is there any tips to train 1dCNN? I tried to follow the GW papers but my model won't learn anything. It get stuck at ROC = 0.5 forever
", "votes": null, "replies": [ { "id": 1515990, "author_name": "abhishek", "author_url": "", "post_date": "09/17/2021 18:26:09", "content": "normalization!
", "votes": null, "replies": [] }, { "id": 1517001, "author_name": "coldfir3", "author_url": "", "post_date": "09/19/2021 04:50:15", "content": "Tnx for answering. Do you mind elaborating a bit more? or maybe pointing me to some good material? Thanks
", "votes": null, "replies": [] } ] }, { "id": 1515889, "author_name": "katew19", "author_url": "", "post_date": "09/17/2021 16:07:09", "content": "Great work!
", "votes": null, "replies": [] }, { "id": 1559723, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 07:09:30", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris