i spend the last week on this. Did anyone really get this to work?
\nI cannot understand why it doesn't work …. this has been widely used in the GW communities.
i spend the last week on this. Did anyone really get this to work?
\nI cannot understand why it doesn't work …. this has been widely used in the GW communities.
It works well for me. Well, it doesnt if you do it wrong of course. It doesn't most probably because you are messing up your activation distributions from noisy low data (num of points, 4096) samples.
\nI did very simple fft - PSD -ifft type of thing, but with PSD taken mean batchwise for each channel, [1, 3, 4096].
", "rawMarkdown": "It works well for me. Well, it doesnt if you do it wrong of course. It doesn't most probably because you are messing up your activation distributions from noisy low data (num of points, 4096) samples.\n\nI did very simple fft - PSD -ifft type of thing, but with PSD taken mean batchwise for each channel, [1, 3, 4096].", "votes": null }, { "id": "1528909", "postDate": "09/30/2021 03:10:01", "content": "thanks for the comment. how much improvement does it bring? some teams reported poorer accuracy in the forum.
\n\" PSD taken mean batchwise …\"
\nso it is batch-dependent. won't that affect your testing?
It should, and in final submissions I fixed that with precomputed value. I did it very early in competition, and results without it \\ or with \"unfixed\" PSD whitening were so bad i did not submit any of it. I definitely think it helps.
", "rawMarkdown": "It should, and in final submissions I fixed that with precomputed value. I did it very early in competition, and results without it \\ or with \"unfixed\" PSD whitening were so bad i did not submit any of it. I definitely think it helps.", "votes": null }, { "id": "1530300", "postDate": "10/01/2021 04:58:55", "content": "@hengck23 Similar whitening approach gave us ~0.003 boost early on (8650 range) and ~0.001+ boost later (8800 range)
", "rawMarkdown": "hengck23 Similar whitening approach gave us ~0.003 boost early on (8650 range) and ~0.001+ boost later (8800 range)", "votes": null }, { "id": "1530341", "postDate": "10/01/2021 05:33:55", "content": "The trick is using mean over the train set PSD for normalization… as found by my teammates
", "rawMarkdown": "The trick is using mean over the train set PSD for normalization... as found by my teammates", "votes": null }, { "id": "1530354", "postDate": "10/01/2021 05:45:28", "content": "In theory whitening should not work here. When you are doing whitening properly you are basically applying filter with length around few seconds, so entire signal should be messed up. Period.
\nBUT.
\nHere we had generated noise. PSD was same for entire dataset and PSD was smooth. In reality only few hundred samples where corrupted badly, so you can make it work \"good enough for practical application\".
\n@bakeryproducts -style - just do it and apply window to eliminate insane border effects, than normalize. Key here is to pre-calculate PSD on noise-only samples, than you are good to go. Don't write this in whitepaper tho.
\nAlso you can just find in dataset sequence of samples to smoothly continue signal and then do whitening. I personally just train linear AR on signal, predicted few samples and search for close match in dataset. Visually yielded much cleaner result, but did not put it in our pipeline, already was no use for us at a time.
thanks for the information.
\n\" I personally just train linear AR on signal\"
\nI would like to try that since I read and self-study on the topic.
\nDo you have code on that?
It depends on how you define whitening.
\nUsing the usual whitening methods did not help us. Dividing by average noise helped.
", "rawMarkdown": "It depends on how you define whitening. \n\nUsing the usual whitening methods did not help us. Dividing by average noise helped.", "votes": null }, { "id": "1532965", "postDate": "10/03/2021 14:38:21", "content": "Median over the noise samples might be more robust to Glitchs. As I understand, we want to keep the glitches in the whitened signal and then the model should be able to differentiate between the Glitchs vs the GW signal. There is a median correction that needs to be applied (using _median_bias from scipy/signal/spectral.py) and then one could smooth along the frequency access but a special treatment is needed around the few peaks of the PSD to avoid smearing those -- see https://arxiv.org/pdf/2108.10588.pdf
", "rawMarkdown": "Median over the noise samples might be more robust to Glitchs. As I understand, we want to keep the glitches in the whitened signal and then the model should be able to differentiate between the Glitchs vs the GW signal. There is a median correction that needs to be applied (using _median_bias from scipy/signal/spectral.py) and then one could smooth along the frequency access but a special treatment is needed around the few peaks of the PSD to avoid smearing those -- see https://arxiv.org/pdf/2108.10588.pdf", "votes": null }, { "id": "1559906", "postDate": "10/27/2021 08:07:47", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
It works well for me. Well, it doesnt if you do it wrong of course. It doesn't most probably because you are messing up your activation distributions from noisy low data (num of points, 4096) samples.
\nI did very simple fft - PSD -ifft type of thing, but with PSD taken mean batchwise for each channel, [1, 3, 4096].
", "votes": null, "replies": [ { "id": 1528909, "author_name": "hengck23", "author_url": "", "post_date": "09/30/2021 03:10:01", "content": "thanks for the comment. how much improvement does it bring? some teams reported poorer accuracy in the forum.
\n\" PSD taken mean batchwise …\"
\nso it is batch-dependent. won't that affect your testing?
It should, and in final submissions I fixed that with precomputed value. I did it very early in competition, and results without it \\ or with \"unfixed\" PSD whitening were so bad i did not submit any of it. I definitely think it helps.
", "votes": null, "replies": [] }, { "id": 1530300, "author_name": "richx86", "author_url": "", "post_date": "10/01/2021 04:58:55", "content": "@hengck23 Similar whitening approach gave us ~0.003 boost early on (8650 range) and ~0.001+ boost later (8800 range)
", "votes": null, "replies": [] }, { "id": 1530341, "author_name": "iafoss", "author_url": "", "post_date": "10/01/2021 05:33:55", "content": "The trick is using mean over the train set PSD for normalization… as found by my teammates
", "votes": null, "replies": [] }, { "id": 1532965, "author_name": "ggrizzly", "author_url": "", "post_date": "10/03/2021 14:38:21", "content": "Median over the noise samples might be more robust to Glitchs. As I understand, we want to keep the glitches in the whitened signal and then the model should be able to differentiate between the Glitchs vs the GW signal. There is a median correction that needs to be applied (using _median_bias from scipy/signal/spectral.py) and then one could smooth along the frequency access but a special treatment is needed around the few peaks of the PSD to avoid smearing those -- see https://arxiv.org/pdf/2108.10588.pdf
", "votes": null, "replies": [] } ] }, { "id": 1530354, "author_name": "denisbsu", "author_url": "", "post_date": "10/01/2021 05:45:28", "content": "In theory whitening should not work here. When you are doing whitening properly you are basically applying filter with length around few seconds, so entire signal should be messed up. Period.
\nBUT.
\nHere we had generated noise. PSD was same for entire dataset and PSD was smooth. In reality only few hundred samples where corrupted badly, so you can make it work \"good enough for practical application\".
\n@bakeryproducts -style - just do it and apply window to eliminate insane border effects, than normalize. Key here is to pre-calculate PSD on noise-only samples, than you are good to go. Don't write this in whitepaper tho.
\nAlso you can just find in dataset sequence of samples to smoothly continue signal and then do whitening. I personally just train linear AR on signal, predicted few samples and search for close match in dataset. Visually yielded much cleaner result, but did not put it in our pipeline, already was no use for us at a time.
thanks for the information.
\n\" I personally just train linear AR on signal\"
\nI would like to try that since I read and self-study on the topic.
\nDo you have code on that?
It depends on how you define whitening.
\nUsing the usual whitening methods did not help us. Dividing by average noise helped.
", "votes": null, "replies": [] }, { "id": 1559906, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 08:07:47", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris