Many kernels suggest noise-removing tasks, but I think this can be very dangerous.
\nIn reality, we cannot determine in which area gravitational waves are detected.
\nI'm interested in embedding the time series directly rather than processing it as an image, but noise in the process is too harmful.
\nI've tried various methods to filter out the noise, but I think it works best when the noise is removed as little as possible.
\nCurrently, most kernels work with images, and I'd wish to hear what you think is an appropriate useful freq band for image preprocessing.
", "messages": [ { "id": "1393322", "postDate": "07/19/2021 14:42:35", "content": "Many kernels suggest noise-removing tasks, but I think this can be very dangerous.
\nIn reality, we cannot determine in which area gravitational waves are detected.
\nI'm interested in embedding the time series directly rather than processing it as an image, but noise in the process is too harmful.
\nI've tried various methods to filter out the noise, but I think it works best when the noise is removed as little as possible.
\nCurrently, most kernels work with images, and I'd wish to hear what you think is an appropriate useful freq band for image preprocessing.
", "rawMarkdown": "Many kernels suggest noise-removing tasks, but I think this can be very dangerous.\n\nIn reality, we cannot determine in which area gravitational waves are detected.\n\nI'm interested in embedding the time series directly rather than processing it as an image, but noise in the process is too harmful.\n\nI've tried various methods to filter out the noise, but I think it works best when the noise is removed as little as possible.\n\nCurrently, most kernels work with images, and I'd wish to hear what you think is an appropriate useful freq band for image preprocessing.", "votes": null }, { "id": "1395063", "postDate": "07/20/2021 21:18:34", "content": "\n\nbut I think it works best when the noise is removed as little as possible.
\n
Can you explain why you think so?
", "rawMarkdown": "> but I think it works best when the noise is removed as little as possible.\n\nCan you explain why you think so?", "votes": null }, { "id": "1395286", "postDate": "07/21/2021 05:30:28", "content": "Although I applied noise filtering, currently being discussed in many kernels, it has not resulted in a good score in the validation set.
\nMaybe it's because I haven't tried the signal to image conversion process (e.g., Q transform).
\nMeanwhile, the best score I got using Eff-net after converting to an image was when I applied the Tukey filter.
", "rawMarkdown": "Although I applied noise filtering, currently being discussed in many kernels, it has not resulted in a good score in the validation set.\n\nMaybe it's because I haven't tried the signal to image conversion process (e.g., Q transform).\n\nMeanwhile, the best score I got using Eff-net after converting to an image was when I applied the Tukey filter.", "votes": null }, { "id": "1396162", "postDate": "07/21/2021 21:02:24", "content": "Hey,
\nAccording to this article ( https://iopscience.iop.org/article/10.1088/1361-6382/ab685e ), the gravitational waves should produce Signals in the Frequency Band between 20 and 350 Hz. I have compared the Fourier Transformed dataset and apparently the biggest difference in Frequency Occurence appears between 20 and 75 Hz:
\nhttps://www.kaggle.com/maxdner/preprocessing-most-important-frequencies
Thank you! ๐๐๐
", "rawMarkdown": "Thank you! ๐๐๐", "votes": null }, { "id": "1396573", "postDate": "07/22/2021 08:09:11", "content": "Hi Max, just to clarify, the frequency content of our signals is variable and the upper frequency depends strongly on the total mass of the system. Every signal in the dataset has a different total mass drawn from an astrophysically motivated distribution.
\nCheers, Chris
", "rawMarkdown": "Hi Max, just to clarify, the frequency content of our signals is variable and the upper frequency depends strongly on the total mass of the system. Every signal in the dataset has a different total mass drawn from an astrophysically motivated distribution.\n\nCheers, Chris", "votes": null }, { "id": "1397611", "postDate": "07/23/2021 10:50:07", "content": "The whitening of the image is not easy to use, that is, the effect of removing noise is not obvious.This dataset has not found the right way to remove noise effectively yet. We look forward to your discovery๐
", "rawMarkdown": "The whitening of the image is not easy to use, that is, the effect of removing noise is not obvious.This dataset has not found the right way to remove noise effectively yet. We look forward to your discovery๐", "votes": null }, { "id": "1399771", "postDate": "07/25/2021 16:49:04", "content": "I'm putting together a larger literature review notebook in the next few days (keep an eye out or follow me) that addresses this question, among other things.
\nThere's a nice review here that goes over noise artifacts in the various detectors (different detectors have different noise artifacts!): https://arxiv.org/pdf/2101.11673.pdf
\nAlso, here are some collected frequency bands for narrow noise transients and what they mean. It's not pretty to read, but it might be used to generate features.
\nMore soon
", "rawMarkdown": "I'm putting together a larger literature review notebook in the next few days (keep an eye out or follow me) that addresses this question, among other things.\n\nThere's a nice review here that goes over noise artifacts in the various detectors (different detectors have different noise artifacts!): https://arxiv.org/pdf/2101.11673.pdf\n\nAlso, here are some collected frequency bands for narrow noise transients and what they mean. It's not pretty to read, but it might be used to generate features.\n\n- https://www.gw-openscience.org/static/speclines/o3a/O3aH1lines_gated_v1.10.txt\n- https://www.gw-openscience.org/static/speclines/o3a/O3aL1lines_gated_v1.10.txt\n- https://www.gw-openscience.org/static/speclines/o3a/O3V_LinesList_v2.txt\n\nMore soon", "votes": null }, { "id": "1404451", "postDate": "07/29/2021 21:43:24", "content": "I think this is an astute question that would benefit from the insights of a theoretical physicist/paper.
\nI've seen the calculations for a 2 black-hole event, which maxes out around 350Hz if I recall correctly.
\nCan you point to papers for other types of events.
I found a simple 30-500Hz bandpass filter made a big difference. I then looked at Spectral Whitening, which works well for the traditional GW Events, but found it was not that useful for the low SNR events.
", "rawMarkdown": "I found a simple 30-500Hz bandpass filter made a big difference. I then looked at [Spectral Whitening](https://www.kaggle.com/kevinmcisaac/g2net-spectral-whitening), which works well for the traditional GW Events, but found it was not that useful for the low SNR events.", "votes": null }, { "id": "1561286", "postDate": "10/27/2021 13:12:22", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
\n\nbut I think it works best when the noise is removed as little as possible.
\n
Can you explain why you think so?
", "votes": null, "replies": [ { "id": 1395286, "author_name": "tenduck", "author_url": "", "post_date": "07/21/2021 05:30:28", "content": "Although I applied noise filtering, currently being discussed in many kernels, it has not resulted in a good score in the validation set.
\nMaybe it's because I haven't tried the signal to image conversion process (e.g., Q transform).
\nMeanwhile, the best score I got using Eff-net after converting to an image was when I applied the Tukey filter.
", "votes": null, "replies": [] }, { "id": 1404452, "author_name": "kevinmcisaac", "author_url": "", "post_date": "07/29/2021 21:45:29", "content": "I found a simple 30-500Hz bandpass filter made a big difference. I then looked at Spectral Whitening, which works well for the traditional GW Events, but found it was not that useful for the low SNR events.
", "votes": null, "replies": [] } ] }, { "id": 1396162, "author_name": "maxdner", "author_url": "", "post_date": "07/21/2021 21:02:24", "content": "Hey,
\nAccording to this article ( https://iopscience.iop.org/article/10.1088/1361-6382/ab685e ), the gravitational waves should produce Signals in the Frequency Band between 20 and 350 Hz. I have compared the Fourier Transformed dataset and apparently the biggest difference in Frequency Occurence appears between 20 and 75 Hz:
\nhttps://www.kaggle.com/maxdner/preprocessing-most-important-frequencies
Thank you! ๐๐๐
", "votes": null, "replies": [] }, { "id": 1396573, "author_name": "bayeswolf", "author_url": "", "post_date": "07/22/2021 08:09:11", "content": "Hi Max, just to clarify, the frequency content of our signals is variable and the upper frequency depends strongly on the total mass of the system. Every signal in the dataset has a different total mass drawn from an astrophysically motivated distribution.
\nCheers, Chris
", "votes": null, "replies": [] }, { "id": 1404451, "author_name": "kevinmcisaac", "author_url": "", "post_date": "07/29/2021 21:43:24", "content": "I think this is an astute question that would benefit from the insights of a theoretical physicist/paper.
\nI've seen the calculations for a 2 black-hole event, which maxes out around 350Hz if I recall correctly.
\nCan you point to papers for other types of events.
The whitening of the image is not easy to use, that is, the effect of removing noise is not obvious.This dataset has not found the right way to remove noise effectively yet. We look forward to your discovery๐
", "votes": null, "replies": [] }, { "id": 1399771, "author_name": "tpmeli", "author_url": "", "post_date": "07/25/2021 16:49:04", "content": "I'm putting together a larger literature review notebook in the next few days (keep an eye out or follow me) that addresses this question, among other things.
\nThere's a nice review here that goes over noise artifacts in the various detectors (different detectors have different noise artifacts!): https://arxiv.org/pdf/2101.11673.pdf
\nAlso, here are some collected frequency bands for narrow noise transients and what they mean. It's not pretty to read, but it might be used to generate features.
\nMore soon
", "votes": null, "replies": [] }, { "id": 1561286, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 13:12:22", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris