{ "id": 268706, "title": "Has anyone tried matched filter?", "url": "/competitions/g2net-gravitational-wave-detection/discussion/268706", "author_name": "", "post_date": "2021-08-28T13:46:14.581268200Z", "votes": 8, "comment_count": 16, "views": 0, "content": "

Mached filter is a common signal processing method to extract a known signal (called the template) from an unknown signal.

\n

It seems to be possible here since the GW signal can be modeled using several parameters so it could have an approximate formula, for example the binary inspiral chirp waveform (check the details in the screenshot below).
\n\"matched-filtering-gw\"

\n

In this master thesis, there is a whole chapter (chapter 1) with more details. You can find more details as well in these slides.

\n

As the title of this discussion asks, has anyone tried this method either as a processing step or as a model? Let me know if you have found any useful discussions and/or notebooks, thanks in advance.

\n

[EDIT] As suggested by @hengck23, it seems that there is something for matched filtering in the PyCBC library: https://www.atlas.aei.uni-hannover.de/work/ahnitz/projects/tplot/html/workflow/matched_filter.html. There is also a colab notebook.

", "messages": [ { "id": "1494206", "postDate": "08/28/2021 13:46:14", "content": "

Mached filter is a common signal processing method to extract a known signal (called the template) from an unknown signal.

\n

It seems to be possible here since the GW signal can be modeled using several parameters so it could have an approximate formula, for example the binary inspiral chirp waveform (check the details in the screenshot below).
\n\"matched-filtering-gw\"

\n

In this master thesis, there is a whole chapter (chapter 1) with more details. You can find more details as well in these slides.

\n

As the title of this discussion asks, has anyone tried this method either as a processing step or as a model? Let me know if you have found any useful discussions and/or notebooks, thanks in advance.

\n

[EDIT] As suggested by @hengck23, it seems that there is something for matched filtering in the PyCBC library: https://www.atlas.aei.uni-hannover.de/work/ahnitz/projects/tplot/html/workflow/matched_filter.html. There is also a colab notebook.

", "rawMarkdown": "[**Mached filter**](https://en.wikipedia.org/wiki/Matched_filter) is a common signal processing method to extract a known signal (called the template) from an unknown signal.\n\nIt seems to be possible here since the GW signal can be modeled using several parameters so it could have an approximate formula, for example the **binary inspiral chirp waveform** (check the details in the screenshot below). \n\"matched-filtering-gw\"\n\n\nIn this [master thesis](https://matheo.uliege.be/bitstream/2268.2/9211/4/Master_thesis_Janquart.pdf), there is a whole chapter (chapter 1) with more details. You can find more details as well in these [slides](https://indico.cern.ch/event/779256/contributions/3242644/attachments/1780516/2896456/dent_ggww_2.pdf).\n\nAs the title of this discussion asks, has anyone tried this method either as a processing step or as a model? Let me know if you have found any useful discussions and/or notebooks, thanks in advance. \n\n\n[EDIT] As suggested by @hengck23, it seems that there is something for matched filtering in the [**PyCBC**](https://pycbc.org/) library: https://www.atlas.aei.uni-hannover.de/work/ahnitz/projects/tplot/html/workflow/matched_filter.html. There is also a colab [notebook](https://colab.research.google.com/github/gw-odw/odw-2020/blob/master/Day_2/Tuto_2.1_Matched_filtering_introduction.ipynb).", "votes": null }, { "id": "1494226", "postDate": "08/28/2021 14:02:35", "content": "

Not directly related but I think I finally found an easy service to upload images: https://imgbb.com/. 😄

", "rawMarkdown": "Not directly related but I think I finally found an easy service to upload images: https://imgbb.com/. 😄", "votes": null }, { "id": "1494228", "postDate": "08/28/2021 14:06:22", "content": "

kinda brute force method. see figure.1 of https://arxiv.org/pdf/1904.08693.pdf
\nyou can google for :\"PyCBC template matching\"

\n

https://colab.research.google.com/github/gw-odw/odw-2020/blob/master/Day_2/Tuto_2.1_Matched_filtering_introduction.ipynb

\n

https://www.atlas.aei.uni-hannover.de/work/ahnitz/projects/tplot/html/workflow/matched_filter.html

\n

it looks something like this:
\nhttps://www.youtube.com/watch?v=bBBDR5jf9oU

\n

there could be about 250k templates? in the match filter bank. it is very computational intensive

", "rawMarkdown": "kinda brute force method. see figure.1 of https://arxiv.org/pdf/1904.08693.pdf\nyou can google for :\"PyCBC template matching\"\n\nhttps://colab.research.google.com/github/gw-odw/odw-2020/blob/master/Day_2/Tuto_2.1_Matched_filtering_introduction.ipynb\n\nhttps://www.atlas.aei.uni-hannover.de/work/ahnitz/projects/tplot/html/workflow/matched_filter.html\n\nit looks something like this:\nhttps://www.youtube.com/watch?v=bBBDR5jf9oU\n\nthere could be about 250k templates? in the match filter bank. it is very computational intensive", "votes": null }, { "id": "1494237", "postDate": "08/28/2021 14:13:22", "content": "

Thanks for the links, I will add these. 👌

\n

I have also noticed that PyCBC seems to be THE GW library so there should be something there indeed!

", "rawMarkdown": "Thanks for the links, I will add these. 👌\n\nI have also noticed that [PyCBC](https://pycbc.org/) seems to be THE GW library so there should be something there indeed!", "votes": null }, { "id": "1494247", "postDate": "08/28/2021 14:21:17", "content": "

if someone can upload whiten waveform dataset, then we can try 1d convolution net or wavenet model

", "rawMarkdown": "if someone can upload whiten waveform dataset, then we can try 1d convolution net or wavenet model", "votes": null }, { "id": "1494319", "postDate": "08/28/2021 15:33:19", "content": "

I'm pretty sure the entire point of them holding this competition is to find a more scalable approach for future detection since match filtering is computationally prohibitive. This paper said something similar https://arxiv.org/pdf/1711.03121.pdf

", "rawMarkdown": "I'm pretty sure the entire point of them holding this competition is to find a more scalable approach for future detection since match filtering is computationally prohibitive. This paper said something similar https://arxiv.org/pdf/1711.03121.pdf", "votes": null }, { "id": "1494340", "postDate": "08/28/2021 15:44:43", "content": "

Thanks for the paper. Do you have an idea about the time difference between the two approaches: matched filter vs deep learning model?

", "rawMarkdown": "Thanks for the paper. Do you have an idea about the time difference between the two approaches: matched filter vs deep learning model?", "votes": null }, { "id": "1494352", "postDate": "08/28/2021 15:56:36", "content": "

I'm new to signal processing so I don't know when matched filtering was first proposed but deep learning approach is obviously a recent thing. Matched filtering was at least used in the Nobel prize winning first detection back in 2015 https://arxiv.org/ftp/arxiv/papers/1602/1602.03837.pdf

", "rawMarkdown": "I'm new to signal processing so I don't know when matched filtering was first proposed but deep learning approach is obviously a recent thing. Matched filtering was at least used in the Nobel prize winning first detection back in 2015 https://arxiv.org/ftp/arxiv/papers/1602/1602.03837.pdf", "votes": null }, { "id": "1494387", "postDate": "08/28/2021 16:20:39", "content": "

I am a new comer to this field as well. I will keep in mind the time constraint and post an update if I find more details. 👌

", "rawMarkdown": "I am a new comer to this field as well. I will keep in mind the time constraint and post an update if I find more details. 👌", "votes": null }, { "id": "1495553", "postDate": "08/29/2021 16:20:43", "content": "

\"\"

", "rawMarkdown": "![](https://i.imgur.com/ca2eCfK.png)", "votes": null }, { "id": "1495566", "postDate": "08/29/2021 16:26:52", "content": "
\n

I'm pretty sure the entire point of them holding this competition is to find a more scalable approach for future detection since match filtering is computationally prohibitive

\n
\n

Agreed. MF is literally state of the art and is currently (mathematically) the most robust way for finding signal in noise that swamps it by orders of magnitude. Even more so because it allows for he discovery of the signal's parameters even if they aren't known, so long as the form/equation of the signal is known. But its slowwwww and due to the number of templates needed and the amount of data being generated, I believe the desire here isn't to replace MF entirely, but rather to have us create realtime / near-realtime candidate algorithms, that will then have their proposals fed into the exiting MF systems for proper classification and parameter estimation, etc.

\n

In the context of this competition, if it's discovered that they only used a small subset of templates to generate the GWs and did not vary it much, and if one were able to discover those templates, it theoretically should be very easy to own this thing. lol. I doubt they made that mistake but who knows…. :-)

", "rawMarkdown": "> I'm pretty sure the entire point of them holding this competition is to find a more scalable approach for future detection since match filtering is computationally prohibitive\n\nAgreed. MF is literally state of the art and is currently (mathematically) the most robust way for finding signal in noise that swamps it by orders of magnitude. Even more so because it allows for he discovery of the signal's parameters even if they aren't known, so long as the form/equation of the signal is known. But its slowwwww and due to the number of templates needed and the amount of data being generated, I believe the desire here isn't to replace MF entirely, but rather to have us create realtime / near-realtime candidate algorithms, that will then have their proposals fed into the exiting MF systems for proper classification and parameter estimation, etc.\n\nIn the context of this competition, if it's discovered that they only used a small subset of templates to generate the GWs and did not vary it much, and if one were able to discover those templates, it theoretically should be very easy to own this thing. lol. I doubt they made that mistake but who knows.... :-)", "votes": null }, { "id": "1495572", "postDate": "08/29/2021 16:28:44", "content": "

@authman thanks for the graph and indeed, that's a big speed-up! 👌

", "rawMarkdown": "authman thanks for the graph and indeed, that's a big speed-up! 👌", "votes": null }, { "id": "1498281", "postDate": "08/31/2021 20:10:29", "content": "

i have a feeling that modern matched filtering may be accomplished by a transformer.

\n

assuming input is patches of (LIGO Hanford, LIGO Livingston, and Virgo), in self-attention, you can computing similarity (i.e. matching) of the 3 measurements. the network will learn the tempates

\n

you can also use input = (LIGO Hanford, LIGO Livingston, Virgo, other out-of-distribution template)

", "rawMarkdown": "i have a feeling that modern matched filtering may be accomplished by a transformer.\n\nassuming input is patches of (LIGO Hanford, LIGO Livingston, and Virgo), in self-attention, you can computing similarity (i.e. matching) of the 3 measurements. the network will learn the tempates\n\nyou can also use input = (LIGO Hanford, LIGO Livingston, Virgo, other out-of-distribution template)", "votes": null }, { "id": "1502122", "postDate": "09/03/2021 21:35:20", "content": "

\"\"

\n

Generalized Approach to Matched Filtering using Neural Networks

\n

Everything sounds good until they basically say FNN are universal function approximaters and argue a 1-hidden layer net can learn the mapping 🤥. They do also offer a few layer deep FNN though.

", "rawMarkdown": "![](https://i.imgur.com/2iVbHGc.png)\n\n[Generalized Approach to Matched Filtering using Neural Networks](https://arxiv.org/abs/2104.03961)\n\nEverything sounds good until they basically say FNN are universal function approximaters and argue a 1-hidden layer net can learn the mapping 🤥. They do also offer a few layer deep FNN though.", "votes": null }, { "id": "1502306", "postDate": "09/04/2021 05:53:01", "content": "

Haha, yes everything is a neural network with enough layers and/or neurons I guess. Thanks for sharing the link! 👌

", "rawMarkdown": "Haha, yes everything is a neural network with enough layers and/or neurons I guess. Thanks for sharing the link! 👌", "votes": null }, { "id": "1502307", "postDate": "09/04/2021 05:53:53", "content": "

That looks very interesting as an idea. Have you found any implementations? 👀

", "rawMarkdown": "That looks very interesting as an idea. Have you found any implementations? 👀", "votes": null }, { "id": "1559955", "postDate": "10/27/2021 08:37:10", "content": "

Hey All,

\n

Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.

\n

Thank you all,

\n

Regards,
\nChris

", "rawMarkdown": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris", "votes": null } ], "comments": [ { "id": 1494226, "author_name": "yassinealouini", "author_url": "", "post_date": "08/28/2021 14:02:35", "content": "

Not directly related but I think I finally found an easy service to upload images: https://imgbb.com/. 😄

", "votes": null, "replies": [] }, { "id": 1494228, "author_name": "hengck23", "author_url": "", "post_date": "08/28/2021 14:06:22", "content": "

kinda brute force method. see figure.1 of https://arxiv.org/pdf/1904.08693.pdf
\nyou can google for :\"PyCBC template matching\"

\n

https://colab.research.google.com/github/gw-odw/odw-2020/blob/master/Day_2/Tuto_2.1_Matched_filtering_introduction.ipynb

\n

https://www.atlas.aei.uni-hannover.de/work/ahnitz/projects/tplot/html/workflow/matched_filter.html

\n

it looks something like this:
\nhttps://www.youtube.com/watch?v=bBBDR5jf9oU

\n

there could be about 250k templates? in the match filter bank. it is very computational intensive

", "votes": null, "replies": [ { "id": 1494237, "author_name": "yassinealouini", "author_url": "", "post_date": "08/28/2021 14:13:22", "content": "

Thanks for the links, I will add these. 👌

\n

I have also noticed that PyCBC seems to be THE GW library so there should be something there indeed!

", "votes": null, "replies": [] }, { "id": 1494247, "author_name": "hengck23", "author_url": "", "post_date": "08/28/2021 14:21:17", "content": "

if someone can upload whiten waveform dataset, then we can try 1d convolution net or wavenet model

", "votes": null, "replies": [] } ] }, { "id": 1494319, "author_name": "brachester", "author_url": "", "post_date": "08/28/2021 15:33:19", "content": "

I'm pretty sure the entire point of them holding this competition is to find a more scalable approach for future detection since match filtering is computationally prohibitive. This paper said something similar https://arxiv.org/pdf/1711.03121.pdf

", "votes": null, "replies": [ { "id": 1494340, "author_name": "yassinealouini", "author_url": "", "post_date": "08/28/2021 15:44:43", "content": "

Thanks for the paper. Do you have an idea about the time difference between the two approaches: matched filter vs deep learning model?

", "votes": null, "replies": [] }, { "id": 1494352, "author_name": "brachester", "author_url": "", "post_date": "08/28/2021 15:56:36", "content": "

I'm new to signal processing so I don't know when matched filtering was first proposed but deep learning approach is obviously a recent thing. Matched filtering was at least used in the Nobel prize winning first detection back in 2015 https://arxiv.org/ftp/arxiv/papers/1602/1602.03837.pdf

", "votes": null, "replies": [] }, { "id": 1494387, "author_name": "yassinealouini", "author_url": "", "post_date": "08/28/2021 16:20:39", "content": "

I am a new comer to this field as well. I will keep in mind the time constraint and post an update if I find more details. 👌

", "votes": null, "replies": [] }, { "id": 1495553, "author_name": "authman", "author_url": "", "post_date": "08/29/2021 16:20:43", "content": "

\"\"

", "votes": null, "replies": [] }, { "id": 1495566, "author_name": "authman", "author_url": "", "post_date": "08/29/2021 16:26:52", "content": "
\n

I'm pretty sure the entire point of them holding this competition is to find a more scalable approach for future detection since match filtering is computationally prohibitive

\n
\n

Agreed. MF is literally state of the art and is currently (mathematically) the most robust way for finding signal in noise that swamps it by orders of magnitude. Even more so because it allows for he discovery of the signal's parameters even if they aren't known, so long as the form/equation of the signal is known. But its slowwwww and due to the number of templates needed and the amount of data being generated, I believe the desire here isn't to replace MF entirely, but rather to have us create realtime / near-realtime candidate algorithms, that will then have their proposals fed into the exiting MF systems for proper classification and parameter estimation, etc.

\n

In the context of this competition, if it's discovered that they only used a small subset of templates to generate the GWs and did not vary it much, and if one were able to discover those templates, it theoretically should be very easy to own this thing. lol. I doubt they made that mistake but who knows…. :-)

", "votes": null, "replies": [] }, { "id": 1495572, "author_name": "yassinealouini", "author_url": "", "post_date": "08/29/2021 16:28:44", "content": "

@authman thanks for the graph and indeed, that's a big speed-up! 👌

", "votes": null, "replies": [] } ] }, { "id": 1498281, "author_name": "hengck23", "author_url": "", "post_date": "08/31/2021 20:10:29", "content": "

i have a feeling that modern matched filtering may be accomplished by a transformer.

\n

assuming input is patches of (LIGO Hanford, LIGO Livingston, and Virgo), in self-attention, you can computing similarity (i.e. matching) of the 3 measurements. the network will learn the tempates

\n

you can also use input = (LIGO Hanford, LIGO Livingston, Virgo, other out-of-distribution template)

", "votes": null, "replies": [ { "id": 1502307, "author_name": "yassinealouini", "author_url": "", "post_date": "09/04/2021 05:53:53", "content": "

That looks very interesting as an idea. Have you found any implementations? 👀

", "votes": null, "replies": [] } ] }, { "id": 1502122, "author_name": "authman", "author_url": "", "post_date": "09/03/2021 21:35:20", "content": "

\"\"

\n

Generalized Approach to Matched Filtering using Neural Networks

\n

Everything sounds good until they basically say FNN are universal function approximaters and argue a 1-hidden layer net can learn the mapping 🤥. They do also offer a few layer deep FNN though.

", "votes": null, "replies": [ { "id": 1502306, "author_name": "yassinealouini", "author_url": "", "post_date": "09/04/2021 05:53:01", "content": "

Haha, yes everything is a neural network with enough layers and/or neurons I guess. Thanks for sharing the link! 👌

", "votes": null, "replies": [] } ] }, { "id": 1559955, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 08:37:10", "content": "

Hey All,

\n

Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.

\n

Thank you all,

\n

Regards,
\nChris

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1494206": "[**Mached filter**](https://en.wikipedia.org/wiki/Matched_filter) is a common signal processing method to extract a known signal (called the template) from an unknown signal.\n\nIt seems to be possible here since the GW signal can be modeled using several parameters so it could have an approximate formula, for example the **binary inspiral chirp waveform** (check the details in the screenshot below). \n\"matched-filtering-gw\"\n\n\nIn this [master thesis](https://matheo.uliege.be/bitstream/2268.2/9211/4/Master_thesis_Janquart.pdf), there is a whole chapter (chapter 1) with more details. You can find more details as well in these [slides](https://indico.cern.ch/event/779256/contributions/3242644/attachments/1780516/2896456/dent_ggww_2.pdf).\n\nAs the title of this discussion asks, has anyone tried this method either as a processing step or as a model? Let me know if you have found any useful discussions and/or notebooks, thanks in advance. \n\n\n[EDIT] As suggested by @hengck23, it seems that there is something for matched filtering in the [**PyCBC**](https://pycbc.org/) library: https://www.atlas.aei.uni-hannover.de/work/ahnitz/projects/tplot/html/workflow/matched_filter.html. There is also a colab [notebook](https://colab.research.google.com/github/gw-odw/odw-2020/blob/master/Day_2/Tuto_2.1_Matched_filtering_introduction.ipynb).", "1494226": "Not directly related but I think I finally found an easy service to upload images: https://imgbb.com/. 😄", "1494228": "kinda brute force method. see figure.1 of https://arxiv.org/pdf/1904.08693.pdf\nyou can google for :\"PyCBC template matching\"\n\nhttps://colab.research.google.com/github/gw-odw/odw-2020/blob/master/Day_2/Tuto_2.1_Matched_filtering_introduction.ipynb\n\nhttps://www.atlas.aei.uni-hannover.de/work/ahnitz/projects/tplot/html/workflow/matched_filter.html\n\nit looks something like this:\nhttps://www.youtube.com/watch?v=bBBDR5jf9oU\n\nthere could be about 250k templates? in the match filter bank. it is very computational intensive", "1494237": "Thanks for the links, I will add these. 👌\n\nI have also noticed that [PyCBC](https://pycbc.org/) seems to be THE GW library so there should be something there indeed!", "1494247": "if someone can upload whiten waveform dataset, then we can try 1d convolution net or wavenet model", "1494319": "I'm pretty sure the entire point of them holding this competition is to find a more scalable approach for future detection since match filtering is computationally prohibitive. This paper said something similar https://arxiv.org/pdf/1711.03121.pdf", "1494340": "Thanks for the paper. Do you have an idea about the time difference between the two approaches: matched filter vs deep learning model?", "1494352": "I'm new to signal processing so I don't know when matched filtering was first proposed but deep learning approach is obviously a recent thing. Matched filtering was at least used in the Nobel prize winning first detection back in 2015 https://arxiv.org/ftp/arxiv/papers/1602/1602.03837.pdf", "1494387": "I am a new comer to this field as well. I will keep in mind the time constraint and post an update if I find more details. 👌", "1495553": "![](https://i.imgur.com/ca2eCfK.png)", "1495566": "> I'm pretty sure the entire point of them holding this competition is to find a more scalable approach for future detection since match filtering is computationally prohibitive\n\nAgreed. MF is literally state of the art and is currently (mathematically) the most robust way for finding signal in noise that swamps it by orders of magnitude. Even more so because it allows for he discovery of the signal's parameters even if they aren't known, so long as the form/equation of the signal is known. But its slowwwww and due to the number of templates needed and the amount of data being generated, I believe the desire here isn't to replace MF entirely, but rather to have us create realtime / near-realtime candidate algorithms, that will then have their proposals fed into the exiting MF systems for proper classification and parameter estimation, etc.\n\nIn the context of this competition, if it's discovered that they only used a small subset of templates to generate the GWs and did not vary it much, and if one were able to discover those templates, it theoretically should be very easy to own this thing. lol. I doubt they made that mistake but who knows.... :-)", "1495572": "authman thanks for the graph and indeed, that's a big speed-up! 👌", "1498281": "i have a feeling that modern matched filtering may be accomplished by a transformer.\n\nassuming input is patches of (LIGO Hanford, LIGO Livingston, and Virgo), in self-attention, you can computing similarity (i.e. matching) of the 3 measurements. the network will learn the tempates\n\nyou can also use input = (LIGO Hanford, LIGO Livingston, Virgo, other out-of-distribution template)", "1502122": "![](https://i.imgur.com/2iVbHGc.png)\n\n[Generalized Approach to Matched Filtering using Neural Networks](https://arxiv.org/abs/2104.03961)\n\nEverything sounds good until they basically say FNN are universal function approximaters and argue a 1-hidden layer net can learn the mapping 🤥. They do also offer a few layer deep FNN though.", "1502306": "Haha, yes everything is a neural network with enough layers and/or neurons I guess. Thanks for sharing the link! 👌", "1502307": "That looks very interesting as an idea. Have you found any implementations? 👀", "1559955": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris" }, "source": "meta" }