Hi all ,
\nToday the SETI competition ended , and one of the magic used there was removing background to increase signal to noise ratio .
The removing of background noise was possible because the same background image, by itself and the image(with same background) into which alien signal was injected , were present in the training set at the same time.
\nMore can be read about it here
\nI have not gone through the data yet but I wonder if that kind of reverse engineering is possible with this data as well ,as this also contains simulated data and isn't a code competition
", "messages": [ { "id": "1481626", "postDate": "08/19/2021 15:52:52", "content": "Hi all ,
\nToday the SETI competition ended , and one of the magic used there was removing background to increase signal to noise ratio .
The removing of background noise was possible because the same background image, by itself and the image(with same background) into which alien signal was injected , were present in the training set at the same time.
\nMore can be read about it here
\nI have not gone through the data yet but I wonder if that kind of reverse engineering is possible with this data as well ,as this also contains simulated data and isn't a code competition
", "rawMarkdown": "Hi all , \nToday the SETI competition ended , and one of the magic used there was removing background to increase signal to noise ratio . \n\n The removing of background noise was possible because the same background image, by itself and the image(with same background) into which alien signal was injected , were present in the training set at the same time. \n\nMore can be read about it [here](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/266385)\n\nI have not gone through the data yet but I wonder if that kind of reverse engineering is possible with this data as well ,as this also contains simulated data and isn't a code competition", "votes": null }, { "id": "1482687", "postDate": "08/20/2021 07:36:56", "content": "You can also read here and try it out for yourself by modifying the code in this notebook.
\nHowever, I doubt that it will work here (though I will check it of course). The data in this competition is fully synthetic, including the background, and it is given as a time series. Unless they messed up the random seeds, there shouldn't be repeated data anywhere.
", "rawMarkdown": "You can also read [here](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/266513) and try it out for yourself by modifying the code [in this notebook](https://www.kaggle.com/friedchips/magic-2-an-explanation).\n\nHowever, I doubt that it will work here (though I will check it of course). The data in this competition is fully synthetic, including the background, and it is given as a time series. Unless they messed up the random seeds, there shouldn't be repeated data anywhere.", "votes": null }, { "id": "1482751", "postDate": "08/20/2021 08:45:02", "content": "Hi @friedchips thanks for the comment and the notebook , its really a nice one regarding the explanation. I was going through the LIGO website and found a paper where they discuss on how they separate out the noise and the GW waves , more can be read about it here
", "rawMarkdown": "Hi @friedchips thanks for the comment and the notebook , its really a nice one regarding the explanation. I was going through the LIGO website and found a paper where they discuss on how they separate out the noise and the GW waves , more can be read about it [here](https://arxiv.org/pdf/1908.11170.pdf)", "votes": null }, { "id": "1483027", "postDate": "08/20/2021 12:04:14", "content": "There is a notebook here where you can generate clean GW signals. Combining with noise (i.e. samples where target=0) allows more data generation
", "rawMarkdown": "There is a notebook [here](https://www.kaggle.com/mistag/reverse-engineering-create-clean-gw-signals/execution) where you can generate clean GW signals. Combining with noise (i.e. samples where target=0) allows more data generation", "votes": null }, { "id": "1483544", "postDate": "08/20/2021 17:37:50", "content": "Great notebook @anjum48 thanks for sharing it here
", "rawMarkdown": "Great notebook @anjum48 thanks for sharing it here", "votes": null }, { "id": "1483546", "postDate": "08/20/2021 17:38:10", "content": "The SETI data was real (observed) background data with a simulated injected signal. (We have no idea what an intelligent ET signal would be, so it's more about finding anomalies). In this case, we do know what a gravitational wave would look like, and here the data is 100% simulated, so there's no chance of background leakage like in the SETI.
\nBut, sure, there's always a possibility that the generation process can be reversed. 😃
", "rawMarkdown": "The SETI data was real (observed) background data with a simulated injected signal. (We have no idea what an intelligent ET signal would be, so it's more about finding anomalies). In this case, we *do* know what a gravitational wave would look like, and here the data is 100% simulated, so there's no chance of background leakage like in the SETI. \n\nBut, sure, there's always a possibility that the generation process can be reversed. 😃", "votes": null }, { "id": "1484480", "postDate": "08/21/2021 10:53:42", "content": "@tanulsingh077 https://www.kaggle.com/mistag/reverse-engineering-create-clean-gw-signals in this notebook he made this
", "rawMarkdown": "tanulsingh077 https://www.kaggle.com/mistag/reverse-engineering-create-clean-gw-signals in this notebook he made this", "votes": null }, { "id": "1484699", "postDate": "08/21/2021 14:11:02", "content": "Thanks for the clarification @inversion , as always the great kaggle community will come with creative solutions I am sure
", "rawMarkdown": "Thanks for the clarification @inversion , as always the great kaggle community will come with creative solutions I am sure", "votes": null }, { "id": "1495350", "postDate": "08/29/2021 13:44:04", "content": "I remember there was a competition (Ion Switching) with simulated data where the simulation code was found and used. 😄
\nHere is the first place solution: https://www.kaggle.com/c/liverpool-ion-switching/discussion/153940.
", "rawMarkdown": "I remember there was a competition (Ion Switching) with simulated data where the simulation code was found and used. 😄\n\nHere is the first place solution: https://www.kaggle.com/c/liverpool-ion-switching/discussion/153940.", "votes": null }, { "id": "1561065", "postDate": "10/27/2021 09:53:53", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
You can also read here and try it out for yourself by modifying the code in this notebook.
\nHowever, I doubt that it will work here (though I will check it of course). The data in this competition is fully synthetic, including the background, and it is given as a time series. Unless they messed up the random seeds, there shouldn't be repeated data anywhere.
", "votes": null, "replies": [ { "id": 1482751, "author_name": "tanulsingh077", "author_url": "", "post_date": "08/20/2021 08:45:02", "content": "Hi @friedchips thanks for the comment and the notebook , its really a nice one regarding the explanation. I was going through the LIGO website and found a paper where they discuss on how they separate out the noise and the GW waves , more can be read about it here
", "votes": null, "replies": [] } ] }, { "id": 1483027, "author_name": "anjum48", "author_url": "", "post_date": "08/20/2021 12:04:14", "content": "There is a notebook here where you can generate clean GW signals. Combining with noise (i.e. samples where target=0) allows more data generation
", "votes": null, "replies": [ { "id": 1483544, "author_name": "tanulsingh077", "author_url": "", "post_date": "08/20/2021 17:37:50", "content": "Great notebook @anjum48 thanks for sharing it here
", "votes": null, "replies": [] } ] }, { "id": 1483546, "author_name": "inversion", "author_url": "", "post_date": "08/20/2021 17:38:10", "content": "The SETI data was real (observed) background data with a simulated injected signal. (We have no idea what an intelligent ET signal would be, so it's more about finding anomalies). In this case, we do know what a gravitational wave would look like, and here the data is 100% simulated, so there's no chance of background leakage like in the SETI.
\nBut, sure, there's always a possibility that the generation process can be reversed. 😃
", "votes": null, "replies": [ { "id": 1484699, "author_name": "tanulsingh077", "author_url": "", "post_date": "08/21/2021 14:11:02", "content": "Thanks for the clarification @inversion , as always the great kaggle community will come with creative solutions I am sure
", "votes": null, "replies": [] }, { "id": 1495350, "author_name": "yassinealouini", "author_url": "", "post_date": "08/29/2021 13:44:04", "content": "I remember there was a competition (Ion Switching) with simulated data where the simulation code was found and used. 😄
\nHere is the first place solution: https://www.kaggle.com/c/liverpool-ion-switching/discussion/153940.
", "votes": null, "replies": [] } ] }, { "id": 1484480, "author_name": "estebansnchezgmez", "author_url": "", "post_date": "08/21/2021 10:53:42", "content": "@tanulsingh077 https://www.kaggle.com/mistag/reverse-engineering-create-clean-gw-signals in this notebook he made this
", "votes": null, "replies": [] }, { "id": 1561065, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 09:53:53", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris