Welcome to our gravitational wave detection challenge.
\nWe are members of a collaboration of scientists working towards the application of machine learning for gravitational waves and geophysics (G2Net). We are also members of the global scientific collaboration responsible for the detection of gravitational waves, first achieved in 2015. Since that first detection of signals from binary black holes, there have been a number of additional gravitational wave discoveries (summarised here), and in 2017 the Nobel Prize was awarded for these ground-breaking achievements. Scientists in our field are still striving to detect more signals and learn about the universe through Einstein’s “ripples in space-time”, and in fact, a new discovery has recently been announced - signals from 2 systems containing black holes and neutron stars (press release).
\nWe would really appreciate your help in exploring new ways to detect signals and we hope you enjoy the binary black hole challenge that we’ve set for you.
", "messages": [ { "id": "1371075", "postDate": "06/30/2021 16:21:22", "content": "Welcome to our gravitational wave detection challenge.
\nWe are members of a collaboration of scientists working towards the application of machine learning for gravitational waves and geophysics (G2Net). We are also members of the global scientific collaboration responsible for the detection of gravitational waves, first achieved in 2015. Since that first detection of signals from binary black holes, there have been a number of additional gravitational wave discoveries (summarised here), and in 2017 the Nobel Prize was awarded for these ground-breaking achievements. Scientists in our field are still striving to detect more signals and learn about the universe through Einstein’s “ripples in space-time”, and in fact, a new discovery has recently been announced - signals from 2 systems containing black holes and neutron stars (press release).
\nWe would really appreciate your help in exploring new ways to detect signals and we hope you enjoy the binary black hole challenge that we’ve set for you.
", "rawMarkdown": "Welcome to our gravitational wave detection challenge.\n\nWe are members of a collaboration of scientists working towards the application of machine learning for gravitational waves and geophysics ([G2Net](https://www.g2net.eu)). We are also members of the global scientific collaboration responsible for the detection of gravitational waves, first achieved in 2015. Since that first detection of signals from binary black holes, there have been a number of additional gravitational wave discoveries (summarised [here](https://www.ligo.org/science/outreach.php)), and in 2017 the [Nobel Prize](https://www.nobelprize.org/prizes/physics/2017/press-release/) was awarded for these ground-breaking achievements. Scientists in our field are still striving to detect more signals and learn about the universe through Einstein’s “ripples in space-time”, and in fact, a new discovery has recently been announced - signals from 2 systems containing black holes and neutron stars ([press release](https://www.ligo.caltech.edu/news/ligo20210629)). \n\nWe would really appreciate your help in exploring new ways to detect signals and we hope you enjoy the binary black hole challenge that we’ve set for you.", "votes": null }, { "id": "1371123", "postDate": "06/30/2021 17:26:46", "content": "Really excited to detect the ripples through space-time.
", "rawMarkdown": "Really excited to detect the ripples through space-time.", "votes": null }, { "id": "1371189", "postDate": "06/30/2021 18:32:54", "content": "Each time series contains either detector noise or detector noise plus a simulated gravitational wave signal. The task is to identify when a signal is present in the data
\nSorry I didn't get this point, if a data point contains a simulated gravitational wave, does it means all the 3 sources of that data point will have it?
I do not understand your methodology: You use simulated data; which is based on some model of G waves. So the ideal Kaggle entry will be the one that is the inverse of your modelling process. How will that be a realistic and useful filter if it only regurgitates your simulation model and does not include actual 'truthed' measurements ?
", "rawMarkdown": "I do not understand your methodology: You use simulated data; which is based on some model of G waves. So the ideal Kaggle entry will be the one that is the inverse of your modelling process. How will that be a realistic and useful filter if it only regurgitates your simulation model and does not include actual 'truthed' measurements ?", "votes": null }, { "id": "1371475", "postDate": "07/01/2021 03:27:17", "content": "Good day,
\nI hope to contribute to this as I just completed two research projects with IARPA I am a new DS so there’s a learning curve for me studied at Santa Monica computer courses also crowd sourcing experience and a Good judgment Open super forecaster.
\nThanks for the experience
\nBrian
\ncaptbullett
this sounds good and really excited to detect ripples through space time.
", "rawMarkdown": "this sounds good and really excited to detect ripples through space time.", "votes": null }, { "id": "1371541", "postDate": "07/01/2021 04:52:20", "content": "There is an assumption that there must be time shifts. Since the detectors are in different parts of the world. But I still need to read the articles to find out for sure.
", "rawMarkdown": "There is an assumption that there must be time shifts. Since the detectors are in different parts of the world. But I still need to read the articles to find out for sure.", "votes": null }, { "id": "1371662", "postDate": "07/01/2021 06:56:39", "content": "i am not understanding …could you help me how to do this
", "rawMarkdown": "i am not understanding ...could you help me how to do this", "votes": null }, { "id": "1371684", "postDate": "07/01/2021 07:09:01", "content": "I was born in Iran and what is my sin?
\nWith all my heart, I would like to participate in this contest …
Hi!
\nEach data file contains data for three ground-based detectors at different locations on the Earth. When a gravitational wave passes through Earth, it will interact with all of the detectors. So if a data file contains a simulated gravitational wave, it will be present in all three detectors. Though as Pavel alluded to, there will be differences between each detector.
", "rawMarkdown": "Hi!\n\nEach data file contains data for three ground-based detectors at different locations on the Earth. When a gravitational wave passes through Earth, it will interact with all of the detectors. So if a data file contains a simulated gravitational wave, it will be present in all three detectors. Though as Pavel alluded to, there will be differences between each detector.", "votes": null }, { "id": "1371749", "postDate": "07/01/2021 08:08:52", "content": "Its explains everything, Thanks
", "rawMarkdown": "Its explains everything, Thanks", "votes": null }, { "id": "1371753", "postDate": "07/01/2021 08:11:54", "content": "\n\nTexact form of a binary black hole waveform are the masses, sky location, distance, black hole spins, binary orientation angle, gravitational wave polarisation, time of arrival, and phase at coalescence (merger).
\n
Hi!
\nWe trust our gravitational wave models pretty well; they've enabled us to make more than 50 detections to date and they're continuously being improved. So if an algorithm can reliably detect/reconstruct the model we've injected into the detector noise, we'd expect that algorithm to translate reasonably well to real data.
", "rawMarkdown": "Hi!\n\nWe trust our gravitational wave models pretty well; they've enabled us to make more than 50 detections to date and they're continuously being improved. So if an algorithm can reliably detect/reconstruct the model we've injected into the detector noise, we'd expect that algorithm to translate reasonably well to real data.", "votes": null }, { "id": "1371914", "postDate": "07/01/2021 10:00:03", "content": "Hi Jim,
\nIn other words, we have very accurate models based on Einsteins theories of General Relativity. These come in different forms but have been calibrated against incredibly costly numerical relativity simulations. So we are very confident that the signals we have simulated for this challenge are exactly the kind of signal that we will continue to detect in the future. The difficulty here is understanding the noise behaviour and accumulating confidence that a signal is present in the data.
", "rawMarkdown": "Hi Jim, \n\nIn other words, we have very accurate models based on Einsteins theories of General Relativity. These come in different forms but have been calibrated against incredibly costly numerical relativity simulations. So we are very confident that the signals we have simulated for this challenge are exactly the kind of signal that we will continue to detect in the future. The difficulty here is understanding the noise behaviour and accumulating confidence that a signal is present in the data.", "votes": null }, { "id": "1371950", "postDate": "07/01/2021 10:22:15", "content": "This competition is very interesting. As a high school student, I would love to join and learn more about gravitational waves and their underlying principle.
", "rawMarkdown": "This competition is very interesting. As a high school student, I would love to join and learn more about gravitational waves and their underlying principle.", "votes": null }, { "id": "1372056", "postDate": "07/01/2021 11:50:42", "content": "Hii …. I ama new bee in kaggle competition platform , I got stuck in one confusion …
\nEver .npy file contains 3 images from three different LIGO observatories , should I use the three images simultaneously , as an input to my model ?
\nThanks in advance!!
", "rawMarkdown": "Hii .... I ama new bee in kaggle competition platform , I got stuck in one confusion ...\n\nEver .npy file contains 3 images from three different LIGO observatories , should I use the three images simultaneously , as an input to my model ?\n\nThanks in advance!!", "votes": null }, { "id": "1372125", "postDate": "07/01/2021 12:50:26", "content": "Hi Akash,
\nEach .npy file contains 3 different timeseries (not images) from the 2 LIGO observatories and the 1 Virgo observatory. Our detectors are more like cosmic microphones since we don't take images like many other astronomical detectors/telescopes - hence we record a single string of outputs for each detector sampled at 2048Hz for this dataset.
\nIf a signal is present then it will be present in each detector BUT each detector may react differently to a signal depending on where the signal originated from on the sky (plus other factors) so when a signal is present it will appear differently in each detector.
\nSo in response to your question, each detector will provide you with some additional information if a signal is present. It's your choice what to do with that.
\nCheers, Chris
", "rawMarkdown": "Hi Akash,\n\nEach .npy file contains 3 different *timeseries* (not images) from the 2 LIGO observatories and the 1 Virgo observatory. Our detectors are more like cosmic microphones since we don't take images like many other astronomical detectors/telescopes - hence we record a single string of outputs for each detector sampled at 2048Hz for this dataset.\n\nIf a signal is present then it will be present in each detector BUT each detector may react differently to a signal depending on where the signal originated from on the sky (plus other factors) so when a signal is present it will appear differently in each detector.\n\nSo in response to your question, each detector will provide you with *some* additional information if a signal is present. It's your choice what to do with that. \n\nCheers, Chris", "votes": null }, { "id": "1372386", "postDate": "07/01/2021 16:40:05", "content": "Very interesting competition, and looks similar to another ongoing competition - SETI, which is suffering serious leak issue, may I ask have you learned from SETI and ensure there is no leak in this competition? Thanks.
", "rawMarkdown": "Very interesting competition, and looks similar to another ongoing competition - SETI, which is suffering serious leak issue, may I ask have you learned from SETI and ensure there is no leak in this competition? Thanks.", "votes": null }, { "id": "1373134", "postDate": "07/02/2021 08:40:14", "content": "Thank you very much for this precious opportunity to experience astrophysics through data science.🪐
\nI majored in physics at university, but I have not been exposed to it at all since I started working.
\nI would like to enjoy this competition very much😊
I hope that those who have no expertise in astrophysics will also be able to discover the fun of it through this competition!
", "rawMarkdown": "Thank you very much for this precious opportunity to experience astrophysics through data science.🪐\n\nI majored in physics at university, but I have not been exposed to it at all since I started working.\nI would like to enjoy this competition very much😊\n\nI hope that those who have no expertise in astrophysics will also be able to discover the fun of it through this competition!", "votes": null }, { "id": "1373136", "postDate": "07/02/2021 08:41:29", "content": "A very interesting competition. Since the discovery of gravitational waves back in 2015, this has been a topic of interest and research in modern physics. Hope we'll learn a lot from it.
", "rawMarkdown": "A very interesting competition. Since the discovery of gravitational waves back in 2015, this has been a topic of interest and research in modern physics. Hope we'll learn a lot from it.", "votes": null }, { "id": "1373501", "postDate": "07/02/2021 14:22:02", "content": "We were made aware of the potential for leak issues and worked hard with the Kaggle team to avoid that scenario.
\nChris
", "rawMarkdown": "We were made aware of the potential for leak issues and worked hard with the Kaggle team to avoid that scenario.\n\nChris", "votes": null }, { "id": "1374248", "postDate": "07/03/2021 06:39:03", "content": "I understand that these sample include noise or noise + signal, however has the noise been preprocessed?
\nBased on the writeup in GWpy for the 2015 event they filtered out A/C noise at 60 and 120Hz and applied a bandpass filter from 50 to 250hz (i.e., bandpass(50, 250).notch(60).notch(120). This makes sense as the signal from the black hole events should be in this range, i.e., start around 20hz and peak at 250-300hz
\nMy question is has filtering like this been applied to the competition data?
", "rawMarkdown": "I understand that these sample include noise or noise + signal, however has the noise been preprocessed? \n\nBased on the writeup in [GWpy](https://gwpy.github.io/docs/stable/overview.html#) for the 2015 event they filtered out A/C noise at 60 and 120Hz and applied a bandpass filter from 50 to 250hz (i.e., bandpass(50, 250).notch(60).notch(120). This makes sense as the signal from the black hole events should be in this range, i.e., start around 20hz and peak at 250-300hz\n \nMy question is has filtering like this been applied to the competition data?", "votes": null }, { "id": "1374705", "postDate": "07/03/2021 14:29:22", "content": "Good day
\nWhen we talk about noise in this instance could a BIN model help solve this problem as the BIN model is bias information and noise
\nHere’s a paper from Google scholar
\nhttps://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&inst=7311101921447552140&scioq=IARPA+research+projects+BIN+&scilib=1&q=BIN&btnG=#d=gs_qabs&u=%23p%3D4BXLPow-38MJ
Hi,
\nWe wanted to keep the options as open as possible to allow for possible novel approaches, so the competition data has not been pre-processed or filtered in any way.
", "rawMarkdown": "Hi,\n\nWe wanted to keep the options as open as possible to allow for possible novel approaches, so the competition data has not been pre-processed or filtered in any way.", "votes": null }, { "id": "1374760", "postDate": "07/03/2021 15:18:03", "content": "Good day,
\nYes this site is what we want to model as it give one the proper tool sets to do what need to be done on Kaggle:
\nhttps://gwpy.github.io/docs/stable/
Good day someone asked why a 64 float is being used instead of a 16 floating number
\nAnswer: (over fitting) is what I understand is the reason now you can read about it …
B.
", "rawMarkdown": "Good day someone asked why a 64 float is being used instead of a 16 floating number \nAnswer: (over fitting) is what I understand is the reason now you can read about it …\n\nhttps://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53\n\nB.", "votes": null }, { "id": "1376260", "postDate": "07/05/2021 02:13:08", "content": "Good article for a different perspective
\nhttps://www.scientificamerican.com/article/black-holes-quantum-entanglement-and-the-no-go-theorem/?fbclid=IwAR3WOYCW142VhGeJefWaqt4jPbAITk773U7xddIrs3zLYhKRPvUUn9XUmPw
Dear @bayeswolf ,
\nMay I ask what is the meaning of the subfolders name under train and test directory, like input\\g2net-gravitational-wave-detection\\train\\1\\1\\3\\113a205592.npy, why it's organized by 3 levels subfolders and each level's range from 0 to f, and looks all *.npy files' name start with such subfolders' name.
", "rawMarkdown": "Dear @bayeswolf , \n\nMay I ask what is the meaning of the subfolders name under train and test directory, like input\\g2net-gravitational-wave-detection\\train\\1\\1\\3\\113a205592.npy, why it's organized by 3 levels subfolders and each level's range from 0 to f, and looks all *.npy files' name start with such subfolders' name.", "votes": null }, { "id": "1381386", "postDate": "07/09/2021 00:03:45", "content": "The competition is just starting, but I wonder how far we can go. Have you tried your detection pipelines on this data to see how good it gets? If you can't share, then, as a proxy, can you tell us the level of performance that would make you happy?
", "rawMarkdown": "The competition is just starting, but I wonder how far we can go. Have you tried your detection pipelines on this data to see how good it gets? If you can't share, then, as a proxy, can you tell us the level of performance that would make you happy?", "votes": null }, { "id": "1381795", "postDate": "07/09/2021 08:19:11", "content": "This method has been used in other competitions like the Bristol-Myers Squibb - Molecular Translation. The thing is that the data has many files (560.000 train set and 226.000 test set), so having all the files in a single folder can hinder performance. That's why the files are placed in different subfolders, and the id of the file encodes the subfolder the file is in.
\nFor example, in the aforementioned competition, I used the following method to get the path of a file from its identifier, and the same method could be used in this competition, as the number of subfolder levels is also three:
\n\ndef get_path(basedir, file_id):\n return os.path.join(basedir, file_id[0], file_id[1], file_id[2], file_id+ '.npy' )\n
Thanks for your explanation @jcesquiveld
", "rawMarkdown": "Thanks for your explanation @jcesquiveld", "votes": null }, { "id": "1382332", "postDate": "07/09/2021 17:44:37", "content": "Hi. Just a simple question. From your problem description it seems that the 'strain' is basically a dimensionless amplitude, is this correct?
", "rawMarkdown": "Hi. Just a simple question. From your problem description it seems that the 'strain' is basically a dimensionless amplitude, is this correct?", "votes": null }, { "id": "1386537", "postDate": "07/13/2021 14:36:55", "content": "We haven't explicitly tried our existing analyses on the challenge data but we will ultimately like to do so when the challenge is complete. We haven't done this blind though - we do have a good understanding of what is detectable and what isn't and the dataset was designed to be a difficult challenge. All I can say is that we are happy with the current scores and/but we would love it if the Kaggle winners were able to outdo our standard approaches. Sorry to be so vague.
", "rawMarkdown": "We haven't explicitly tried our existing analyses on the challenge data but we will ultimately like to do so when the challenge is complete. We haven't done this blind though - we do have a good understanding of what is detectable and what isn't and the dataset was designed to be a difficult challenge. All I can say is that we are happy with the current scores and/but we would love it if the Kaggle winners were able to outdo our standard approaches. Sorry to be so vague.", "votes": null }, { "id": "1386617", "postDate": "07/13/2021 15:24:24", "content": "Thanks for the response. I hope you'll be happy with the competition end results.
", "rawMarkdown": "Thanks for the response. I hope you'll be happy with the competition end results.", "votes": null }, { "id": "1387555", "postDate": "07/14/2021 09:11:00", "content": "Hi Alexander,
\nYes, strain is defined as twice the fractional change in the interferometer arm length (so it's a ratio of lengths) and hence dimensionless.
\nCheers, Chris
", "rawMarkdown": "Hi Alexander, \n\nYes, strain is defined as twice the fractional change in the interferometer arm length (so it's a ratio of lengths) and hence dimensionless.\n\nCheers, Chris", "votes": null }, { "id": "1396163", "postDate": "07/21/2021 21:09:03", "content": "Hey,
\nI'm sorry, if this is a detail that`s supossed to remain disclosed, but i was wondering how high the approximate SNR of this dataset is?
Hi, there isn't a single SNR that applies to all the data but rather a continuous distribution. I can't give you the exact numbers, but the 15 parameters for the signals were sampled such that the distribution of SNR included a range of values from quiet, hard to detect signals; up to some very loud signals, which may even be visible by the eye (see for example this post).
", "rawMarkdown": "Hi, there isn't a single SNR that applies to all the data but rather a continuous distribution. I can't give you the exact numbers, but the 15 parameters for the signals were sampled such that the distribution of SNR included a range of values from quiet, hard to detect signals; up to some very loud signals, which may even be visible by the eye (see for example [this post](https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/253619#1393179)).", "votes": null }, { "id": "1405300", "postDate": "07/30/2021 17:19:44", "content": "Greetings.
\nI have a legal question about the rules.
\nMy nationality and citizenship happen to Iran, which is included in the list of US-based sanctions. However, I am not residing there. I am a graduate student residing in Turkey, with a legal address, bank account, student residence permit, etc. all affiliated with Turkey, which is not included in the US-based sanctions list. I was wondering about my eligibility in joining the competition. Since I am not residing in Iran and I am not a representative of any of its affiliations whatsoever, my thoughts were that I am eligible. I want to be sure, however.
\nThank you in advance.
", "rawMarkdown": "Greetings.\n\nI have a legal question about the rules.\n\nMy nationality and citizenship happen to Iran, which is included in the list of US-based sanctions. However, I am not residing there. I am a graduate student residing in Turkey, with a legal address, bank account, student residence permit, etc. all affiliated with Turkey, which is not included in the US-based sanctions list. I was wondering about my eligibility in joining the competition. Since I am not residing in Iran and I am not a representative of any of its affiliations whatsoever, my thoughts were that I am eligible. I want to be sure, however.\n\nThank you in advance.", "votes": null }, { "id": "1405465", "postDate": "07/30/2021 22:06:16", "content": "Such an interesting Competition to apply Data Science to solve real world problems!
", "rawMarkdown": "Such an interesting Competition to apply Data Science to solve real world problems!", "votes": null }, { "id": "1405508", "postDate": "07/31/2021 00:32:41", "content": "\nThank you guys for hosting this competition - very interesting problem indeed.
\nI have one question for you guys.
Given a data point from the training data, how would one approach separating signal from the noise through conventional methods? Any references would be helpful. I think it would be a great help to everyone if you guys can create a sample notebook dissecting a signal from the data for a particular ID.
", "rawMarkdown": "michaeljwill @bayeswolf \n\nThank you guys for hosting this competition - very interesting problem indeed. \nI have one question for you guys. \n\nGiven a data point from the training data, how would one approach separating signal from the noise through conventional methods? Any references would be helpful. I think it would be a great help to everyone if you guys can create a sample notebook dissecting a signal from the data for a particular ID.", "votes": null }, { "id": "1446066", "postDate": "08/04/2021 10:56:59", "content": "Thanks for this challenge! Hopefully I would be able to contribute something of use.
", "rawMarkdown": "Thanks for this challenge! Hopefully I would be able to contribute something of use.", "votes": null }, { "id": "1479714", "postDate": "08/18/2021 15:46:01", "content": "Hi Pouya,
\nI am not familiar with the legal aspects of the competition rules and I advise that you contact Kaggle directly to get an answer to your question.
\nCheers, Chris
", "rawMarkdown": "Hi Pouya,\n\nI am not familiar with the legal aspects of the competition rules and I advise that you contact Kaggle directly to get an answer to your question.\n\nCheers, Chris", "votes": null }, { "id": "1485267", "postDate": "08/21/2021 23:32:42", "content": "For the simulated noise, are they supposed to be close to the real noise of the detectors? Are the noise from each detector simulated from the same distribution?
", "rawMarkdown": "For the simulated noise, are they supposed to be close to the real noise of the detectors? Are the noise from each detector simulated from the same distribution?", "votes": null }, { "id": "1502553", "postDate": "09/04/2021 11:45:15", "content": "Fascinating challenge! An opportunity to learn more about gravitational waves. I hope to contribute.
", "rawMarkdown": "Fascinating challenge! An opportunity to learn more about gravitational waves. I hope to contribute.", "votes": null }, { "id": "1507966", "postDate": "09/09/2021 17:34:26", "content": "\n\nIn this competition you are provided with a training set of time series data containing simulated gravitational wave measurements from a network of 3 gravitational wave interferometers (LIGO Hanford, LIGO Livingston, and Virgo). Each time series contains either detector noise or detector noise plus a simulated gravitational wave signal.
\n
Does this mean, in a sample, if there is a signal, all of the waves will contain the signal or just one of the waves will contain the signal?
", "rawMarkdown": "> In this competition you are provided with a training set of time series data containing simulated gravitational wave measurements from a network of 3 gravitational wave interferometers (LIGO Hanford, LIGO Livingston, and Virgo). Each time series contains either detector noise or detector noise plus a simulated gravitational wave signal.\n\nDoes this mean, in a sample, if there is a signal, all of the waves will contain the signal or **just one** of the waves will contain the signal?", "votes": null }, { "id": "1508027", "postDate": "09/09/2021 18:31:46", "content": "Hi, this means that for a single sample (.npy file) if it contains a signal, the signal will be present in all three of the detectors. However, the signal will vary slightly between detectors because of their different characteristics and locations on the Earth.
\nHope this helps.
", "rawMarkdown": "Hi, this means that for a single sample (.npy file) if it contains a signal, the signal will be present in all three of the detectors. However, the signal will vary slightly between detectors because of their different characteristics and locations on the Earth.\n\nHope this helps.", "votes": null }, { "id": "1530691", "postDate": "10/01/2021 10:48:15", "content": "Hi everyone,
\nFirst of all, congratulations to the winners! We are very excited about following up and finding out how you all did so well in this challenge. The challenge deadline unfortunately coincides with the start of teaching here in the UK so I'm personally a bit distracted right now, BUT, we are meeting up with the Kaggle team soon to discuss what happens next.
\nMany thanks again to everyone that took part. We really hope that you found the idea of searching for gravitational waves from black holes as interesting as we do.
\nCheers
\nChris
", "rawMarkdown": "Hi everyone,\n\nFirst of all, congratulations to the winners! We are very excited about following up and finding out how you all did so well in this challenge. The challenge deadline unfortunately coincides with the start of teaching here in the UK so I'm personally a bit distracted right now, BUT, we are meeting up with the Kaggle team soon to discuss what happens next.\n\nMany thanks again to everyone that took part. We really hope that you found the idea of searching for gravitational waves from black holes as interesting as we do.\n\nCheers\n\nChris", "votes": null }, { "id": "1531095", "postDate": "10/01/2021 16:03:35", "content": "Is there a hope to start a new G2Net competition too? :)
\nEven right tomorrow? :D
Hey,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
Really excited to detect the ripples through space-time.
", "votes": null, "replies": [] }, { "id": 1371189, "author_name": "mrinath", "author_url": "", "post_date": "06/30/2021 18:32:54", "content": "Each time series contains either detector noise or detector noise plus a simulated gravitational wave signal. The task is to identify when a signal is present in the data
\nSorry I didn't get this point, if a data point contains a simulated gravitational wave, does it means all the 3 sources of that data point will have it?
There is an assumption that there must be time shifts. Since the detectors are in different parts of the world. But I still need to read the articles to find out for sure.
", "votes": null, "replies": [] }, { "id": 1371719, "author_name": "michaeljwill", "author_url": "", "post_date": "07/01/2021 07:41:35", "content": "Hi!
\nEach data file contains data for three ground-based detectors at different locations on the Earth. When a gravitational wave passes through Earth, it will interact with all of the detectors. So if a data file contains a simulated gravitational wave, it will be present in all three detectors. Though as Pavel alluded to, there will be differences between each detector.
", "votes": null, "replies": [] }, { "id": 1371749, "author_name": "mrinath", "author_url": "", "post_date": "07/01/2021 08:08:52", "content": "Its explains everything, Thanks
", "votes": null, "replies": [] } ] }, { "id": 1371278, "author_name": "jmorgie", "author_url": "", "post_date": "06/30/2021 20:19:27", "content": "I do not understand your methodology: You use simulated data; which is based on some model of G waves. So the ideal Kaggle entry will be the one that is the inverse of your modelling process. How will that be a realistic and useful filter if it only regurgitates your simulation model and does not include actual 'truthed' measurements ?
", "votes": null, "replies": [ { "id": 1371753, "author_name": "shtrausslearning", "author_url": "", "post_date": "07/01/2021 08:11:54", "content": "\n\nTexact form of a binary black hole waveform are the masses, sky location, distance, black hole spins, binary orientation angle, gravitational wave polarisation, time of arrival, and phase at coalescence (merger).
\n
Hi!
\nWe trust our gravitational wave models pretty well; they've enabled us to make more than 50 detections to date and they're continuously being improved. So if an algorithm can reliably detect/reconstruct the model we've injected into the detector noise, we'd expect that algorithm to translate reasonably well to real data.
", "votes": null, "replies": [] }, { "id": 1371914, "author_name": "bayeswolf", "author_url": "", "post_date": "07/01/2021 10:00:03", "content": "Hi Jim,
\nIn other words, we have very accurate models based on Einsteins theories of General Relativity. These come in different forms but have been calibrated against incredibly costly numerical relativity simulations. So we are very confident that the signals we have simulated for this challenge are exactly the kind of signal that we will continue to detect in the future. The difficulty here is understanding the noise behaviour and accumulating confidence that a signal is present in the data.
", "votes": null, "replies": [] } ] }, { "id": 1371475, "author_name": "captbullett", "author_url": "", "post_date": "07/01/2021 03:27:17", "content": "Good day,
\nI hope to contribute to this as I just completed two research projects with IARPA I am a new DS so there’s a learning curve for me studied at Santa Monica computer courses also crowd sourcing experience and a Good judgment Open super forecaster.
\nThanks for the experience
\nBrian
\ncaptbullett
this sounds good and really excited to detect ripples through space time.
", "votes": null, "replies": [] }, { "id": 1371662, "author_name": "seolatha", "author_url": "", "post_date": "07/01/2021 06:56:39", "content": "i am not understanding …could you help me how to do this
", "votes": null, "replies": [] }, { "id": 1371684, "author_name": "yasamanyektaeian", "author_url": "", "post_date": "07/01/2021 07:09:01", "content": "I was born in Iran and what is my sin?
\nWith all my heart, I would like to participate in this contest …
This competition is very interesting. As a high school student, I would love to join and learn more about gravitational waves and their underlying principle.
", "votes": null, "replies": [] }, { "id": 1372056, "author_name": "zemansky", "author_url": "", "post_date": "07/01/2021 11:50:42", "content": "Hii …. I ama new bee in kaggle competition platform , I got stuck in one confusion …
\nEver .npy file contains 3 images from three different LIGO observatories , should I use the three images simultaneously , as an input to my model ?
\nThanks in advance!!
", "votes": null, "replies": [ { "id": 1372125, "author_name": "bayeswolf", "author_url": "", "post_date": "07/01/2021 12:50:26", "content": "Hi Akash,
\nEach .npy file contains 3 different timeseries (not images) from the 2 LIGO observatories and the 1 Virgo observatory. Our detectors are more like cosmic microphones since we don't take images like many other astronomical detectors/telescopes - hence we record a single string of outputs for each detector sampled at 2048Hz for this dataset.
\nIf a signal is present then it will be present in each detector BUT each detector may react differently to a signal depending on where the signal originated from on the sky (plus other factors) so when a signal is present it will appear differently in each detector.
\nSo in response to your question, each detector will provide you with some additional information if a signal is present. It's your choice what to do with that.
\nCheers, Chris
", "votes": null, "replies": [] } ] }, { "id": 1372386, "author_name": "superchenhao", "author_url": "", "post_date": "07/01/2021 16:40:05", "content": "Very interesting competition, and looks similar to another ongoing competition - SETI, which is suffering serious leak issue, may I ask have you learned from SETI and ensure there is no leak in this competition? Thanks.
", "votes": null, "replies": [ { "id": 1373501, "author_name": "bayeswolf", "author_url": "", "post_date": "07/02/2021 14:22:02", "content": "We were made aware of the potential for leak issues and worked hard with the Kaggle team to avoid that scenario.
\nChris
", "votes": null, "replies": [] } ] }, { "id": 1373134, "author_name": "yhirakawa", "author_url": "", "post_date": "07/02/2021 08:40:14", "content": "Thank you very much for this precious opportunity to experience astrophysics through data science.🪐
\nI majored in physics at university, but I have not been exposed to it at all since I started working.
\nI would like to enjoy this competition very much😊
I hope that those who have no expertise in astrophysics will also be able to discover the fun of it through this competition!
", "votes": null, "replies": [] }, { "id": 1373136, "author_name": "azazurrehmanbutt", "author_url": "", "post_date": "07/02/2021 08:41:29", "content": "A very interesting competition. Since the discovery of gravitational waves back in 2015, this has been a topic of interest and research in modern physics. Hope we'll learn a lot from it.
", "votes": null, "replies": [] }, { "id": 1374248, "author_name": "kevinmcisaac", "author_url": "", "post_date": "07/03/2021 06:39:03", "content": "I understand that these sample include noise or noise + signal, however has the noise been preprocessed?
\nBased on the writeup in GWpy for the 2015 event they filtered out A/C noise at 60 and 120Hz and applied a bandpass filter from 50 to 250hz (i.e., bandpass(50, 250).notch(60).notch(120). This makes sense as the signal from the black hole events should be in this range, i.e., start around 20hz and peak at 250-300hz
\nMy question is has filtering like this been applied to the competition data?
", "votes": null, "replies": [ { "id": 1374747, "author_name": "michaeljwill", "author_url": "", "post_date": "07/03/2021 15:04:07", "content": "Hi,
\nWe wanted to keep the options as open as possible to allow for possible novel approaches, so the competition data has not been pre-processed or filtered in any way.
", "votes": null, "replies": [] } ] }, { "id": 1374705, "author_name": "captbullett", "author_url": "", "post_date": "07/03/2021 14:29:22", "content": "Good day
\nWhen we talk about noise in this instance could a BIN model help solve this problem as the BIN model is bias information and noise
\nHere’s a paper from Google scholar
\nhttps://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&inst=7311101921447552140&scioq=IARPA+research+projects+BIN+&scilib=1&q=BIN&btnG=#d=gs_qabs&u=%23p%3D4BXLPow-38MJ
Good day,
\nYes this site is what we want to model as it give one the proper tool sets to do what need to be done on Kaggle:
\nhttps://gwpy.github.io/docs/stable/
Good day someone asked why a 64 float is being used instead of a 16 floating number
\nAnswer: (over fitting) is what I understand is the reason now you can read about it …
B.
", "votes": null, "replies": [] }, { "id": 1376260, "author_name": "captbullett", "author_url": "", "post_date": "07/05/2021 02:13:08", "content": "Good article for a different perspective
\nhttps://www.scientificamerican.com/article/black-holes-quantum-entanglement-and-the-no-go-theorem/?fbclid=IwAR3WOYCW142VhGeJefWaqt4jPbAITk773U7xddIrs3zLYhKRPvUUn9XUmPw
Dear @bayeswolf ,
\nMay I ask what is the meaning of the subfolders name under train and test directory, like input\\g2net-gravitational-wave-detection\\train\\1\\1\\3\\113a205592.npy, why it's organized by 3 levels subfolders and each level's range from 0 to f, and looks all *.npy files' name start with such subfolders' name.
", "votes": null, "replies": [ { "id": 1381795, "author_name": "jcesquiveld", "author_url": "", "post_date": "07/09/2021 08:19:11", "content": "This method has been used in other competitions like the Bristol-Myers Squibb - Molecular Translation. The thing is that the data has many files (560.000 train set and 226.000 test set), so having all the files in a single folder can hinder performance. That's why the files are placed in different subfolders, and the id of the file encodes the subfolder the file is in.
\nFor example, in the aforementioned competition, I used the following method to get the path of a file from its identifier, and the same method could be used in this competition, as the number of subfolder levels is also three:
\n\ndef get_path(basedir, file_id):\n return os.path.join(basedir, file_id[0], file_id[1], file_id[2], file_id+ '.npy' )\n
Thanks for your explanation @jcesquiveld
", "votes": null, "replies": [] } ] }, { "id": 1381386, "author_name": "cpmpml", "author_url": "", "post_date": "07/09/2021 00:03:45", "content": "The competition is just starting, but I wonder how far we can go. Have you tried your detection pipelines on this data to see how good it gets? If you can't share, then, as a proxy, can you tell us the level of performance that would make you happy?
", "votes": null, "replies": [ { "id": 1386537, "author_name": "bayeswolf", "author_url": "", "post_date": "07/13/2021 14:36:55", "content": "We haven't explicitly tried our existing analyses on the challenge data but we will ultimately like to do so when the challenge is complete. We haven't done this blind though - we do have a good understanding of what is detectable and what isn't and the dataset was designed to be a difficult challenge. All I can say is that we are happy with the current scores and/but we would love it if the Kaggle winners were able to outdo our standard approaches. Sorry to be so vague.
", "votes": null, "replies": [] }, { "id": 1386617, "author_name": "cpmpml", "author_url": "", "post_date": "07/13/2021 15:24:24", "content": "Thanks for the response. I hope you'll be happy with the competition end results.
", "votes": null, "replies": [] } ] }, { "id": 1382332, "author_name": "aramos", "author_url": "", "post_date": "07/09/2021 17:44:37", "content": "Hi. Just a simple question. From your problem description it seems that the 'strain' is basically a dimensionless amplitude, is this correct?
", "votes": null, "replies": [ { "id": 1387555, "author_name": "bayeswolf", "author_url": "", "post_date": "07/14/2021 09:11:00", "content": "Hi Alexander,
\nYes, strain is defined as twice the fractional change in the interferometer arm length (so it's a ratio of lengths) and hence dimensionless.
\nCheers, Chris
", "votes": null, "replies": [] } ] }, { "id": 1396163, "author_name": "maxdner", "author_url": "", "post_date": "07/21/2021 21:09:03", "content": "Hey,
\nI'm sorry, if this is a detail that`s supossed to remain disclosed, but i was wondering how high the approximate SNR of this dataset is?
Hi, there isn't a single SNR that applies to all the data but rather a continuous distribution. I can't give you the exact numbers, but the 15 parameters for the signals were sampled such that the distribution of SNR included a range of values from quiet, hard to detect signals; up to some very loud signals, which may even be visible by the eye (see for example this post).
", "votes": null, "replies": [] } ] }, { "id": 1405300, "author_name": "pniaz20", "author_url": "", "post_date": "07/30/2021 17:19:44", "content": "Greetings.
\nI have a legal question about the rules.
\nMy nationality and citizenship happen to Iran, which is included in the list of US-based sanctions. However, I am not residing there. I am a graduate student residing in Turkey, with a legal address, bank account, student residence permit, etc. all affiliated with Turkey, which is not included in the US-based sanctions list. I was wondering about my eligibility in joining the competition. Since I am not residing in Iran and I am not a representative of any of its affiliations whatsoever, my thoughts were that I am eligible. I want to be sure, however.
\nThank you in advance.
", "votes": null, "replies": [ { "id": 1479714, "author_name": "bayeswolf", "author_url": "", "post_date": "08/18/2021 15:46:01", "content": "Hi Pouya,
\nI am not familiar with the legal aspects of the competition rules and I advise that you contact Kaggle directly to get an answer to your question.
\nCheers, Chris
", "votes": null, "replies": [] } ] }, { "id": 1405465, "author_name": "safarimoubarek", "author_url": "", "post_date": "07/30/2021 22:06:16", "content": "Such an interesting Competition to apply Data Science to solve real world problems!
", "votes": null, "replies": [] }, { "id": 1405508, "author_name": "harshpatel1692", "author_url": "", "post_date": "07/31/2021 00:32:41", "content": "\nThank you guys for hosting this competition - very interesting problem indeed.
\nI have one question for you guys.
Given a data point from the training data, how would one approach separating signal from the noise through conventional methods? Any references would be helpful. I think it would be a great help to everyone if you guys can create a sample notebook dissecting a signal from the data for a particular ID.
", "votes": null, "replies": [] }, { "id": 1446066, "author_name": "yassinealouini", "author_url": "", "post_date": "08/04/2021 10:56:59", "content": "Thanks for this challenge! Hopefully I would be able to contribute something of use.
", "votes": null, "replies": [] }, { "id": 1485267, "author_name": "brachester", "author_url": "", "post_date": "08/21/2021 23:32:42", "content": "For the simulated noise, are they supposed to be close to the real noise of the detectors? Are the noise from each detector simulated from the same distribution?
", "votes": null, "replies": [] }, { "id": 1502553, "author_name": "juanvallalta", "author_url": "", "post_date": "09/04/2021 11:45:15", "content": "Fascinating challenge! An opportunity to learn more about gravitational waves. I hope to contribute.
", "votes": null, "replies": [] }, { "id": 1507966, "author_name": "mushfirat", "author_url": "", "post_date": "09/09/2021 17:34:26", "content": "\n\nIn this competition you are provided with a training set of time series data containing simulated gravitational wave measurements from a network of 3 gravitational wave interferometers (LIGO Hanford, LIGO Livingston, and Virgo). Each time series contains either detector noise or detector noise plus a simulated gravitational wave signal.
\n
Does this mean, in a sample, if there is a signal, all of the waves will contain the signal or just one of the waves will contain the signal?
", "votes": null, "replies": [ { "id": 1508027, "author_name": "michaeljwill", "author_url": "", "post_date": "09/09/2021 18:31:46", "content": "Hi, this means that for a single sample (.npy file) if it contains a signal, the signal will be present in all three of the detectors. However, the signal will vary slightly between detectors because of their different characteristics and locations on the Earth.
\nHope this helps.
", "votes": null, "replies": [] } ] }, { "id": 1530691, "author_name": "bayeswolf", "author_url": "", "post_date": "10/01/2021 10:48:15", "content": "Hi everyone,
\nFirst of all, congratulations to the winners! We are very excited about following up and finding out how you all did so well in this challenge. The challenge deadline unfortunately coincides with the start of teaching here in the UK so I'm personally a bit distracted right now, BUT, we are meeting up with the Kaggle team soon to discuss what happens next.
\nMany thanks again to everyone that took part. We really hope that you found the idea of searching for gravitational waves from black holes as interesting as we do.
\nCheers
\nChris
", "votes": null, "replies": [ { "id": 1531095, "author_name": "killimi", "author_url": "", "post_date": "10/01/2021 16:03:35", "content": "Is there a hope to start a new G2Net competition too? :)
\nEven right tomorrow? :D
Hey,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris