\n
![\"\"](\"https://i.imgur.com/kN9IbMg.png\")
\nFor the unknown sources (i.e. things that were not designed for), e.g. AC current, disturbance from transport etc, this is where notch and bandpass filters come in.
I think that a lot of people found the saga around whitening frustrating, so I wanted to share my views on it (our full 3rd place solution will be coming soon)
\nMy interpretation of the intent of whitening is to normalise the amplitude of different frequencies so that it is easier to \"turn the volume down\" on known sources of noise of a certain frequency which can drown out the signal. The question is what are these known sources of noise?
\nFor LIGO, they are actually able to simulate this and create a \"design curve\" which takes into account known sources of noise in the apparatus. You can download the actual LIGO design curves here.
\n
\nFor the unknown sources (i.e. things that were not designed for), e.g. AC current, disturbance from transport etc, this is where notch and bandpass filters come in.
The issue here is that our data is simulated to look like LIGO, but is not exactly the same, so we cannot use the actual design curves. That's ok though since we know that all the target=0 samples are only noise, so we can derive the design curves for each of the 3 detectors from that.
\nWe calculate the average PSD using scipy.signal.periodogram (Welch's method) like this:
def build_design_curves(window=(\"tukey\", 0.2)):\n df = (\n pd.read_csv(INPUT_PATH / \"training_labels.csv\")\n .query(\"target == 0\")\n .reset_index(drop=True)\n )\n hanford = 0\n livingston = 0\n virgo = 0\n n = len(df)\n\n for i in tqdm(df[\"id\"]):\n data = load_file(i)\n hanford += signal.periodogram(data[0], fs=2048, window=window)[1]\n livingston += signal.periodogram(data[1], fs=2048, window=window)[1]\n virgo += signal.periodogram(data[2], fs=2048, window=window)[1]\n\n hanford /= n\n livingston /= n\n virgo /= n\n\n design_curves = np.stack([hanford, livingston, virgo])[:, :-1] ** 0.5\n return design_curves\nThe curves look like this with Hanford & Livingston on top of each other (blue & orange) and Virgo in green (similar to the ones shared here):
\n![\"\"](\"https://i.imgur.com/XVaeFHv.png\")
\nThe \"design curves\" can then be applied to raw data like this:
def apply_whiten(signal, design_curves):\n \"\"\"Whitens a waveform according to a design curve\n\n Args:\n signal (tensor): A waveform with a window already applied\n design_curves (tensor): Design curves for the window used\n\n Returns:\n tensor: Whitened waveform scaled between -1 and +1\n \"\"\"\n spec = torch.fft.fft(signal)\n n = signal.shape[-1]\n dc_len = design_curves.shape[-1]\n whitened = torch.real(torch.fft.ifft(spec[:, :dc_len] / design_curves, n=n))\n whitened *= np.sqrt(n / 2)\n whitened /= torch.max(torch.abs(whitened), axis=1)[0].reshape(3, 1)\n return whitened.to(dtype=torch.float32)\nSince we have all the sources of noise (e.g. the spike at 306Hz), not a lot of post-processing is needed. In fact, my models didn't need bandpass or notch filters after this. If you look at the curves, they will dampen a lot of the low-frequency noise. This really helps the long low-frequency tail of the GW emerge out of the noise on CQT or CWT and also help 1D models too.
", "messages": [ { "id": "1529335", "postDate": "09/30/2021 10:00:47", "content": "I think that a lot of people found the saga around whitening frustrating, so I wanted to share my views on it (our full 3rd place solution will be coming soon)
\nMy interpretation of the intent of whitening is to normalise the amplitude of different frequencies so that it is easier to \"turn the volume down\" on known sources of noise of a certain frequency which can drown out the signal. The question is what are these known sources of noise?
\nFor LIGO, they are actually able to simulate this and create a \"design curve\" which takes into account known sources of noise in the apparatus. You can download the actual LIGO design curves here.
\n
\nFor the unknown sources (i.e. things that were not designed for), e.g. AC current, disturbance from transport etc, this is where notch and bandpass filters come in.
The issue here is that our data is simulated to look like LIGO, but is not exactly the same, so we cannot use the actual design curves. That's ok though since we know that all the target=0 samples are only noise, so we can derive the design curves for each of the 3 detectors from that.
\nWe calculate the average PSD using scipy.signal.periodogram (Welch's method) like this:
def build_design_curves(window=(\"tukey\", 0.2)):\n df = (\n pd.read_csv(INPUT_PATH / \"training_labels.csv\")\n .query(\"target == 0\")\n .reset_index(drop=True)\n )\n hanford = 0\n livingston = 0\n virgo = 0\n n = len(df)\n\n for i in tqdm(df[\"id\"]):\n data = load_file(i)\n hanford += signal.periodogram(data[0], fs=2048, window=window)[1]\n livingston += signal.periodogram(data[1], fs=2048, window=window)[1]\n virgo += signal.periodogram(data[2], fs=2048, window=window)[1]\n\n hanford /= n\n livingston /= n\n virgo /= n\n\n design_curves = np.stack([hanford, livingston, virgo])[:, :-1] ** 0.5\n return design_curves\nThe curves look like this with Hanford & Livingston on top of each other (blue & orange) and Virgo in green (similar to the ones shared here):
\n
\nThe \"design curves\" can then be applied to raw data like this:
def apply_whiten(signal, design_curves):\n \"\"\"Whitens a waveform according to a design curve\n\n Args:\n signal (tensor): A waveform with a window already applied\n design_curves (tensor): Design curves for the window used\n\n Returns:\n tensor: Whitened waveform scaled between -1 and +1\n \"\"\"\n spec = torch.fft.fft(signal)\n n = signal.shape[-1]\n dc_len = design_curves.shape[-1]\n whitened = torch.real(torch.fft.ifft(spec[:, :dc_len] / design_curves, n=n))\n whitened *= np.sqrt(n / 2)\n whitened /= torch.max(torch.abs(whitened), axis=1)[0].reshape(3, 1)\n return whitened.to(dtype=torch.float32)\nSince we have all the sources of noise (e.g. the spike at 306Hz), not a lot of post-processing is needed. In fact, my models didn't need bandpass or notch filters after this. If you look at the curves, they will dampen a lot of the low-frequency noise. This really helps the long low-frequency tail of the GW emerge out of the noise on CQT or CWT and also help 1D models too.
", "rawMarkdown": "I think that a lot of people found the saga around whitening frustrating, so I wanted to share my views on it (our full 3rd place solution will be coming soon)\n\nMy interpretation of the intent of whitening is to normalise the amplitude of different frequencies so that it is easier to \"turn the volume down\" on known sources of noise of a certain frequency which can drown out the signal. The question is what are these known sources of noise?\n\nFor LIGO, they are actually able to simulate this and create a \"design curve\" which takes into account known sources of noise in the apparatus. You can download the actual LIGO design curves [here](https://dcc.ligo.org/cgi-bin/DocDB/ShowDocument?.submit=Identifier&docid=T1800044&version=5).\n\nFor the unknown sources (i.e. things that were not designed for), e.g. AC current, disturbance from transport etc, this is where notch and bandpass filters come in.\n\nThe issue here is that our data is simulated to look like LIGO, but is not exactly the same, so we cannot use the actual design curves. That's ok though since we know that all the target=0 samples are only noise, so we can derive the design curves for each of the 3 detectors from that. \n\nWe calculate the average PSD using `scipy.signal.periodogram` (Welch's method) like this:\n\n```\ndef build_design_curves(window=(\"tukey\", 0.2)):\n df = (\n pd.read_csv(INPUT_PATH / \"training_labels.csv\")\n .query(\"target == 0\")\n .reset_index(drop=True)\n )\n hanford = 0\n livingston = 0\n virgo = 0\n n = len(df)\n\n for i in tqdm(df[\"id\"]):\n data = load_file(i)\n hanford += signal.periodogram(data[0], fs=2048, window=window)[1]\n livingston += signal.periodogram(data[1], fs=2048, window=window)[1]\n virgo += signal.periodogram(data[2], fs=2048, window=window)[1]\n\n hanford /= n\n livingston /= n\n virgo /= n\n\n design_curves = np.stack([hanford, livingston, virgo])[:, :-1] ** 0.5\n return design_curves\n\n```\n\nThe curves look like this with Hanford & Livingston on top of each other (blue & orange) and Virgo in green (similar to the ones shared [here](https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/274520#1524582)):\n\nThe \"design curves\" can then be applied to raw data like this:\n```\ndef apply_whiten(signal, design_curves):\n \"\"\"Whitens a waveform according to a design curve\n\n Args:\n signal (tensor): A waveform with a window already applied\n design_curves (tensor): Design curves for the window used\n\n Returns:\n tensor: Whitened waveform scaled between -1 and +1\n \"\"\"\n spec = torch.fft.fft(signal)\n n = signal.shape[-1]\n dc_len = design_curves.shape[-1]\n whitened = torch.real(torch.fft.ifft(spec[:, :dc_len] / design_curves, n=n))\n whitened *= np.sqrt(n / 2)\n whitened /= torch.max(torch.abs(whitened), axis=1)[0].reshape(3, 1)\n return whitened.to(dtype=torch.float32)\n```\n\nSince we have all the sources of noise (e.g. the spike at 306Hz), not a lot of post-processing is needed. In fact, my models didn't need bandpass or notch filters after this. If you look at the curves, they will dampen a lot of the low-frequency noise. This really helps the long low-frequency tail of the GW emerge out of the noise on CQT or CWT and also help 1D models too.", "votes": null }, { "id": "1529337", "postDate": "09/30/2021 10:02:24", "content": "\"In fact, my models didn't need bandpass or notch filters after this.\"
\nThanks. This is proof that your whitening works
", "rawMarkdown": "\"In fact, my models didn't need bandpass or notch filters after this.\"\n\nThanks. This is proof that your whitening works", "votes": null }, { "id": "1529455", "postDate": "09/30/2021 12:20:56", "content": "Excellent job at bridging that gap from domain to delivery, and thank you for sharing.
", "rawMarkdown": "Excellent job at bridging that gap from domain to delivery, and thank you for sharing.", "votes": null }, { "id": "1559884", "postDate": "10/27/2021 08:05:44", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
\"In fact, my models didn't need bandpass or notch filters after this.\"
\nThanks. This is proof that your whitening works
", "votes": null, "replies": [] }, { "id": 1529455, "author_name": "authman", "author_url": "", "post_date": "09/30/2021 12:20:56", "content": "Excellent job at bridging that gap from domain to delivery, and thank you for sharing.
", "votes": null, "replies": [] }, { "id": 1559884, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 08:05:44", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris