{
  "id": 252396,
  "title": "Spectral Whitening",
  "url": "/competitions/g2net-gravitational-wave-detection/discussion/252396",
  "author_name": "",
  "post_date": "2021-07-12T08:14:13.613645800Z",
  "votes": 26,
  "comment_count": 10,
  "views": 0,
  "content": "<p>Having seen in  GWPy how valuable spectra whitening was I decided to try this with my modeling.  The summary is</p>\n<ul>\n<li>Spectrograms in nnAudio are 10-100 faster than GWPy</li>\n<li>As there wasn't a whitening algorithm in Torch I wrote a very basic one.</li>\n<li>While whitening the signal before taking a spectrogram worked really well for the real GW events this reduce my AUC by 10%</li>\n</ul>\n<p>See <a href=\"https://www.kaggle.com/kevinmcisaac/g2net-spectral-whitening\" target=\"_blank\">here</a> for details.</p>",
  "messages": [
    {
      "id": "1384811",
      "postDate": "07/12/2021 08:14:13",
      "content": "<p>Having seen in  GWPy how valuable spectra whitening was I decided to try this with my modeling.  The summary is</p>\n<ul>\n<li>Spectrograms in nnAudio are 10-100 faster than GWPy</li>\n<li>As there wasn't a whitening algorithm in Torch I wrote a very basic one.</li>\n<li>While whitening the signal before taking a spectrogram worked really well for the real GW events this reduce my AUC by 10%</li>\n</ul>\n<p>See <a href=\"https://www.kaggle.com/kevinmcisaac/g2net-spectral-whitening\" target=\"_blank\">here</a> for details.</p>",
      "rawMarkdown": "Having seen in  GWPy how valuable spectra whitening was I decided to try this with my modeling.  The summary is\n- Spectrograms in nnAudio are 10-100 faster than GWPy\n- As there wasn't a whitening algorithm in Torch I wrote a very basic one.\n- While whitening the signal before taking a spectrogram worked really well for the real GW events this reduce my AUC by 10%\n\nSee [here](https://www.kaggle.com/kevinmcisaac/g2net-spectral-whitening) for details.",
      "votes": null
    },
    {
      "id": "1384866",
      "postDate": "07/12/2021 09:29:02",
      "content": "<p>Thanks for sharing.  I also have negative results with whitening.</p>",
      "rawMarkdown": "Thanks for sharing.  I also have negative results with whitening.",
      "votes": null
    },
    {
      "id": "1387447",
      "postDate": "07/14/2021 07:35:00",
      "content": "<p>That is great to know. I was wondering if I'd made a mistake somewhere.</p>\n<p>Did you come to any conclusion why this did not help when whitening greatly enhances the  \"chirp\" signature in the Spectrograms  for the GW data?</p>\n<p>When I applied it to the competition data whitening this not \"reveal\" the chirp signature. My thoughts are SNR is so low the whitened  Spectrogram signature is swamped by the noise. I've updated the notebook to show this.</p>",
      "rawMarkdown": "That is great to know. I was wondering if I'd made a mistake somewhere.\n\nDid you come to any conclusion why this did not help when whitening greatly enhances the  \"chirp\" signature in the Spectrograms  for the GW data?\n\nWhen I applied it to the competition data whitening this not \"reveal\" the chirp signature. My thoughts are SNR is so low the whitened  Spectrogram signature is swamped by the noise. I've updated the notebook to show this.",
      "votes": null
    },
    {
      "id": "1387544",
      "postDate": "07/14/2021 09:02:06",
      "content": "<p>I don't know.  Given all research papers do it there must be something we are missing.</p>",
      "rawMarkdown": "I don't know.  Given all research papers do it there must be something we are missing.",
      "votes": null
    },
    {
      "id": "1469690",
      "postDate": "08/13/2021 04:48:18",
      "content": "<p>I using the whitening by GWPy first then using CQT by nnAduin,the result reduec by 2%.</p>",
      "rawMarkdown": "I using the whitening by GWPy first then using CQT by nnAduin,the result reduec by 2%.",
      "votes": null
    },
    {
      "id": "1470299",
      "postDate": "08/13/2021 12:51:20",
      "content": "<p>I also have tried whitening + normalize + CQT, but CV dropped slightly. <br>\nI'm struggling with finding proper signal processing…</p>",
      "rawMarkdown": "I also have tried whitening + normalize + CQT, but CV dropped slightly. \nI'm struggling with finding proper signal processing...",
      "votes": null
    },
    {
      "id": "1477104",
      "postDate": "08/17/2021 10:56:24",
      "content": "<p><a href=\"https://www.kaggle.com/kevinmcisaac\" target=\"_blank\">@kevinmcisaac</a>  did bandpass filtering help any way</p>",
      "rawMarkdown": "kevinmcisaac  did bandpass filtering help any way",
      "votes": null
    },
    {
      "id": "1478461",
      "postDate": "08/18/2021 02:18:35",
      "content": "<p>I thought it did but I did not do  a proper experiment on that. With CWT and CQT, the other option is to limit the frequency band, which probably does a similar thing.</p>",
      "rawMarkdown": "I thought it did but I did not do  a proper experiment on that. With CWT and CQT, the other option is to limit the frequency band, which probably does a similar thing.",
      "votes": null
    },
    {
      "id": "1483747",
      "postDate": "08/20/2021 20:29:15",
      "content": "<p>I wonder if whitening is failing us here due to artifacting from the relatively low sample rate of the data. Published research papers are likely to be using a much higher sample rate maybe?</p>\n<p>In the image below I've resampled a clean simulated GW to various sample rates using <code>interp1d</code>. The PSD is quite variable…</p>\n<p><img src=\"https://i.imgur.com/EYvIb6Q.png\" alt=\"\"></p>",
      "rawMarkdown": "I wonder if whitening is failing us here due to artifacting from the relatively low sample rate of the data. Published research papers are likely to be using a much higher sample rate maybe?\n\nIn the image below I've resampled a clean simulated GW to various sample rates using `interp1d`. The PSD is quite variable...\n\n![](https://i.imgur.com/EYvIb6Q.png)",
      "votes": null
    },
    {
      "id": "1504483",
      "postDate": "09/06/2021 12:20:24",
      "content": "<p>how about whiten only samples that have relatively high score by an initial model?</p>\n<p>instead of throwing away the noise, how about using the discovered noise as additional input?</p>",
      "rawMarkdown": "how about whiten only samples that have relatively high score by an initial model?\n\ninstead of throwing away the noise, how about using the discovered noise as additional input?",
      "votes": null
    },
    {
      "id": "1559961",
      "postDate": "10/27/2021 08:38:51",
      "content": "<p>Hey All,</p>\n<p>Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey <a href=\"https://forms.gle/QP9L16niPexozyhu5\" target=\"_blank\">https://forms.gle/QP9L16niPexozyhu5</a>.</p>\n<p>Thank you all,</p>\n<p>Regards,<br>\nChris</p>",
      "rawMarkdown": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1384866,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "07/12/2021 09:29:02",
      "content": "<p>Thanks for sharing.  I also have negative results with whitening.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1387447,
          "author_name": "kevinmcisaac",
          "author_url": "",
          "post_date": "07/14/2021 07:35:00",
          "content": "<p>That is great to know. I was wondering if I'd made a mistake somewhere.</p>\n<p>Did you come to any conclusion why this did not help when whitening greatly enhances the  \"chirp\" signature in the Spectrograms  for the GW data?</p>\n<p>When I applied it to the competition data whitening this not \"reveal\" the chirp signature. My thoughts are SNR is so low the whitened  Spectrogram signature is swamped by the noise. I've updated the notebook to show this.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1387544,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "07/14/2021 09:02:06",
          "content": "<p>I don't know.  Given all research papers do it there must be something we are missing.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1469690,
      "author_name": "hanson0910",
      "author_url": "",
      "post_date": "08/13/2021 04:48:18",
      "content": "<p>I using the whitening by GWPy first then using CQT by nnAduin,the result reduec by 2%.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1470299,
      "author_name": "yoichi7yamakawa",
      "author_url": "",
      "post_date": "08/13/2021 12:51:20",
      "content": "<p>I also have tried whitening + normalize + CQT, but CV dropped slightly. <br>\nI'm struggling with finding proper signal processing…</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1477104,
      "author_name": "jaideepvalani",
      "author_url": "",
      "post_date": "08/17/2021 10:56:24",
      "content": "<p><a href=\"https://www.kaggle.com/kevinmcisaac\" target=\"_blank\">@kevinmcisaac</a>  did bandpass filtering help any way</p>",
      "votes": null,
      "replies": [
        {
          "id": 1478461,
          "author_name": "kevinmcisaac",
          "author_url": "",
          "post_date": "08/18/2021 02:18:35",
          "content": "<p>I thought it did but I did not do  a proper experiment on that. With CWT and CQT, the other option is to limit the frequency band, which probably does a similar thing.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1483747,
      "author_name": "anjum48",
      "author_url": "",
      "post_date": "08/20/2021 20:29:15",
      "content": "<p>I wonder if whitening is failing us here due to artifacting from the relatively low sample rate of the data. Published research papers are likely to be using a much higher sample rate maybe?</p>\n<p>In the image below I've resampled a clean simulated GW to various sample rates using <code>interp1d</code>. The PSD is quite variable…</p>\n<p><img src=\"https://i.imgur.com/EYvIb6Q.png\" alt=\"\"></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1504483,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "09/06/2021 12:20:24",
      "content": "<p>how about whiten only samples that have relatively high score by an initial model?</p>\n<p>instead of throwing away the noise, how about using the discovered noise as additional input?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1559961,
      "author_name": "zerafachris",
      "author_url": "",
      "post_date": "10/27/2021 08:38:51",
      "content": "<p>Hey All,</p>\n<p>Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey <a href=\"https://forms.gle/QP9L16niPexozyhu5\" target=\"_blank\">https://forms.gle/QP9L16niPexozyhu5</a>.</p>\n<p>Thank you all,</p>\n<p>Regards,<br>\nChris</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1384811": "Having seen in  GWPy how valuable spectra whitening was I decided to try this with my modeling.  The summary is\n- Spectrograms in nnAudio are 10-100 faster than GWPy\n- As there wasn't a whitening algorithm in Torch I wrote a very basic one.\n- While whitening the signal before taking a spectrogram worked really well for the real GW events this reduce my AUC by 10%\n\nSee [here](https://www.kaggle.com/kevinmcisaac/g2net-spectral-whitening) for details.",
    "1384866": "Thanks for sharing.  I also have negative results with whitening.",
    "1387447": "That is great to know. I was wondering if I'd made a mistake somewhere.\n\nDid you come to any conclusion why this did not help when whitening greatly enhances the  \"chirp\" signature in the Spectrograms  for the GW data?\n\nWhen I applied it to the competition data whitening this not \"reveal\" the chirp signature. My thoughts are SNR is so low the whitened  Spectrogram signature is swamped by the noise. I've updated the notebook to show this.",
    "1387544": "I don't know.  Given all research papers do it there must be something we are missing.",
    "1469690": "I using the whitening by GWPy first then using CQT by nnAduin,the result reduec by 2%.",
    "1470299": "I also have tried whitening + normalize + CQT, but CV dropped slightly. \nI'm struggling with finding proper signal processing...",
    "1477104": "kevinmcisaac  did bandpass filtering help any way",
    "1478461": "I thought it did but I did not do  a proper experiment on that. With CWT and CQT, the other option is to limit the frequency band, which probably does a similar thing.",
    "1483747": "I wonder if whitening is failing us here due to artifacting from the relatively low sample rate of the data. Published research papers are likely to be using a much higher sample rate maybe?\n\nIn the image below I've resampled a clean simulated GW to various sample rates using `interp1d`. The PSD is quite variable...\n\n![](https://i.imgur.com/EYvIb6Q.png)",
    "1504483": "how about whiten only samples that have relatively high score by an initial model?\n\ninstead of throwing away the noise, how about using the discovered noise as additional input?",
    "1559961": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris"
  },
  "source": "meta"
}