{
  "id": 475463,
  "title": "High Pass Filter",
  "url": "/competitions/hms-harmful-brain-activity-classification/discussion/475463",
  "author_name": "",
  "post_date": "2024-02-08T14:13:59.190286300Z",
  "votes": 12,
  "comment_count": 10,
  "views": 0,
  "content": "<p>I'm thinking about applying a High Pass filter to the EEG data before creating the spectograms. When researching I saw that it is possible to apply cutoff bands 0.1Hz to 0.5Hz.</p>\n<p>Below is the code I am using to apply the filter</p>\n<pre><code>\n scipy.signal  butter, filtfilt, iirnotch\n ():\n     nyq =  * fs\n     normal_cutoff = cutoff / nyq\n     b, a = butter(order, normal_cutoff, btype=, analog=)\n     y = filtfilt(b, a, data)\n      y\n</code></pre>\n<p>I'm doing experiments to determine the best cutoff value, theoretically from what I've seen in the literature, applying a filter of 0.1 is common in studies that want to preserve the majority of brain oscillations, including those in the very low frequency spectrum, such as the delta band (1-4 Hz), which is important for sleep and other states of altered consciousness.</p>\n<p>In the case of applying a 0.5 filter, it is often used for analyzes that can dispense with the lower part of the delta band or when the interest is predominantly in higher frequencies, such as theta bands (4-8 Hz), alpha (8-12 Hz ), beta (13-30 Hz), and gamma (30-100 Hz).</p>\n<p>In your experience, what would be the ideal cutoff for the high pass filter, or the most correct one for the competition data?</p>",
  "messages": [
    {
      "id": "2642944",
      "postDate": "02/08/2024 14:13:59",
      "content": "<p>I'm thinking about applying a High Pass filter to the EEG data before creating the spectograms. When researching I saw that it is possible to apply cutoff bands 0.1Hz to 0.5Hz.</p>\n<p>Below is the code I am using to apply the filter</p>\n<pre><code>\n scipy.signal  butter, filtfilt, iirnotch\n ():\n     nyq =  * fs\n     normal_cutoff = cutoff / nyq\n     b, a = butter(order, normal_cutoff, btype=, analog=)\n     y = filtfilt(b, a, data)\n      y\n</code></pre>\n<p>I'm doing experiments to determine the best cutoff value, theoretically from what I've seen in the literature, applying a filter of 0.1 is common in studies that want to preserve the majority of brain oscillations, including those in the very low frequency spectrum, such as the delta band (1-4 Hz), which is important for sleep and other states of altered consciousness.</p>\n<p>In the case of applying a 0.5 filter, it is often used for analyzes that can dispense with the lower part of the delta band or when the interest is predominantly in higher frequencies, such as theta bands (4-8 Hz), alpha (8-12 Hz ), beta (13-30 Hz), and gamma (30-100 Hz).</p>\n<p>In your experience, what would be the ideal cutoff for the high pass filter, or the most correct one for the competition data?</p>",
      "rawMarkdown": "I'm thinking about applying a High Pass filter to the EEG data before creating the spectograms. When researching I saw that it is possible to apply cutoff bands 0.1Hz to 0.5Hz.\n\nBelow is the code I am using to apply the filter\n```python\n# High Pass Filter Function\nfrom scipy.signal import butter, filtfilt, iirnotch\ndef butter_highpass_filter(data, cutoff, fs=200, order=5):\n     nyq = 0.5 * fs\n     normal_cutoff = cutoff / nyq\n     b, a = butter(order, normal_cutoff, btype='high', analog=False)\n     y = filtfilt(b, a, data)\n     return y\n```\nI'm doing experiments to determine the best cutoff value, theoretically from what I've seen in the literature, applying a filter of 0.1 is common in studies that want to preserve the majority of brain oscillations, including those in the very low frequency spectrum, such as the delta band (1-4 Hz), which is important for sleep and other states of altered consciousness.\n\nIn the case of applying a 0.5 filter, it is often used for analyzes that can dispense with the lower part of the delta band or when the interest is predominantly in higher frequencies, such as theta bands (4-8 Hz), alpha (8-12 Hz ), beta (13-30 Hz), and gamma (30-100 Hz).\n\nIn your experience, what would be the ideal cutoff for the high pass filter, or the most correct one for the competition data?",
      "votes": null
    },
    {
      "id": "2643148",
      "postDate": "02/08/2024 16:43:42",
      "content": "<p>I have experimented with 0.5 vs 1 Hz and found 1 Hz to produce better results but I would be really interested in your experience!</p>\n<p>Best,<br>\nJan</p>",
      "rawMarkdown": "I have experimented with 0.5 vs 1 Hz and found 1 Hz to produce better results but I would be really interested in your experience!\n\nBest,\nJan",
      "votes": null
    },
    {
      "id": "2643182",
      "postDate": "02/08/2024 17:15:13",
      "content": "<p>So far I managed to do an experiment with 0.1, the result got a little worse, I'm doing other experiments now, but it takes a long time to create the spectograms, in the next experiments I intend to use higher values like 0.5 and 1 and see if it improves, another experiment I intend to do is check whether it is better to pass the filter before computing the differences between pairs or after.</p>",
      "rawMarkdown": "So far I managed to do an experiment with 0.1, the result got a little worse, I'm doing other experiments now, but it takes a long time to create the spectograms, in the next experiments I intend to use higher values like 0.5 and 1 and see if it improves, another experiment I intend to do is check whether it is better to pass the filter before computing the differences between pairs or after.",
      "votes": null
    },
    {
      "id": "2643428",
      "postDate": "02/08/2024 20:31:28",
      "content": "<p>Ah, I might have misunderstood you - I was talking about models with the raw eegs. Still stuck on that I guess, motivated to improve them as far as possible before adding the 2D models…</p>\n<p>Best,<br>\nJan</p>",
      "rawMarkdown": "Ah, I might have misunderstood you - I was talking about models with the raw eegs. Still stuck on that I guess, motivated to improve them as far as possible before adding the 2D models...\n\nBest,\nJan",
      "votes": null
    },
    {
      "id": "2643544",
      "postDate": "02/08/2024 22:22:30",
      "content": "<p>No problem, unfortunately I haven't worked on the raw eegs model yet. I intend to do it in the future, but for now I have no experience to share :(</p>",
      "rawMarkdown": "No problem, unfortunately I haven't worked on the raw eegs model yet. I intend to do it in the future, but for now I have no experience to share :(",
      "votes": null
    },
    {
      "id": "2645815",
      "postDate": "02/10/2024 13:31:08",
      "content": "<blockquote>\n  <p>In the case of applying a 0.5 filter, it is often used for analyzes that can dispense with the lower part of the delta band or when the interest is predominantly in higher frequencies, such as theta bands (4-8 Hz), alpha (8-12 Hz ), beta (13-30 Hz), and gamma (30-100 Hz).</p>\n</blockquote>\n<p>As far as I know, the main reason to use a high-pass filter is to remove DC drift and/or sweat artifacts from the raw data, but your point also makes sense. You can use a high-pass filter to ensure good ICA performance as well (with a 1-2 Hz cutoff frequency as recommended in <a href=\"https://ieeexplore.ieee.org/document/7319296\" target=\"_blank\">here</a>).</p>\n<p>I agree with Jan Brederecke that 1 Hz produces better results. I think the ideal cutoff depends on your mel filter as well. May I ask what frequency interval you chose for your spectrogram if you used a mel spectrogram?</p>",
      "rawMarkdown": ">In the case of applying a 0.5 filter, it is often used for analyzes that can dispense with the lower part of the delta band or when the interest is predominantly in higher frequencies, such as theta bands (4-8 Hz), alpha (8-12 Hz ), beta (13-30 Hz), and gamma (30-100 Hz).\n\nAs far as I know, the main reason to use a high-pass filter is to remove DC drift and/or sweat artifacts from the raw data, but your point also makes sense. You can use a high-pass filter to ensure good ICA performance as well (with a 1-2 Hz cutoff frequency as recommended in [here](https://ieeexplore.ieee.org/document/7319296)).\n\nI agree with Jan Brederecke that 1 Hz produces better results. I think the ideal cutoff depends on your mel filter as well. May I ask what frequency interval you chose for your spectrogram if you used a mel spectrogram?",
      "votes": null
    },
    {
      "id": "2645937",
      "postDate": "02/10/2024 14:56:29",
      "content": "<p>Yes, I'm using the standard cut that most people are using from 0 to 20Hz, the competition specs are also using this cut, apparently it seems to be the best optimized.</p>\n<p>My experience with filters to create spectograms was that they did not improve the results in any test scenario I did, I'm sure the data is full of noise, but at least in my case, using CNN, this type of filter did not obtain result, moving on to other experiments with different filters.</p>\n<p>In my opinion, the biggest problem with these noise filters is that they remove noise, but they also remove signal data that is useful for the model.</p>",
      "rawMarkdown": "Yes, I'm using the standard cut that most people are using from 0 to 20Hz, the competition specs are also using this cut, apparently it seems to be the best optimized.\n\nMy experience with filters to create spectograms was that they did not improve the results in any test scenario I did, I'm sure the data is full of noise, but at least in my case, using CNN, this type of filter did not obtain result, moving on to other experiments with different filters.\n\nIn my opinion, the biggest problem with these noise filters is that they remove noise, but they also remove signal data that is useful for the model.",
      "votes": null
    },
    {
      "id": "2646080",
      "postDate": "02/10/2024 16:21:56",
      "content": "<p>This EEG (and of course ECG) data is a big mess that requires a meticulous preprocessing pipeline. We need to manually inspect the samples, as some of them even cause tracebacks in the pipeline because of their unusual structure.</p>\n<blockquote>\n  <p>In my opinion, the biggest problem with these noise filters is that they remove noise, but they also remove signal data that is useful for the model.</p>\n</blockquote>\n<p>Yes, some filters, such as the notch filter, which is commonly used to remove power-line noise artifacts, have a high likelihood of distorting the signal and removing brain signals from the data if it is not applied carefully. But, this is not necessarily true. Actually, we don't remove any signal unlike cropping; we <strong>attenuate</strong> the signal in order to increase the signal-to-noise ratio (SNR). For example, I generally prefer a 2nd order Butterworth filter because the phase distortion gets worse as the filter order increases. You can consider lowering the order of the Butterworth filter or using other filters with different cutoffs (e.g. FIR filters) if you suspect that there is information loss. PSD plots are quite useful for observing the effect of your filter on the signal.</p>",
      "rawMarkdown": "This EEG (and of course ECG) data is a big mess that requires a meticulous preprocessing pipeline. We need to manually inspect the samples, as some of them even cause tracebacks in the pipeline because of their unusual structure.\n\n>In my opinion, the biggest problem with these noise filters is that they remove noise, but they also remove signal data that is useful for the model.\n\nYes, some filters, such as the notch filter, which is commonly used to remove power-line noise artifacts, have a high likelihood of distorting the signal and removing brain signals from the data if it is not applied carefully. But, this is not necessarily true. Actually, we don't remove any signal unlike cropping; we **attenuate** the signal in order to increase the signal-to-noise ratio (SNR). For example, I generally prefer a 2nd order Butterworth filter because the phase distortion gets worse as the filter order increases. You can consider lowering the order of the Butterworth filter or using other filters with different cutoffs (e.g. FIR filters) if you suspect that there is information loss. PSD plots are quite useful for observing the effect of your filter on the signal.",
      "votes": null
    },
    {
      "id": "2646251",
      "postDate": "02/10/2024 18:37:55",
      "content": "<p>Thanks for the class Yusuf, you really helped me a lot :)</p>\n<p>Based on the materials you mentioned, I started doing some studies and realized that there are many different filters to try, I was using the function above in Numpy, I ended up modifying it to use MNE.</p>\n<p>Here are some materials that I thought would be cool to share, they might help someone in the same vein as me:</p>\n<p>I found this resource on the MNE that talks about various types of filters, like the FIR you mentioned:<br>\n<a href=\"https://mne.tools/stable/auto_tutorials/preprocessing/25_background_filtering.html\" target=\"_blank\">link1</a><br>\n<a href=\"https://neuraldatascience.io/7-eeg/erp_filtering.html\" target=\"_blank\">link2</a></p>\n<p>Butterworth Filter:<br>\n<a href=\"https://mne.discourse.group/t/butterworth-filter/5760\" target=\"_blank\">link3</a></p>\n<p>If it's not too much to ask, I would like to ask a question, in my studies I realized that it is recommended to always apply the filter first, but I noticed that some people first calculate the differences as 'F3-C3' and then calculate the filter.</p>\n<p>Shouldn't the filter be passed before calculating the differences?</p>",
      "rawMarkdown": "Thanks for the class Yusuf, you really helped me a lot :)\n\nBased on the materials you mentioned, I started doing some studies and realized that there are many different filters to try, I was using the function above in Numpy, I ended up modifying it to use MNE.\n\nHere are some materials that I thought would be cool to share, they might help someone in the same vein as me:\n\nI found this resource on the MNE that talks about various types of filters, like the FIR you mentioned:\n[link1](https://mne.tools/stable/auto_tutorials/preprocessing/25_background_filtering.html)\n[link2](https://neuraldatascience.io/7-eeg/erp_filtering.html)\n\nButterworth Filter:\n[link3](https://mne.discourse.group/t/butterworth-filter/5760)\n\nIf it's not too much to ask, I would like to ask a question, in my studies I realized that it is recommended to always apply the filter first, but I noticed that some people first calculate the differences as 'F3-C3' and then calculate the filter.\n\nShouldn't the filter be passed before calculating the differences?",
      "votes": null
    },
    {
      "id": "2646666",
      "postDate": "02/11/2024 05:56:16",
      "content": "<p>I am really glad to help you, and it is never too much to ask questions 🤓</p>\n<p>Personally, I always apply the high-pass filter first and set the bipolar reference at the end of my preprocessing pipeline. In fact, if you are trying to repair artifacts, most of the artifact repair algorithms either require a high-pass filter for better performance or apply filters automatically. </p>\n<p>For low-pass filters, it depends on the frequency you are targeting (e.g. if you are interested in low-frequency waves, you can simply apply a low-pass filter with a cutoff frequency of, say, 40 Hz.) and the dataset. Almost everyone in this competition doesn't use signals above 20 Hz, and I still don't know why we completely remove gamma waves and some of the beta waves from our spectrograms.</p>",
      "rawMarkdown": "I am really glad to help you, and it is never too much to ask questions 🤓\n\nPersonally, I always apply the high-pass filter first and set the bipolar reference at the end of my preprocessing pipeline. In fact, if you are trying to repair artifacts, most of the artifact repair algorithms either require a high-pass filter for better performance or apply filters automatically. \n\nFor low-pass filters, it depends on the frequency you are targeting (e.g. if you are interested in low-frequency waves, you can simply apply a low-pass filter with a cutoff frequency of, say, 40 Hz.) and the dataset. Almost everyone in this competition doesn't use signals above 20 Hz, and I still don't know why we completely remove gamma waves and some of the beta waves from our spectrograms.",
      "votes": null
    },
    {
      "id": "2647842",
      "postDate": "02/11/2024 19:43:49",
      "content": "<p>Yes, I believe it makes more sense to work as you said, create a pipeline to do the preprocessing and then make the bipolar reference.</p>\n<p>I'm working on it, it's not easy</p>\n<p>Regarding the frequencies used, I believe that the 30Hz frequency could easily be used in the spectograms, I did some tests with this type of spec and they gave a promising result.</p>\n<p>Thank you for your help</p>",
      "rawMarkdown": "Yes, I believe it makes more sense to work as you said, create a pipeline to do the preprocessing and then make the bipolar reference.\n\nI'm working on it, it's not easy\n\nRegarding the frequencies used, I believe that the 30Hz frequency could easily be used in the spectograms, I did some tests with this type of spec and they gave a promising result.\n\nThank you for your help",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2643148,
      "author_name": "janbrederecke",
      "author_url": "",
      "post_date": "02/08/2024 16:43:42",
      "content": "<p>I have experimented with 0.5 vs 1 Hz and found 1 Hz to produce better results but I would be really interested in your experience!</p>\n<p>Best,<br>\nJan</p>",
      "votes": null,
      "replies": [
        {
          "id": 2643182,
          "author_name": "rafaelzimmermann1",
          "author_url": "",
          "post_date": "02/08/2024 17:15:13",
          "content": "<p>So far I managed to do an experiment with 0.1, the result got a little worse, I'm doing other experiments now, but it takes a long time to create the spectograms, in the next experiments I intend to use higher values like 0.5 and 1 and see if it improves, another experiment I intend to do is check whether it is better to pass the filter before computing the differences between pairs or after.</p>",
          "votes": null,
          "replies": [
            {
              "id": 2643428,
              "author_name": "janbrederecke",
              "author_url": "",
              "post_date": "02/08/2024 20:31:28",
              "content": "<p>Ah, I might have misunderstood you - I was talking about models with the raw eegs. Still stuck on that I guess, motivated to improve them as far as possible before adding the 2D models…</p>\n<p>Best,<br>\nJan</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2643544,
                  "author_name": "rafaelzimmermann1",
                  "author_url": "",
                  "post_date": "02/08/2024 22:22:30",
                  "content": "<p>No problem, unfortunately I haven't worked on the raw eegs model yet. I intend to do it in the future, but for now I have no experience to share :(</p>",
                  "votes": null,
                  "replies": []
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 2645815,
      "author_name": "yusufserdaraytekin",
      "author_url": "",
      "post_date": "02/10/2024 13:31:08",
      "content": "<blockquote>\n  <p>In the case of applying a 0.5 filter, it is often used for analyzes that can dispense with the lower part of the delta band or when the interest is predominantly in higher frequencies, such as theta bands (4-8 Hz), alpha (8-12 Hz ), beta (13-30 Hz), and gamma (30-100 Hz).</p>\n</blockquote>\n<p>As far as I know, the main reason to use a high-pass filter is to remove DC drift and/or sweat artifacts from the raw data, but your point also makes sense. You can use a high-pass filter to ensure good ICA performance as well (with a 1-2 Hz cutoff frequency as recommended in <a href=\"https://ieeexplore.ieee.org/document/7319296\" target=\"_blank\">here</a>).</p>\n<p>I agree with Jan Brederecke that 1 Hz produces better results. I think the ideal cutoff depends on your mel filter as well. May I ask what frequency interval you chose for your spectrogram if you used a mel spectrogram?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2645937,
          "author_name": "rafaelzimmermann1",
          "author_url": "",
          "post_date": "02/10/2024 14:56:29",
          "content": "<p>Yes, I'm using the standard cut that most people are using from 0 to 20Hz, the competition specs are also using this cut, apparently it seems to be the best optimized.</p>\n<p>My experience with filters to create spectograms was that they did not improve the results in any test scenario I did, I'm sure the data is full of noise, but at least in my case, using CNN, this type of filter did not obtain result, moving on to other experiments with different filters.</p>\n<p>In my opinion, the biggest problem with these noise filters is that they remove noise, but they also remove signal data that is useful for the model.</p>",
          "votes": null,
          "replies": [
            {
              "id": 2646080,
              "author_name": "yusufserdaraytekin",
              "author_url": "",
              "post_date": "02/10/2024 16:21:56",
              "content": "<p>This EEG (and of course ECG) data is a big mess that requires a meticulous preprocessing pipeline. We need to manually inspect the samples, as some of them even cause tracebacks in the pipeline because of their unusual structure.</p>\n<blockquote>\n  <p>In my opinion, the biggest problem with these noise filters is that they remove noise, but they also remove signal data that is useful for the model.</p>\n</blockquote>\n<p>Yes, some filters, such as the notch filter, which is commonly used to remove power-line noise artifacts, have a high likelihood of distorting the signal and removing brain signals from the data if it is not applied carefully. But, this is not necessarily true. Actually, we don't remove any signal unlike cropping; we <strong>attenuate</strong> the signal in order to increase the signal-to-noise ratio (SNR). For example, I generally prefer a 2nd order Butterworth filter because the phase distortion gets worse as the filter order increases. You can consider lowering the order of the Butterworth filter or using other filters with different cutoffs (e.g. FIR filters) if you suspect that there is information loss. PSD plots are quite useful for observing the effect of your filter on the signal.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2646251,
                  "author_name": "rafaelzimmermann1",
                  "author_url": "",
                  "post_date": "02/10/2024 18:37:55",
                  "content": "<p>Thanks for the class Yusuf, you really helped me a lot :)</p>\n<p>Based on the materials you mentioned, I started doing some studies and realized that there are many different filters to try, I was using the function above in Numpy, I ended up modifying it to use MNE.</p>\n<p>Here are some materials that I thought would be cool to share, they might help someone in the same vein as me:</p>\n<p>I found this resource on the MNE that talks about various types of filters, like the FIR you mentioned:<br>\n<a href=\"https://mne.tools/stable/auto_tutorials/preprocessing/25_background_filtering.html\" target=\"_blank\">link1</a><br>\n<a href=\"https://neuraldatascience.io/7-eeg/erp_filtering.html\" target=\"_blank\">link2</a></p>\n<p>Butterworth Filter:<br>\n<a href=\"https://mne.discourse.group/t/butterworth-filter/5760\" target=\"_blank\">link3</a></p>\n<p>If it's not too much to ask, I would like to ask a question, in my studies I realized that it is recommended to always apply the filter first, but I noticed that some people first calculate the differences as 'F3-C3' and then calculate the filter.</p>\n<p>Shouldn't the filter be passed before calculating the differences?</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 2646666,
                      "author_name": "yusufserdaraytekin",
                      "author_url": "",
                      "post_date": "02/11/2024 05:56:16",
                      "content": "<p>I am really glad to help you, and it is never too much to ask questions 🤓</p>\n<p>Personally, I always apply the high-pass filter first and set the bipolar reference at the end of my preprocessing pipeline. In fact, if you are trying to repair artifacts, most of the artifact repair algorithms either require a high-pass filter for better performance or apply filters automatically. </p>\n<p>For low-pass filters, it depends on the frequency you are targeting (e.g. if you are interested in low-frequency waves, you can simply apply a low-pass filter with a cutoff frequency of, say, 40 Hz.) and the dataset. Almost everyone in this competition doesn't use signals above 20 Hz, and I still don't know why we completely remove gamma waves and some of the beta waves from our spectrograms.</p>",
                      "votes": null,
                      "replies": [
                        {
                          "id": 2647842,
                          "author_name": "rafaelzimmermann1",
                          "author_url": "",
                          "post_date": "02/11/2024 19:43:49",
                          "content": "<p>Yes, I believe it makes more sense to work as you said, create a pipeline to do the preprocessing and then make the bipolar reference.</p>\n<p>I'm working on it, it's not easy</p>\n<p>Regarding the frequencies used, I believe that the 30Hz frequency could easily be used in the spectograms, I did some tests with this type of spec and they gave a promising result.</p>\n<p>Thank you for your help</p>",
                          "votes": null,
                          "replies": []
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2642944": "I'm thinking about applying a High Pass filter to the EEG data before creating the spectograms. When researching I saw that it is possible to apply cutoff bands 0.1Hz to 0.5Hz.\n\nBelow is the code I am using to apply the filter\n```python\n# High Pass Filter Function\nfrom scipy.signal import butter, filtfilt, iirnotch\ndef butter_highpass_filter(data, cutoff, fs=200, order=5):\n     nyq = 0.5 * fs\n     normal_cutoff = cutoff / nyq\n     b, a = butter(order, normal_cutoff, btype='high', analog=False)\n     y = filtfilt(b, a, data)\n     return y\n```\nI'm doing experiments to determine the best cutoff value, theoretically from what I've seen in the literature, applying a filter of 0.1 is common in studies that want to preserve the majority of brain oscillations, including those in the very low frequency spectrum, such as the delta band (1-4 Hz), which is important for sleep and other states of altered consciousness.\n\nIn the case of applying a 0.5 filter, it is often used for analyzes that can dispense with the lower part of the delta band or when the interest is predominantly in higher frequencies, such as theta bands (4-8 Hz), alpha (8-12 Hz ), beta (13-30 Hz), and gamma (30-100 Hz).\n\nIn your experience, what would be the ideal cutoff for the high pass filter, or the most correct one for the competition data?",
    "2643148": "I have experimented with 0.5 vs 1 Hz and found 1 Hz to produce better results but I would be really interested in your experience!\n\nBest,\nJan",
    "2643182": "So far I managed to do an experiment with 0.1, the result got a little worse, I'm doing other experiments now, but it takes a long time to create the spectograms, in the next experiments I intend to use higher values like 0.5 and 1 and see if it improves, another experiment I intend to do is check whether it is better to pass the filter before computing the differences between pairs or after.",
    "2643428": "Ah, I might have misunderstood you - I was talking about models with the raw eegs. Still stuck on that I guess, motivated to improve them as far as possible before adding the 2D models...\n\nBest,\nJan",
    "2643544": "No problem, unfortunately I haven't worked on the raw eegs model yet. I intend to do it in the future, but for now I have no experience to share :(",
    "2645815": ">In the case of applying a 0.5 filter, it is often used for analyzes that can dispense with the lower part of the delta band or when the interest is predominantly in higher frequencies, such as theta bands (4-8 Hz), alpha (8-12 Hz ), beta (13-30 Hz), and gamma (30-100 Hz).\n\nAs far as I know, the main reason to use a high-pass filter is to remove DC drift and/or sweat artifacts from the raw data, but your point also makes sense. You can use a high-pass filter to ensure good ICA performance as well (with a 1-2 Hz cutoff frequency as recommended in [here](https://ieeexplore.ieee.org/document/7319296)).\n\nI agree with Jan Brederecke that 1 Hz produces better results. I think the ideal cutoff depends on your mel filter as well. May I ask what frequency interval you chose for your spectrogram if you used a mel spectrogram?",
    "2645937": "Yes, I'm using the standard cut that most people are using from 0 to 20Hz, the competition specs are also using this cut, apparently it seems to be the best optimized.\n\nMy experience with filters to create spectograms was that they did not improve the results in any test scenario I did, I'm sure the data is full of noise, but at least in my case, using CNN, this type of filter did not obtain result, moving on to other experiments with different filters.\n\nIn my opinion, the biggest problem with these noise filters is that they remove noise, but they also remove signal data that is useful for the model.",
    "2646080": "This EEG (and of course ECG) data is a big mess that requires a meticulous preprocessing pipeline. We need to manually inspect the samples, as some of them even cause tracebacks in the pipeline because of their unusual structure.\n\n>In my opinion, the biggest problem with these noise filters is that they remove noise, but they also remove signal data that is useful for the model.\n\nYes, some filters, such as the notch filter, which is commonly used to remove power-line noise artifacts, have a high likelihood of distorting the signal and removing brain signals from the data if it is not applied carefully. But, this is not necessarily true. Actually, we don't remove any signal unlike cropping; we **attenuate** the signal in order to increase the signal-to-noise ratio (SNR). For example, I generally prefer a 2nd order Butterworth filter because the phase distortion gets worse as the filter order increases. You can consider lowering the order of the Butterworth filter or using other filters with different cutoffs (e.g. FIR filters) if you suspect that there is information loss. PSD plots are quite useful for observing the effect of your filter on the signal.",
    "2646251": "Thanks for the class Yusuf, you really helped me a lot :)\n\nBased on the materials you mentioned, I started doing some studies and realized that there are many different filters to try, I was using the function above in Numpy, I ended up modifying it to use MNE.\n\nHere are some materials that I thought would be cool to share, they might help someone in the same vein as me:\n\nI found this resource on the MNE that talks about various types of filters, like the FIR you mentioned:\n[link1](https://mne.tools/stable/auto_tutorials/preprocessing/25_background_filtering.html)\n[link2](https://neuraldatascience.io/7-eeg/erp_filtering.html)\n\nButterworth Filter:\n[link3](https://mne.discourse.group/t/butterworth-filter/5760)\n\nIf it's not too much to ask, I would like to ask a question, in my studies I realized that it is recommended to always apply the filter first, but I noticed that some people first calculate the differences as 'F3-C3' and then calculate the filter.\n\nShouldn't the filter be passed before calculating the differences?",
    "2646666": "I am really glad to help you, and it is never too much to ask questions 🤓\n\nPersonally, I always apply the high-pass filter first and set the bipolar reference at the end of my preprocessing pipeline. In fact, if you are trying to repair artifacts, most of the artifact repair algorithms either require a high-pass filter for better performance or apply filters automatically. \n\nFor low-pass filters, it depends on the frequency you are targeting (e.g. if you are interested in low-frequency waves, you can simply apply a low-pass filter with a cutoff frequency of, say, 40 Hz.) and the dataset. Almost everyone in this competition doesn't use signals above 20 Hz, and I still don't know why we completely remove gamma waves and some of the beta waves from our spectrograms.",
    "2647842": "Yes, I believe it makes more sense to work as you said, create a pipeline to do the preprocessing and then make the bipolar reference.\n\nI'm working on it, it's not easy\n\nRegarding the frequencies used, I believe that the 30Hz frequency could easily be used in the spectograms, I did some tests with this type of spec and they gave a promising result.\n\nThank you for your help"
  },
  "source": "meta"
}