![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12236136%2Fb37d24b1c2d3e43e0a10edeca79d3eff%2Fnoise.png?generation=1708546641136863&alt=media\")
Hi all,
\nThank you everyone for sharing so much useful information in this competition. I'm learning so much about signal processing.
\nProblem: for some eeg_ids, I identify artifactual (fake) noise, e.g. the figure attached below.
\n
Question: How are you folks dealing with examples like this?
\nMy first inclination is to design a function that iterates through all rows to flag EEG_ids that have this sort of artifact. For instance, iterate through each eeg channel and identify windows where the standard deviation of values is very close to zero, or stark/not-biological changes in slope. Then I would delete this artifactual elbow-like noise from the EEG dataset.
\nHow are you folks going about this problem? All input or thoughts are appreciated!
\nCheers,
\nShane
Hi all,
\nThank you everyone for sharing so much useful information in this competition. I'm learning so much about signal processing.
\nProblem: for some eeg_ids, I identify artifactual (fake) noise, e.g. the figure attached below.
\n
Question: How are you folks dealing with examples like this?
\nMy first inclination is to design a function that iterates through all rows to flag EEG_ids that have this sort of artifact. For instance, iterate through each eeg channel and identify windows where the standard deviation of values is very close to zero, or stark/not-biological changes in slope. Then I would delete this artifactual elbow-like noise from the EEG dataset.
\nHow are you folks going about this problem? All input or thoughts are appreciated!
\nCheers,
\nShane
Hi. Not directly answer but I've noticed days ago weird peaks at -10, 10 and 35K while cheking signal histograms. This 0 to 10K make me think on it. Could be somehow related?
", "rawMarkdown": "Hi. Not directly answer but I've noticed days ago weird peaks at -10, 10 and 35K while cheking signal histograms. This 0 to 10K make me think on it. Could be somehow related?", "votes": null }, { "id": "2663618", "postDate": "02/22/2024 15:41:30", "content": "I think that they could definitely be related. I'm working on a solution to cut values that simply aren't biological, e.g. over 1000 uV for instance.
", "rawMarkdown": "I think that they could definitely be related. I'm working on a solution to cut values that simply aren't biological, e.g. over 1000 uV for instance.", "votes": null }, { "id": "2663694", "postDate": "02/22/2024 16:35:42", "content": "Many public notebooks are clamping large values at some height to remove artifacts. I've also found companding algorithms like mu-law encoding helpful. This implies that one doesn't need the full dynamic range of the EEG recordings to make better predictions. Finally, are you windowing your EEG signal before filtering? This should help reduce some of the ringing you are seeing in the examples you have shown, although it's not going to go away completely unless those sharp discontinuities are removed somehow.
", "rawMarkdown": "Many public notebooks are clamping large values at some height to remove artifacts. I've also found companding algorithms like [mu-law encoding](https://en.wikipedia.org/wiki/%CE%9C-law_algorithm) helpful. This implies that one doesn't need the full dynamic range of the EEG recordings to make better predictions. Finally, are you windowing your EEG signal before filtering? This should help reduce some of the ringing you are seeing in the examples you have shown, although it's not going to go away completely unless those sharp discontinuities are removed somehow.", "votes": null }, { "id": "2663732", "postDate": "02/22/2024 16:50:43", "content": "Hello @m000sey, first of all, I would like to commend you on your insightful post. Like you, I have encountered these artificial artifacts in the data. Initially unsure of the best approach, I experimented with removing them, only to find that it adversely affected performance. I am inclined to believe there must be a more effective method to address this issue and I'm eager to learn from the community's collective wisdom on this matter.
", "rawMarkdown": "Hello @m000sey, first of all, I would like to commend you on your insightful post. Like you, I have encountered these artificial artifacts in the data. Initially unsure of the best approach, I experimented with removing them, only to find that it adversely affected performance. I am inclined to believe there must be a more effective method to address this issue and I'm eager to learn from the community's collective wisdom on this matter.", "votes": null }, { "id": "2665680", "postDate": "02/23/2024 19:36:40", "content": "Thanks, Rafael. Soon we'll figure this out!
", "rawMarkdown": "Thanks, Rafael. Soon we'll figure this out!", "votes": null }, { "id": "2665682", "postDate": "02/23/2024 19:39:18", "content": "Really helpful ideas, @djhuth. I'll do some more reading on mu-law. Is windowing the same as Epoching? If so, we're working on understanding the best way to do this.
\nMy main concern about clipping is that it may throw off the timings of events in certain channels. However, if the noise is happening in all channels, this is not an issue.
", "rawMarkdown": "Really helpful ideas, @djhuth. I'll do some more reading on mu-law. Is windowing the same as Epoching? If so, we're working on understanding the best way to do this. \n\nMy main concern about clipping is that it may throw off the timings of events in certain channels. However, if the noise is happening in all channels, this is not an issue.", "votes": null }, { "id": "2665780", "postDate": "02/23/2024 20:28:33", "content": "By windowing I mean multiplying the EEG signals with something like a Hann or Blackman window. There are a number of window functions used in signal processing, but my intuition is that the choice of window isn't too important in a deep learning context. What's important is that if a filter is applied to the EEGs then the filtering is done cleanly without introducing this ringing we see in your filtered examples which a model may confuse as a signal and windowing can help with that. Keep in mind that the ringing will remain at some level unless the discontinuities are removed. Step functions like that contain power at all frequencies.
", "rawMarkdown": "By windowing I mean multiplying the EEG signals with something like a [Hann](https://en.wikipedia.org/wiki/Hann_function) or Blackman window. There are [a number](https://en.wikipedia.org/wiki/Window_function) of window functions used in signal processing, but my intuition is that the choice of window isn't too important in a deep learning context. What's important is that if a filter is applied to the EEGs then the filtering is done cleanly without introducing this ringing we see in your filtered examples which a model may confuse as a signal and windowing can help with that. Keep in mind that the ringing will remain at some level unless the discontinuities are removed. Step functions like that contain power at all frequencies.", "votes": null }, { "id": "2666088", "postDate": "02/24/2024 06:20:22", "content": "We tried a filter but it didn't work for us. We aren't dealing with them right now.
", "rawMarkdown": "We tried a filter but it didn't work for us. We aren't dealing with them right now.", "votes": null } ], "comments": [ { "id": 2662463, "author_name": "sacuscreed", "author_url": "", "post_date": "02/21/2024 22:41:08", "content": "Hi. Not directly answer but I've noticed days ago weird peaks at -10, 10 and 35K while cheking signal histograms. This 0 to 10K make me think on it. Could be somehow related?
", "votes": null, "replies": [ { "id": 2663618, "author_name": "m000sey", "author_url": "", "post_date": "02/22/2024 15:41:30", "content": "I think that they could definitely be related. I'm working on a solution to cut values that simply aren't biological, e.g. over 1000 uV for instance.
", "votes": null, "replies": [] } ] }, { "id": 2663694, "author_name": "djhuth", "author_url": "", "post_date": "02/22/2024 16:35:42", "content": "Many public notebooks are clamping large values at some height to remove artifacts. I've also found companding algorithms like mu-law encoding helpful. This implies that one doesn't need the full dynamic range of the EEG recordings to make better predictions. Finally, are you windowing your EEG signal before filtering? This should help reduce some of the ringing you are seeing in the examples you have shown, although it's not going to go away completely unless those sharp discontinuities are removed somehow.
", "votes": null, "replies": [ { "id": 2665682, "author_name": "m000sey", "author_url": "", "post_date": "02/23/2024 19:39:18", "content": "Really helpful ideas, @djhuth. I'll do some more reading on mu-law. Is windowing the same as Epoching? If so, we're working on understanding the best way to do this.
\nMy main concern about clipping is that it may throw off the timings of events in certain channels. However, if the noise is happening in all channels, this is not an issue.
", "votes": null, "replies": [ { "id": 2665780, "author_name": "djhuth", "author_url": "", "post_date": "02/23/2024 20:28:33", "content": "By windowing I mean multiplying the EEG signals with something like a Hann or Blackman window. There are a number of window functions used in signal processing, but my intuition is that the choice of window isn't too important in a deep learning context. What's important is that if a filter is applied to the EEGs then the filtering is done cleanly without introducing this ringing we see in your filtered examples which a model may confuse as a signal and windowing can help with that. Keep in mind that the ringing will remain at some level unless the discontinuities are removed. Step functions like that contain power at all frequencies.
", "votes": null, "replies": [] } ] } ] }, { "id": 2663732, "author_name": "rafaelzimmermann1", "author_url": "", "post_date": "02/22/2024 16:50:43", "content": "Hello @m000sey, first of all, I would like to commend you on your insightful post. Like you, I have encountered these artificial artifacts in the data. Initially unsure of the best approach, I experimented with removing them, only to find that it adversely affected performance. I am inclined to believe there must be a more effective method to address this issue and I'm eager to learn from the community's collective wisdom on this matter.
", "votes": null, "replies": [ { "id": 2665680, "author_name": "m000sey", "author_url": "", "post_date": "02/23/2024 19:36:40", "content": "Thanks, Rafael. Soon we'll figure this out!
", "votes": null, "replies": [] } ] }, { "id": 2666088, "author_name": "gunesevitan", "author_url": "", "post_date": "02/24/2024 06:20:22", "content": "We tried a filter but it didn't work for us. We aren't dealing with them right now.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2662312": "Hi all, \n\nThank you everyone for sharing so much useful information in this competition. I'm learning so much about signal processing. \n\n**Problem:** for some eeg_ids, I identify artifactual (fake) noise, e.g. the figure attached below.\n\n\n\n**Question:** How are you folks dealing with examples like this? \n\nMy first inclination is to design a function that iterates through all rows to flag EEG_ids that have this sort of artifact. For instance, iterate through each eeg channel and identify windows where the standard deviation of values is very close to zero, or stark/not-biological changes in slope. Then I would delete this artifactual elbow-like noise from the EEG dataset.\n\nHow are you folks going about this problem? All input or thoughts are appreciated!\nCheers,\nShane", "2662463": "Hi. Not directly answer but I've noticed days ago weird peaks at -10, 10 and 35K while cheking signal histograms. This 0 to 10K make me think on it. Could be somehow related?", "2663618": "I think that they could definitely be related. I'm working on a solution to cut values that simply aren't biological, e.g. over 1000 uV for instance.", "2663694": "Many public notebooks are clamping large values at some height to remove artifacts. I've also found companding algorithms like [mu-law encoding](https://en.wikipedia.org/wiki/%CE%9C-law_algorithm) helpful. This implies that one doesn't need the full dynamic range of the EEG recordings to make better predictions. Finally, are you windowing your EEG signal before filtering? This should help reduce some of the ringing you are seeing in the examples you have shown, although it's not going to go away completely unless those sharp discontinuities are removed somehow.", "2663732": "Hello @m000sey, first of all, I would like to commend you on your insightful post. Like you, I have encountered these artificial artifacts in the data. Initially unsure of the best approach, I experimented with removing them, only to find that it adversely affected performance. I am inclined to believe there must be a more effective method to address this issue and I'm eager to learn from the community's collective wisdom on this matter.", "2665680": "Thanks, Rafael. Soon we'll figure this out!", "2665682": "Really helpful ideas, @djhuth. I'll do some more reading on mu-law. Is windowing the same as Epoching? If so, we're working on understanding the best way to do this. \n\nMy main concern about clipping is that it may throw off the timings of events in certain channels. However, if the noise is happening in all channels, this is not an issue.", "2665780": "By windowing I mean multiplying the EEG signals with something like a [Hann](https://en.wikipedia.org/wiki/Hann_function) or Blackman window. There are [a number](https://en.wikipedia.org/wiki/Window_function) of window functions used in signal processing, but my intuition is that the choice of window isn't too important in a deep learning context. What's important is that if a filter is applied to the EEGs then the filtering is done cleanly without introducing this ringing we see in your filtered examples which a model may confuse as a signal and windowing can help with that. Keep in mind that the ringing will remain at some level unless the discontinuities are removed. Step functions like that contain power at all frequencies.", "2666088": "We tried a filter but it didn't work for us. We aren't dealing with them right now." }, "source": "meta" }