{ "id": 492273, "title": "Any insights or improvements because of signal processing?", "url": "/competitions/hms-harmful-brain-activity-classification/discussion/492273", "author_name": "", "post_date": "2024-04-09T05:09:59.165042Z", "votes": 2, "comment_count": 3, "views": 0, "content": "

i got really interested by signal processing because of this competition.
\ni can see people mostly use band pass filter of 0.5 ~ 20Hz.
\n is there any different filters or delicate signal processing techniques that made good improvements?

", "messages": [ { "id": "2742798", "postDate": "04/09/2024 05:09:59", "content": "

i got really interested by signal processing because of this competition.
\ni can see people mostly use band pass filter of 0.5 ~ 20Hz.
\n is there any different filters or delicate signal processing techniques that made good improvements?

", "rawMarkdown": "i got really interested by signal processing because of this competition. \ni can see people mostly use band pass filter of 0.5 ~ 20Hz.\n is there any different filters or delicate signal processing techniques that made good improvements?", "votes": null }, { "id": "2744136", "postDate": "04/09/2024 19:16:23", "content": "

Yes. I think signal processing was important. When I used

\n
 mne.filter import filter_data, notch_filter\nsample = data.T[[0,4,5,6, 11,15,16,17, 0,1,2,3, 11,12,13,14]]\\\n         - data.T[[4,5,6,7, 15,16,17,18, 1,2,3,7, 12,13,14,18]]\nsample = notch_filter(sample.astype(), 200, 60, =32, =)\nsample = filter_data(sample.astype(), 200, 0.5, 40, =32, =) \nsample = np.clip(sample,-500,500)\nsample = np.nan_to_num(sample, =0)\n
\n

I noticed a reduction in the gap between CV (with vote>=10) and LB score. In other words, before this I signal processed only using low pass filter as in my public notebooks. Then I used the filter above. The CV score did not change but the LB score improved by 0.03 wow! (so the gap between CV LB got smaller).

\n

I'm not sure why this happens. It surprised me. Perhaps there is something different about test versus train and using the above preprocess helps everything generalize and avoid differences between train and test.

", "rawMarkdown": "Yes. I think signal processing was important. When I used\n\n from mne.filter import filter_data, notch_filter\n sample = data.T[[0,4,5,6, 11,15,16,17, 0,1,2,3, 11,12,13,14]]\\\n - data.T[[4,5,6,7, 15,16,17,18, 1,2,3,7, 12,13,14,18]]\n sample = notch_filter(sample.astype('float64'), 200, 60, n_jobs=32, verbose='ERROR')\n sample = filter_data(sample.astype('float64'), 200, 0.5, 40, n_jobs=32, verbose='ERROR') \n sample = np.clip(sample,-500,500)\n sample = np.nan_to_num(sample, nan=0)\n\nI noticed a reduction in the gap between CV (with vote>=10) and LB score. In other words, before this I signal processed only using low pass filter as in my public notebooks. Then I used the filter above. The CV score did not change but the LB score improved by `0.03` wow! (so the gap between CV LB got smaller).\n\nI'm not sure why this happens. It surprised me. Perhaps there is something different about test versus train and using the above preprocess helps everything generalize and avoid differences between train and test.", "votes": null }, { "id": "2745015", "postDate": "04/10/2024 09:52:15", "content": "
\n

In North America, the notch filter is set at 60 Hz to remove electrical line interference

\n
\n

60Hz with width 1Hz makes sense to remove artifacts.
\nBTW thx for your great notebooks. It helped me a lot to learn from this competition. Would love to see you again in another competition soon!!!

", "rawMarkdown": ">In North America, the notch filter is set at 60 Hz to remove electrical line interference\n\n60Hz with width 1Hz makes sense to remove artifacts.\nBTW thx for your great notebooks. It helped me a lot to learn from this competition. Would love to see you again in another competition soon!!!", "votes": null }, { "id": "2745055", "postDate": "04/10/2024 10:40:01", "content": "

For my experiments 0-40 perform better

", "rawMarkdown": "For my experiments 0-40 perform better", "votes": null } ], "comments": [ { "id": 2744136, "author_name": "cdeotte", "author_url": "", "post_date": "04/09/2024 19:16:23", "content": "

Yes. I think signal processing was important. When I used

\n
 mne.filter import filter_data, notch_filter\nsample = data.T[[0,4,5,6, 11,15,16,17, 0,1,2,3, 11,12,13,14]]\\\n         - data.T[[4,5,6,7, 15,16,17,18, 1,2,3,7, 12,13,14,18]]\nsample = notch_filter(sample.astype(), 200, 60, =32, =)\nsample = filter_data(sample.astype(), 200, 0.5, 40, =32, =) \nsample = np.clip(sample,-500,500)\nsample = np.nan_to_num(sample, =0)\n
\n

I noticed a reduction in the gap between CV (with vote>=10) and LB score. In other words, before this I signal processed only using low pass filter as in my public notebooks. Then I used the filter above. The CV score did not change but the LB score improved by 0.03 wow! (so the gap between CV LB got smaller).

\n

I'm not sure why this happens. It surprised me. Perhaps there is something different about test versus train and using the above preprocess helps everything generalize and avoid differences between train and test.

", "votes": null, "replies": [ { "id": 2745015, "author_name": "beckpro", "author_url": "", "post_date": "04/10/2024 09:52:15", "content": "
\n

In North America, the notch filter is set at 60 Hz to remove electrical line interference

\n
\n

60Hz with width 1Hz makes sense to remove artifacts.
\nBTW thx for your great notebooks. It helped me a lot to learn from this competition. Would love to see you again in another competition soon!!!

", "votes": null, "replies": [] } ] }, { "id": 2745055, "author_name": "goldenlock", "author_url": "", "post_date": "04/10/2024 10:40:01", "content": "

For my experiments 0-40 perform better

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2742798": "i got really interested by signal processing because of this competition. \ni can see people mostly use band pass filter of 0.5 ~ 20Hz.\n is there any different filters or delicate signal processing techniques that made good improvements?", "2744136": "Yes. I think signal processing was important. When I used\n\n from mne.filter import filter_data, notch_filter\n sample = data.T[[0,4,5,6, 11,15,16,17, 0,1,2,3, 11,12,13,14]]\\\n - data.T[[4,5,6,7, 15,16,17,18, 1,2,3,7, 12,13,14,18]]\n sample = notch_filter(sample.astype('float64'), 200, 60, n_jobs=32, verbose='ERROR')\n sample = filter_data(sample.astype('float64'), 200, 0.5, 40, n_jobs=32, verbose='ERROR') \n sample = np.clip(sample,-500,500)\n sample = np.nan_to_num(sample, nan=0)\n\nI noticed a reduction in the gap between CV (with vote>=10) and LB score. In other words, before this I signal processed only using low pass filter as in my public notebooks. Then I used the filter above. The CV score did not change but the LB score improved by `0.03` wow! (so the gap between CV LB got smaller).\n\nI'm not sure why this happens. It surprised me. Perhaps there is something different about test versus train and using the above preprocess helps everything generalize and avoid differences between train and test.", "2745015": ">In North America, the notch filter is set at 60 Hz to remove electrical line interference\n\n60Hz with width 1Hz makes sense to remove artifacts.\nBTW thx for your great notebooks. It helped me a lot to learn from this competition. Would love to see you again in another competition soon!!!", "2745055": "For my experiments 0-40 perform better" }, "source": "meta" }