{ "id": 468684, "title": "UPDATED - WaveNet Starter Notebook - LB 0.52 - Raw EEG Features Only!", "url": "/competitions/hms-harmful-brain-activity-classification/discussion/468684", "author_name": "Chris Deotte", "post_date": "2024-01-17T15:12:14.215000", "votes": 142, "comment_count": 47, "views": 0, "content": "

I am excited to have built a successful model which only trains with raw EEG features here. This model is WaveNet. WaveNet is a convolution neural network which uses dilations to spread out the convolutions and locate the presence of frequencies in waveforms. The research paper is here. WaveNet is a powerful model which has contributed to many Kaggle competition winning solutions.

\n

Example: How WaveNet Finds 5 Hz

\n

When we apply a convolution with kernel_size=3 and dilation_rate=20 to a raw EEG waveform (that is sampled at 200Hz), then this convolution can detect 5Hz.

\n
tf.keras.layers.\n
\n

This is because wherever we apply this convolution, the 3 inspection points of the convolution have the same value when the EEG waveform has 5Hz present. This is show in the two example images below:

\n


\n

\n

Butterworth Low-Pass Filter

\n

In this competition, we (most likely) are only concerned with brain waves of 20Hz and below. Therefore the presence of brain frequencies above 20Hz will confuse WaveNet. We solve this by applying a butterworth low-pass filter. Below shows examples of using a Butterworth filter with cutoff frequencies 16, 8, 4, 2, 1 for illustration purposes.

\n

\n

Example: How Spectrograms Find 5 Hz

\n

A spectrogram is an image which represents a raw EEG waveform. Each row of a spectrogram image represents a specific frequency. Therefore to detect 5Hz with a spectrogram image, we simply need to take the mean across the row that represents 5Hz:

\n

\n

Starter Notebook

\n

The TensorFlow WaveNet starter notebook is here. And the PyTorch version (converted by @alejopaullier ) is here

\n

UPDATE

\n

In version 5 and 6, we use 2 EEG features and achieve CV 0.91 LB 0.66. In version 7 and 8 we use 8 EEG features and achieve CV 0.81 LB 0.52, wow! 🥳. The motivation for version 7 and 8's features is the new Magic Formula described here. The Magic Formula implies that we should process each montage chain separately and then combine the results. The new architecture for WaveNet follows this paradigm.
\n

\n

Enjoy

\n

Enjoy and happy modeling!

", "messages": [ { "id": 2606338, "postDate": "2024-01-17T15:12:14.217Z", "content": "

I am excited to have built a successful model which only trains with raw EEG features here. This model is WaveNet. WaveNet is a convolution neural network which uses dilations to spread out the convolutions and locate the presence of frequencies in waveforms. The research paper is here. WaveNet is a powerful model which has contributed to many Kaggle competition winning solutions.

\n

Example: How WaveNet Finds 5 Hz

\n

When we apply a convolution with kernel_size=3 and dilation_rate=20 to a raw EEG waveform (that is sampled at 200Hz), then this convolution can detect 5Hz.

\n
tf.keras.layers.\n
\n

This is because wherever we apply this convolution, the 3 inspection points of the convolution have the same value when the EEG waveform has 5Hz present. This is show in the two example images below:

\n


\n

\n

Butterworth Low-Pass Filter

\n

In this competition, we (most likely) are only concerned with brain waves of 20Hz and below. Therefore the presence of brain frequencies above 20Hz will confuse WaveNet. We solve this by applying a butterworth low-pass filter. Below shows examples of using a Butterworth filter with cutoff frequencies 16, 8, 4, 2, 1 for illustration purposes.

\n

\n

Example: How Spectrograms Find 5 Hz

\n

A spectrogram is an image which represents a raw EEG waveform. Each row of a spectrogram image represents a specific frequency. Therefore to detect 5Hz with a spectrogram image, we simply need to take the mean across the row that represents 5Hz:

\n

\n

Starter Notebook

\n

The TensorFlow WaveNet starter notebook is here. And the PyTorch version (converted by @alejopaullier ) is here

\n

UPDATE

\n

In version 5 and 6, we use 2 EEG features and achieve CV 0.91 LB 0.66. In version 7 and 8 we use 8 EEG features and achieve CV 0.81 LB 0.52, wow! 🥳. The motivation for version 7 and 8's features is the new Magic Formula described here. The Magic Formula implies that we should process each montage chain separately and then combine the results. The new architecture for WaveNet follows this paradigm.
\n

\n

Enjoy

\n

Enjoy and happy modeling!

", "rawMarkdown": "I am excited to have built a successful model which only trains with raw EEG features [here][2]. This model is WaveNet. WaveNet is a convolution neural network which uses dilations to spread out the convolutions and locate the presence of frequencies in waveforms. The research paper is [here][1]. WaveNet is a powerful model which has contributed to many Kaggle competition winning solutions.\n\n# Example: How WaveNet Finds 5 Hz\nWhen we apply a convolution with `kernel_size=3` and `dilation_rate=20` to a raw EEG waveform (that is sampled at 200Hz), then this convolution can detect 5Hz. \n\n tf.keras.layers.Conv1D(filters, kernel_size=3, dilation_rate=20)\n\nThis is because wherever we apply this convolution, the 3 inspection points of the convolution have the same value when the EEG waveform has 5Hz present. This is show in the two example images below:\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/wave6.png)\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/wave7.png)\n\n# Butterworth Low-Pass Filter\nIn this competition, we (most likely) are only concerned with brain waves of 20Hz and below. Therefore the presence of brain frequencies above 20Hz will confuse WaveNet. We solve this by applying a butterworth low-pass filter. Below shows examples of using a Butterworth filter with cutoff frequencies 16, 8, 4, 2, 1 for illustration purposes.\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/butter.png)\n\n# Example: How Spectrograms Find 5 Hz\nA spectrogram is an image which represents a raw EEG waveform. Each row of a spectrogram image represents a specific frequency. Therefore to detect 5Hz with a spectrogram image, we simply need to take the mean across the row that represents 5Hz:\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/wave3.png)\n\n# Starter Notebook\nThe TensorFlow WaveNet starter notebook is [here][2]. And the PyTorch version (converted by @alejopaullier ) is [here][4]\n\n# UPDATE\nIn version 5 and 6, we use 2 EEG features and achieve CV 0.91 LB 0.66. In version 7 and 8 we use 8 EEG features and achieve **CV 0.81 LB 0.52**, wow! 🥳. The motivation for version 7 and 8's features is the new Magic Formula described [here][3]. The Magic Formula implies that we should process each montage chain separately and then combine the results. The new architecture for WaveNet follows this paradigm.\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/wave-model.png)\n\n# Enjoy\nEnjoy and happy modeling!\n\n[1]: https://arxiv.org/abs/1609.03499\n[2]: https://www.kaggle.com/code/cdeotte/wavenet-starter-lb-0-70\n[3]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469760\n[4]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477610", "votes": 141 }, { "id": 2616776, "postDate": "2024-01-23T20:19:42.880Z", "content": "

UPDATE I added more features to WaveNet and updated the architecture. Version 7 and 8 now achieve CV 0.81 LB 0.52. Woohoo!

", "rawMarkdown": "**UPDATE** I added more features to WaveNet and updated the architecture. Version 7 and 8 now achieve CV 0.81 LB 0.52. Woohoo!", "votes": 8, "replies": [ { "id": 2616977, "postDate": "2024-01-24T02:12:29.097Z", "content": "

Hi @cdeotte one clarification required .I see dataloader outputs 10000,8 size but model input is 2000,8 ,why and how it is achieved, not able to figure out..

", "rawMarkdown": "Hi @cdeotte one clarification required .I see dataloader outputs 10000,8 size but model input is 2000,8 ,why and how it is achieved, not able to figure out..", "votes": 1, "replies": [ { "id": 2616983, "postDate": "2024-01-24T02:19:28.250Z", "content": "

The dataloader outputs a downsample, see code below

\n
 ():\n    \n    indexes = .indexes[index*.(index+)*.batch_size]\n    X, y = .__data_generation(indexes)\n     X[,.downsample,], y\n
", "rawMarkdown": "The dataloader outputs a downsample, see code below\n\n def __getitem__(self, index):\n 'Generate one batch of data'\n indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n X, y = self.__data_generation(indexes)\n return X[:,::self.downsample,:], y", "votes": 2 } ] } ] }, { "id": 2628498, "postDate": "2024-01-31T10:39:23.327Z", "content": "

Thank you for the Wavenet starter, just improved it from 0.52 to 0.51. Now I'll take a look if I can improve the efficientnet starter. Hopefully I can try an ensemble soon to pierce into the medal region.

", "rawMarkdown": "Thank you for the Wavenet starter, just improved it from 0.52 to 0.51. Now I'll take a look if I can improve the efficientnet starter. Hopefully I can try an ensemble soon to pierce into the medal region.", "votes": 3 }, { "id": 2611716, "postDate": "2024-01-20T23:05:37.543Z", "content": "

Is WaveNet limited to N frequencies, where N is the number of dilation rates?

\n

For example, in the WaveNet start notebook here, there are 12 dilation rates in the first WaveBlock. I calculated the matched frequencies using def dil2hz(dil_rate): return (200*0.5)/dil_rate. This gives us 12 frequencies.

\n

Dilation rates: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048]
\nFrequencies: [100.0, 50.0, 25.0, 12.5, 6.25, 3.125, 1.562, 0.781, 0.391, 0.195, 0.098, 0.049]

\n

Can WaveNet identify frequencies other than the 12 above. I am struggling to see how it can identify other frequencies?

", "rawMarkdown": "Is WaveNet limited to N frequencies, where N is the number of dilation rates?\n\nFor example, in the WaveNet start notebook [here](https://www.kaggle.com/code/cdeotte/wavenet-starter-lb-0-70), there are 12 dilation rates in the first WaveBlock. I calculated the matched frequencies using `def dil2hz(dil_rate): return (200*0.5)/dil_rate`. This gives us 12 frequencies.\n\nDilation rates: `[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048]`\nFrequencies: `[100.0, 50.0, 25.0, 12.5, 6.25, 3.125, 1.562, 0.781, 0.391, 0.195, 0.098, 0.049]`\n\nCan WaveNet identify frequencies other than the 12 above. I am struggling to see how it can identify other frequencies?", "votes": 3, "replies": [ { "id": 2614175, "postDate": "2024-01-22T13:32:43.590Z", "content": "

Hi @brendanartley yes it can. Whenever we input a frequency into WaveNet, we will get \"feedback\" from the 12 different convolutions. Even though none of them by itself detects the frequency perfectly, the deep learning model will learn when it see pattern A in convolution 1, and pattern B in convolution 2, and pattern C in convolution 3, etc etc then the frequency is 4Hz or whatever.

\n

In other words, we can say the \"WaveNet has 12 eyeballs\" and although it doesn't see perfectly. It can deduce any frequency from what it sees with its \"12 eyeballs\". Obviously this is simplistic explanation, but hopefully you get my point.

", "rawMarkdown": "Hi @brendanartley yes it can. Whenever we input a frequency into WaveNet, we will get \"feedback\" from the 12 different convolutions. Even though none of them by itself detects the frequency perfectly, the deep learning model will learn when it see pattern A in convolution 1, and pattern B in convolution 2, and pattern C in convolution 3, etc etc then the frequency is 4Hz or whatever.\n\nIn other words, we can say the \"WaveNet has 12 eyeballs\" and although it doesn't see perfectly. It can deduce any frequency from what it sees with its \"12 eyeballs\". Obviously this is simplistic explanation, but hopefully you get my point.", "votes": 5 } ] }, { "id": 2607480, "postDate": "2024-01-18T08:58:27.007Z", "content": "

@cdeotte Thank you for sharing your thoughts. If we are only interested in frequency <= 20Hz, the sampling rate 200Hz seems much higher than we need. Do you think downsampling, say, to 40Hz is a good way to reduce the data size?

", "rawMarkdown": "@cdeotte Thank you for sharing your thoughts. If we are only interested in frequency <= 20Hz, the sampling rate 200Hz seems much higher than we need. Do you think downsampling, say, to 40Hz is a good way to reduce the data size?", "votes": 3, "replies": [ { "id": 2607867, "postDate": "2024-01-18T13:38:23.560Z", "content": "

That's a great idea. We can do data = data[:,::5,:] and reduce data from (batch, 10_000, features) => (batch, 2_000, features) which is sampling 200Hz => 40Hz. This will help RNN and Transformers work better (because they don't like sequences of length 10_000.

\n

We can also try this with WaveNet (i.e. CNN), but CNN can handle longer sequences better than RNN and Transformers.

", "rawMarkdown": "That's a great idea. We can do `data = data[:,::5,:]` and reduce data from `(batch, 10_000, features)` => `(batch, 2_000, features)` which is sampling 200Hz => 40Hz. This will help RNN and Transformers work better (because they don't like sequences of length 10_000.\n\nWe can also try this with WaveNet (i.e. CNN), but CNN can handle longer sequences better than RNN and Transformers.", "votes": 2 }, { "id": 2608635, "postDate": "2024-01-19T01:21:01.473Z", "content": "

You may wanna filter out the 60Hz (sometimes 50Hz) power line noise with a symmetrical notch filter before downsampling.

", "rawMarkdown": "You may wanna filter out the 60Hz (sometimes 50Hz) power line noise with a symmetrical notch filter before downsampling.", "votes": 4 }, { "id": 2620300, "postDate": "2024-01-26T03:48:56.570Z", "content": "

Related to @tolgadincer 's comment, to reduce noise when downsampling to 40 Hz (for a Nyquist frequency of 20 Hz), it woud be good to first do a digital low-pass filter on the 200 Hz data and then apply the data = data[:,::5,:] downsampling. I did a little googling and didn't find what I felt was a good page about it, but the second answer on this Signal Processing Stack Exchange is short and useful:
\n\"If you're using scipy.signal and processing signals offline, then you can just use decimate which handles the filtering for you. It also does zero-phase filtering by default, which you probably want for an EEG signal to avoid shifting the shape of the waveforms? (I know that's desirable for EKG, not sure about EEG.)\"
\nNote that even just smoothing by applying a width-5 uniform window, e.g. with pd.Series.rolling(), is a simple form of lowpass filtering and should improve the downsampling result.

", "rawMarkdown": "Related to @tolgadincer 's comment, to reduce noise when downsampling to 40 Hz (for a Nyquist frequency of 20 Hz), it woud be good to first do a digital low-pass filter on the 200 Hz data and then apply the `data = data[:,::5,:]` downsampling. I did a little googling and didn't find what I felt was a good page about it, but the second answer on this [Signal Processing Stack Exchange](https://dsp.stackexchange.com/questions/73161/how-to-apply-an-anti-aliasing-filter-before-downsampling) is short and useful:\n\"If you're using scipy.signal and processing signals offline, then you can just use [`decimate`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.decimate.html) which handles the filtering for you. It also does zero-phase filtering by default, which you probably want for an EEG signal to avoid shifting the shape of the waveforms? (I know that's desirable for EKG, not sure about EEG.)\" \nNote that even just smoothing by applying a width-5 uniform window, e.g. with [pd.Series.rolling()](https://pandas.pydata.org/docs/reference/api/pandas.Series.rolling.html), is a simple form of lowpass filtering and should improve the downsampling result.", "votes": 5 } ] }, { "id": 2606473, "postDate": "2024-01-17T16:59:11.763Z", "content": "

Inb4 final solution is stacked ensemble of 10 seed 10 fold wavenet, efnet, resnet, swim, catboost, transformer, lgb, xgb

\n

😅 nice to see lots of things work

", "rawMarkdown": "Inb4 final solution is stacked ensemble of 10 seed 10 fold wavenet, efnet, resnet, swim, catboost, transformer, lgb, xgb\n\n😅 nice to see lots of things work", "votes": 3, "replies": [ { "id": 2606504, "postDate": "2024-01-17T17:26:06.463Z", "content": "

Haha, yeah. I think mega ensembles will do good in this competition. Also a multimodal model that can take all the data (spectrogram and eeg waveform and stacked over GBT output) and have the benefits of CNN, RNN, self attention will make very strong \"single\" models.

", "rawMarkdown": "Haha, yeah. I think mega ensembles will do good in this competition. Also a multimodal model that can take all the data (spectrogram and eeg waveform and stacked over GBT output) and have the benefits of CNN, RNN, self attention will make very strong \"single\" models.", "votes": 4, "replies": [ { "id": 2607285, "postDate": "2024-01-18T06:25:27.400Z", "content": "

Wow, thank you so much for your contribution in this competition, providing so much high-quality open source code for us to learn, especially when I saw that you may use multimodal models, which is a method I can hardly imagine and implement. Although I have not touched them, I am still excited to follow your footsteps to learn them 🥰.

", "rawMarkdown": "Wow, thank you so much for your contribution in this competition, providing so much high-quality open source code for us to learn, especially when I saw that you may use multimodal models, which is a method I can hardly imagine and implement. Although I have not touched them, I am still excited to follow your footsteps to learn them 🥰.", "votes": 1 } ] } ] }, { "id": 2606825, "postDate": "2024-01-17T21:42:02.040Z", "content": "

Great post, thank you! Just curious how you know \"we are only concerned with brain waves of 20Hz and below\"?

", "rawMarkdown": "Great post, thank you! Just curious how you know \"we are only concerned with brain waves of 20Hz and below\"?", "votes": 4, "replies": [ { "id": 2606833, "postDate": "2024-01-17T21:54:31.657Z", "content": "

We don't know this for sure. But the current best public notebook uses Kaggle spectrograms. And Kaggle spectrograms only have information for 20Hz and below. I will run some experiments with higher frequencies to see if it helps.

", "rawMarkdown": "We don't know this for sure. But the current best public notebook uses Kaggle spectrograms. And Kaggle spectrograms only have information for 20Hz and below. I will run some experiments with higher frequencies to see if it helps.", "votes": 6, "replies": [ { "id": 2606836, "postDate": "2024-01-17T21:58:57.823Z", "content": "

Cool insight, thanks!

", "rawMarkdown": "Cool insight, thanks!", "votes": 1 }, { "id": 2618192, "postDate": "2024-01-24T16:01:32.033Z", "content": "

Unless I am missing something the events are all < 20Hz

", "rawMarkdown": "Unless I am missing something the events are all < 20Hz", "votes": 2 } ] } ] }, { "id": 2666150, "postDate": "2024-02-24T07:12:41.553Z", "content": "

HI @cdeotte ,

\n

Thank you as usual for the great notebook.
\nI have dabbled with WaveNet previously, but this is a new level of depth.
\nAs a sanity check, in your 5 Hz example, how would you fetch the outputs at the \"3 inspection points of the convolution\" to indeed confirm that the values are the same? Do we have to inspect the kernel outputs themselves?

", "rawMarkdown": "HI @cdeotte ,\n\nThank you as usual for the great notebook.\nI have dabbled with WaveNet previously, but this is a new level of depth.\nAs a sanity check, in your 5 Hz example, how would you fetch the outputs at the \"3 inspection points of the convolution\" to indeed confirm that the values are the same? Do we have to inspect the kernel outputs themselves?\n\n\n\n", "votes": 1, "replies": [ { "id": 2667255, "postDate": "2024-02-25T02:48:00.283Z", "content": "

Hi, thanks. We do not need to do anything. When we use a convolution with kernel_size = 3, then the model uses 3 inspection points to compute a new value (that it passes to the next layer of our NN model). Without dilation, these 3 points will be consecutive time steps. With dilations, they are spaced as shown in the diagram. The model will do this and learn how to use these 3 points during training.

", "rawMarkdown": "Hi, thanks. We do not need to do anything. When we use a convolution with `kernel_size = 3`, then the model uses 3 inspection points to compute a new value (that it passes to the next layer of our NN model). Without dilation, these 3 points will be consecutive time steps. With dilations, they are spaced as shown in the diagram. The model will do this and learn how to use these 3 points during training.", "votes": 1, "replies": [ { "id": 2667267, "postDate": "2024-02-25T03:01:15.313Z", "content": "

Thanks for this, think I had a brainfart over how 1DCNNs work

", "rawMarkdown": "Thanks for this, think I had a brainfart over how 1DCNNs work", "votes": 1 } ] } ] }, { "id": 2607687, "postDate": "2024-01-18T11:31:07.327Z", "content": "

Wow, by reading your discussions I believe there are limitless ways to be creative in this competition, one of the most interesting competitions I've seen in a while for sure. Thank you for sharing :)

\n

@cdeotte you are on fire! Great work yet again!

", "rawMarkdown": "Wow, by reading your discussions I believe there are limitless ways to be creative in this competition, one of the most interesting competitions I've seen in a while for sure. Thank you for sharing :)\n\n@cdeotte you are on fire! Great work yet again!\n", "votes": 1, "replies": [ { "id": 2607863, "postDate": "2024-01-18T13:36:52.977Z", "content": "

Yes I love competitions where there are many successful modeling options. This competition will allow for lots of creativity. I'm excited. Enjoy!

", "rawMarkdown": "Yes I love competitions where there are many successful modeling options. This competition will allow for lots of creativity. I'm excited. Enjoy!", "votes": 1 } ] }, { "id": 2606501, "postDate": "2024-01-17T17:22:18.003Z", "content": "

@cdeotte you are on fire! Great work yet again!

", "rawMarkdown": "@cdeotte you are on fire! Great work yet again!", "votes": 1 }, { "id": 2606484, "postDate": "2024-01-17T17:12:04.523Z", "content": "

hi @cdeotte , Thank you for sharing. The discussion certainly makes a simpler way to start with the competition although I am sure there is more to it than meets the eye. Thanks!

", "rawMarkdown": "hi @cdeotte , Thank you for sharing. The discussion certainly makes a simpler way to start with the competition although I am sure there is more to it than meets the eye. Thanks!", "votes": 1 }, { "id": 2606392, "postDate": "2024-01-17T15:53:34.420Z", "content": "

Interesting post. Thanks for all the sharing you have done this competition.

\n

Why do you think this approach has a large CV/LB gap?

", "rawMarkdown": "Interesting post. Thanks for all the sharing you have done this competition. \n\nWhy do you think this approach has a large CV/LB gap?", "votes": 1, "replies": [ { "id": 2606436, "postDate": "2024-01-17T16:18:04.030Z", "content": "

Here are the CV LB of my 4 starter notebooks:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelDataCVLBGap
EfficientNetSpectrogram0.720.560.16
WaveNetEEG Waveform0.910.660.25
CatBoostSpectrogram0.670.820.15
MLPEEG Spectrogram1.030.770.26
\n

Kaggle provides Spectrograms and EEGs. It appears that there is a larger CV LB gap when using EEG features. This may because the EEG features are more \"raw\". Maybe Kaggle standardized the Spectrograms in a way to make them generalize better.

", "rawMarkdown": "Here are the CV LB of my 4 starter notebooks:\n\n| Model | Data | CV | LB | Gap |\n| --- | --- |\n| EfficientNet | Spectrogram | 0.72 | 0.56 | 0.16 |\n| WaveNet | EEG Waveform | 0.91 | 0.66 | 0.25 |\n| CatBoost | Spectrogram | 0.67 | 0.82 | 0.15 |\n| MLP | EEG Spectrogram | 1.03 | 0.77 | 0.26 |\n\nKaggle provides Spectrograms and EEGs. It appears that there is a larger CV LB gap when using EEG features. This may because the EEG features are more \"raw\". Maybe Kaggle standardized the Spectrograms in a way to make them generalize better. \n", "votes": 5, "replies": [ { "id": 2606991, "postDate": "2024-01-18T02:02:11.577Z", "content": "

@cdeotte If I'm not mistaken, you had posted a discussion post and a notebook of your MLP. Did you delete it, or am I crazy?

", "rawMarkdown": "@cdeotte If I'm not mistaken, you had posted a discussion post and a notebook of your MLP. Did you delete it, or am I crazy?", "votes": 1 }, { "id": 2606995, "postDate": "2024-01-18T02:10:11.430Z", "content": "

Kaggle deleted it. In the post, I asked for people to view my notebook and upvote my dataset which I learned is against Community Guidelines. I received a notification in my account and an email reminding me of the rules below:

\n
\n

Sharing self-promoting posts in other forums or in comments may result in the removal of that content and a warning may be issued to the author.

\n

Requesting upvotes or suggesting that other users should view or “check out” your work is considered upvote begging and is not allowed. Comments or posts of this nature may be removed and a warning may be issued to the author.

\n
", "rawMarkdown": "Kaggle deleted it. In the post, I asked for people to view my notebook and upvote my dataset which I learned is against [Community Guidelines][1]. I received a notification in my account and an email reminding me of the rules below:\n\n>Sharing self-promoting posts in other forums or in comments may result in the removal of that content and a warning may be issued to the author.\n\n>Requesting upvotes or suggesting that other users should view or “check out” your work is considered upvote begging and is not allowed. Comments or posts of this nature may be removed and a warning may be issued to the author.\n\n[1]: https://www.kaggle.com/community-guidelines", "votes": 3 }, { "id": 2612951, "postDate": "2024-01-21T18:12:18.643Z", "content": "

Hi @cdeotte . Is it possible that you can publish that notebook again ? It would be very much great if you do it.

", "rawMarkdown": "Hi @cdeotte . Is it possible that you can publish that notebook again ? It would be very much great if you do it. ", "votes": 1 }, { "id": 2612965, "postDate": "2024-01-21T18:20:03.083Z", "content": "

The notebook still exists, only the discussion post referring to it was deleted. My MLP starter is in version 1-3 of my notebook here

", "rawMarkdown": "The notebook still exists, only the discussion post referring to it was deleted. My MLP starter is in version 1-3 of my notebook [here][1]\n\n[1]: https://www.kaggle.com/code/cdeotte/how-to-make-spectrogram-from-eeg", "votes": 2 }, { "id": 2613968, "postDate": "2024-01-22T11:23:26.220Z", "content": "

Thank you so much for this.

", "rawMarkdown": "Thank you so much for this. " } ] } ] }, { "id": 2656264, "postDate": "2024-02-17T15:06:10.127Z", "content": "

Hi @cdeotte,

\n

I'm a beginner exploring your WaveNet implementation and have a question about dilated causal convolutions. While using conv1D, it seems the convolution process considers both past and future values around each point.

\n

However, the WaveNet paper specifies using dilated causal convolutions, where predictions at each point should only depend on dilated past points. I'm likely missing something in the implementation, and I'd love to hear your thinking about this.

\n

While we are using WaveNet for brain state classification (not next-point prediction), maybe dilated causal convolution doesn't need to be implemented because also the target is aggregated from multiple samples in preprocessing and it is same for each point in the wave.

\n

I am asking because through your code I want to understand model more clearly.

\n

Thank you for sharing your amazing resources!

", "rawMarkdown": "Hi @cdeotte,\n\nI'm a beginner exploring your WaveNet implementation and have a question about dilated causal convolutions. While using conv1D, it seems the convolution process considers both past and future values around each point.\n\nHowever, the WaveNet paper specifies using dilated causal convolutions, where predictions at each point should only depend on dilated past points. I'm likely missing something in the implementation, and I'd love to hear your thinking about this.\n\nWhile we are using WaveNet for brain state classification (not next-point prediction), maybe dilated causal convolution doesn't need to be implemented because also the target is aggregated from multiple samples in preprocessing and it is same for each point in the wave. \n\nI am asking because through your code I want to understand model more clearly.\n\nThank you for sharing your amazing resources!", "votes": 2, "replies": [ { "id": 2656272, "postDate": "2024-02-17T15:13:22.717Z", "content": "

If we wanted to detect EEG events in real time (i.e. is there an EEG event happening now in the present moment), then yes we could only use causal convolutions and use past information. However in this competition, that is not what Kaggle is asking us to do.

\n

Kaggle is asking us to use both 5 minutes of before information and 5 minutes of after information to determine whether there is an EEG event in the middle 10 seconds. Since we have both past and future information, we will use it. Therefore in my WaveNet implementation, I use a convolution that uses both past and future information.

", "rawMarkdown": "If we wanted to detect EEG events in real time (i.e. is there an EEG event happening now in the present moment), then yes we could only use causal convolutions and use past information. However in this competition, that is **not** what Kaggle is asking us to do.\n\nKaggle is asking us to use both 5 minutes of **before** information and 5 minutes of **after** information to determine whether there is an EEG event in the middle 10 seconds. Since we have both past and future information, we will use it. Therefore in my WaveNet implementation, I use a convolution that uses both past and future information.", "votes": 7 } ] }, { "id": 2617602, "postDate": "2024-01-24T10:43:54.583Z", "content": "

@cdeotte sorry to bother you. Can you share the resource for wavenet where I can read more about it.

", "rawMarkdown": "@cdeotte sorry to bother you. Can you share the resource for wavenet where I can read more about it.", "votes": 2, "replies": [ { "id": 2619435, "postDate": "2024-01-25T12:49:46.550Z", "content": "

Here is the research paper here

", "rawMarkdown": "Here is the research paper [here][1]\n\n[1]: https://arxiv.org/abs/1609.03499", "votes": 3 } ] }, { "id": 2613309, "postDate": "2024-01-22T02:15:07.617Z", "content": "

@cdeotte Really appreciate your work and I have some questions. The raw waveform is sampled at 200Hz and dilation_rate is set to 20. I thus assume that 10 kernels will be chosen every second. Since every kernel correspond to one phase of a complete wave, I assume this thus corresponds to a 10Hz wave rather than a 5Hz wave. I must have misunderstood the illustration in the post😭

", "rawMarkdown": "@cdeotte Really appreciate your work and I have some questions. The raw waveform is sampled at 200Hz and `dilation_rate` is set to `20`. I thus assume that 10 kernels will be chosen every second. Since every kernel correspond to one phase of a complete wave, I assume this thus corresponds to a **10Hz** wave rather than a **5Hz** wave. I must have misunderstood the illustration in the post😭", "votes": 2, "replies": [ { "id": 2614167, "postDate": "2024-01-22T13:29:01.033Z", "content": "

Hi. In my diagram, the dilation=20 is the distance from the green mid point to the green end point. The diagram shows kernel_size=3 which has 1 mid point and two end points. Thus the full length of the convolution is:

\n
 * (kernel_size-) =  * (-) = \n
\n

So there are 5 = 200/40 complete kernels when place end to end in one second. So this kernel size = 3 with dilation = 20 will detect 5Hz. Note it will also detect 10Hz, 15Hz, 20Hz, etc, etc because all multiples of 5Hz will have the same phenomena as having same y value at inspection points.

", "rawMarkdown": "Hi. In my diagram, the `dilation=20` is the distance from the green mid point to the green end point. The diagram shows `kernel_size=3` which has 1 mid point and two end points. Thus the full length of the convolution is:\n\n dilation * (kernel_size-1) = 20 * (3-1) = 40\n\nSo there are `5 = 200/40` complete kernels when place end to end in one second. So this `kernel size = 3` with `dilation = 20` will detect 5Hz. Note it will also detect 10Hz, 15Hz, 20Hz, etc, etc because all multiples of 5Hz will have the same phenomena as having same y value at inspection points.", "votes": 9 } ] }, { "id": 2610489, "postDate": "2024-01-20T06:49:25.303Z", "content": "

Thank you, Chris, for sharing your WaveNet Starter Notebook and its impressive results in leveraging raw EEG features. It's fascinating how you've utilized WaveNet's dilation property to effectively detect specific frequencies in EEG waveforms, and the addition of the Butterworth low-pass filter to refine the model's focus is a smart touch. Your approach offers valuable insights into the capabilities of convolutional neural networks in handling waveform data.

\n

I'm curious about the potential unexpected outcomes when applying this model in real-world scenarios. Have you encountered any funny or surprising interpretations by the model, especially given the complex and often unpredictable nature of EEG data? Sharing such anecdotes could not only provide a few laughs but also offer valuable lessons for the Kaggle community on the quirks and challenges of working with neural network models and EEG data.

\n

Best,
\nNishant

", "rawMarkdown": "Thank you, Chris, for sharing your WaveNet Starter Notebook and its impressive results in leveraging raw EEG features. It's fascinating how you've utilized WaveNet's dilation property to effectively detect specific frequencies in EEG waveforms, and the addition of the Butterworth low-pass filter to refine the model's focus is a smart touch. Your approach offers valuable insights into the capabilities of convolutional neural networks in handling waveform data.\n\nI'm curious about the potential unexpected outcomes when applying this model in real-world scenarios. Have you encountered any funny or surprising interpretations by the model, especially given the complex and often unpredictable nature of EEG data? Sharing such anecdotes could not only provide a few laughs but also offer valuable lessons for the Kaggle community on the quirks and challenges of working with neural network models and EEG data.\n\nBest,\nNishant", "votes": 2, "replies": [ { "id": 2610522, "postDate": "2024-01-20T07:21:07.450Z", "content": "

When building models from raw waveforms (in this comp and past comps), i have found WaveNet to be powerful and successful.

", "rawMarkdown": "When building models from raw waveforms (in this comp and past comps), i have found WaveNet to be powerful and successful.", "votes": 2 } ] }, { "id": 2606457, "postDate": "2024-01-17T16:39:26.250Z", "content": "

Wow, by reading your discussions I believe there are limitless ways to be creative in this competition, one of the most interesting competitions I've seen in a while for sure. Thank you for sharing :)

", "rawMarkdown": "Wow, by reading your discussions I believe there are limitless ways to be creative in this competition, one of the most interesting competitions I've seen in a while for sure. Thank you for sharing :) ", "votes": 2, "replies": [ { "id": 2606462, "postDate": "2024-01-17T16:43:31.317Z", "content": "

Yes and this is only the beginning. These models are still simple. The winning model will take both Spectrograms and EGG Waveforms as input and use sophisticated self attention to allow the model to attend to related parts between Spectrogram and EEG Waveform.

", "rawMarkdown": "Yes and this is only the beginning. These models are still **simple**. The winning model will take both Spectrograms and EGG Waveforms as input and use sophisticated self attention to allow the model to attend to related parts between Spectrogram and EEG Waveform. ", "votes": 4, "replies": [ { "id": 2618155, "postDate": "2024-01-24T15:36:14.313Z", "content": "

This contest is so interesting! maybe the machine learning will pick up subtleties in the raw data that the spectrogram doesn't show?

", "rawMarkdown": "This contest is so interesting! maybe the machine learning will pick up subtleties in the raw data that the spectrogram doesn't show? ", "votes": 1 } ] } ] }, { "id": 2624942, "postDate": "2024-01-29T05:03:16.427Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 2624944, "postDate": "2024-01-29T05:07:35.883Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 2624937, "postDate": "2024-01-29T05:01:41.167Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2619361, "postDate": "2024-01-25T12:16:39.403Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 2606618, "postDate": "2024-01-17T18:51:31.273Z", "content": "

@cdeotte Thanks for sharing wavnet

", "rawMarkdown": "@cdeotte Thanks for sharing wavnet", "votes": 2 } ], "comments": [ { "id": 2616776, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-01-23T20:19:42.880000", "content": "

UPDATE I added more features to WaveNet and updated the architecture. Version 7 and 8 now achieve CV 0.81 LB 0.52. Woohoo!

", "votes": 8, "replies": [ { "id": 2616977, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2024-01-24T02:12:29.097000", "content": "

Hi @cdeotte one clarification required .I see dataloader outputs 10000,8 size but model input is 2000,8 ,why and how it is achieved, not able to figure out..

", "votes": 1, "replies": [ { "id": 2616983, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-01-24T02:19:28.250000", "content": "

The dataloader outputs a downsample, see code below

\n
 ():\n    \n    indexes = .indexes[index*.(index+)*.batch_size]\n    X, y = .__data_generation(indexes)\n     X[,.downsample,], y\n
", "votes": 2, "replies": [] } ] } ] }, { "id": 2628498, "author_name": "stefanoclss", "author_url": "", "post_date": "2024-01-31T10:39:23.327000", "content": "

Thank you for the Wavenet starter, just improved it from 0.52 to 0.51. Now I'll take a look if I can improve the efficientnet starter. Hopefully I can try an ensemble soon to pierce into the medal region.

", "votes": 3, "replies": [] }, { "id": 2611716, "author_name": "Bartley", "author_url": "", "post_date": "2024-01-20T23:05:37.543000", "content": "

Is WaveNet limited to N frequencies, where N is the number of dilation rates?

\n

For example, in the WaveNet start notebook here, there are 12 dilation rates in the first WaveBlock. I calculated the matched frequencies using def dil2hz(dil_rate): return (200*0.5)/dil_rate. This gives us 12 frequencies.

\n

Dilation rates: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048]
\nFrequencies: [100.0, 50.0, 25.0, 12.5, 6.25, 3.125, 1.562, 0.781, 0.391, 0.195, 0.098, 0.049]

\n

Can WaveNet identify frequencies other than the 12 above. I am struggling to see how it can identify other frequencies?

", "votes": 3, "replies": [ { "id": 2614175, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-01-22T13:32:43.590000", "content": "

Hi @brendanartley yes it can. Whenever we input a frequency into WaveNet, we will get \"feedback\" from the 12 different convolutions. Even though none of them by itself detects the frequency perfectly, the deep learning model will learn when it see pattern A in convolution 1, and pattern B in convolution 2, and pattern C in convolution 3, etc etc then the frequency is 4Hz or whatever.

\n

In other words, we can say the \"WaveNet has 12 eyeballs\" and although it doesn't see perfectly. It can deduce any frequency from what it sees with its \"12 eyeballs\". Obviously this is simplistic explanation, but hopefully you get my point.

", "votes": 5, "replies": [] } ] }, { "id": 2607480, "author_name": "Taichi Uemura", "author_url": "", "post_date": "2024-01-18T08:58:27.007000", "content": "

@cdeotte Thank you for sharing your thoughts. If we are only interested in frequency <= 20Hz, the sampling rate 200Hz seems much higher than we need. Do you think downsampling, say, to 40Hz is a good way to reduce the data size?

", "votes": 3, "replies": [ { "id": 2607867, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-01-18T13:38:23.560000", "content": "

That's a great idea. We can do data = data[:,::5,:] and reduce data from (batch, 10_000, features) => (batch, 2_000, features) which is sampling 200Hz => 40Hz. This will help RNN and Transformers work better (because they don't like sequences of length 10_000.

\n

We can also try this with WaveNet (i.e. CNN), but CNN can handle longer sequences better than RNN and Transformers.

", "votes": 2, "replies": [] }, { "id": 2608635, "author_name": "Tolga", "author_url": "", "post_date": "2024-01-19T01:21:01.473000", "content": "

You may wanna filter out the 60Hz (sometimes 50Hz) power line noise with a symmetrical notch filter before downsampling.

", "votes": 4, "replies": [] }, { "id": 2620300, "author_name": "Daniel Dewey", "author_url": "", "post_date": "2024-01-26T03:48:56.570000", "content": "

Related to @tolgadincer 's comment, to reduce noise when downsampling to 40 Hz (for a Nyquist frequency of 20 Hz), it woud be good to first do a digital low-pass filter on the 200 Hz data and then apply the data = data[:,::5,:] downsampling. I did a little googling and didn't find what I felt was a good page about it, but the second answer on this Signal Processing Stack Exchange is short and useful:
\n\"If you're using scipy.signal and processing signals offline, then you can just use decimate which handles the filtering for you. It also does zero-phase filtering by default, which you probably want for an EEG signal to avoid shifting the shape of the waveforms? (I know that's desirable for EKG, not sure about EEG.)\"
\nNote that even just smoothing by applying a width-5 uniform window, e.g. with pd.Series.rolling(), is a simple form of lowpass filtering and should improve the downsampling result.

", "votes": 5, "replies": [] } ] }, { "id": 2606473, "author_name": "JM", "author_url": "", "post_date": "2024-01-17T16:59:11.763000", "content": "

Inb4 final solution is stacked ensemble of 10 seed 10 fold wavenet, efnet, resnet, swim, catboost, transformer, lgb, xgb

\n

😅 nice to see lots of things work

", "votes": 3, "replies": [ { "id": 2606504, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-01-17T17:26:06.463000", "content": "

Haha, yeah. I think mega ensembles will do good in this competition. Also a multimodal model that can take all the data (spectrogram and eeg waveform and stacked over GBT output) and have the benefits of CNN, RNN, self attention will make very strong \"single\" models.

", "votes": 4, "replies": [ { "id": 2607285, "author_name": "Orzlala", "author_url": "", "post_date": "2024-01-18T06:25:27.400000", "content": "

Wow, thank you so much for your contribution in this competition, providing so much high-quality open source code for us to learn, especially when I saw that you may use multimodal models, which is a method I can hardly imagine and implement. Although I have not touched them, I am still excited to follow your footsteps to learn them 🥰.

", "votes": 1, "replies": [] } ] } ] }, { "id": 2606825, "author_name": "mk0052", "author_url": "", "post_date": "2024-01-17T21:42:02.040000", "content": "

Great post, thank you! Just curious how you know \"we are only concerned with brain waves of 20Hz and below\"?

", "votes": 4, "replies": [ { "id": 2606833, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-01-17T21:54:31.657000", "content": "

We don't know this for sure. But the current best public notebook uses Kaggle spectrograms. And Kaggle spectrograms only have information for 20Hz and below. I will run some experiments with higher frequencies to see if it helps.

", "votes": 6, "replies": [ { "id": 2606836, "author_name": "mk0052", "author_url": "", "post_date": "2024-01-17T21:58:57.823000", "content": "

Cool insight, thanks!

", "votes": 1, "replies": [] }, { "id": 2618192, "author_name": "Edward Tuck", "author_url": "", "post_date": "2024-01-24T16:01:32.033000", "content": "

Unless I am missing something the events are all < 20Hz

", "votes": 2, "replies": [] } ] } ] }, { "id": 2666150, "author_name": "Yijie Xu", "author_url": "", "post_date": "2024-02-24T07:12:41.553000", "content": "

HI @cdeotte ,

\n

Thank you as usual for the great notebook.
\nI have dabbled with WaveNet previously, but this is a new level of depth.
\nAs a sanity check, in your 5 Hz example, how would you fetch the outputs at the \"3 inspection points of the convolution\" to indeed confirm that the values are the same? Do we have to inspect the kernel outputs themselves?

", "votes": 1, "replies": [ { "id": 2667255, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-02-25T02:48:00.283000", "content": "

Hi, thanks. We do not need to do anything. When we use a convolution with kernel_size = 3, then the model uses 3 inspection points to compute a new value (that it passes to the next layer of our NN model). Without dilation, these 3 points will be consecutive time steps. With dilations, they are spaced as shown in the diagram. The model will do this and learn how to use these 3 points during training.

", "votes": 1, "replies": [ { "id": 2667267, "author_name": "Yijie Xu", "author_url": "", "post_date": "2024-02-25T03:01:15.313000", "content": "

Thanks for this, think I had a brainfart over how 1DCNNs work

", "votes": 1, "replies": [] } ] } ] }, { "id": 2607687, "author_name": "Tanishq dublish", "author_url": "", "post_date": "2024-01-18T11:31:07.327000", "content": "

Wow, by reading your discussions I believe there are limitless ways to be creative in this competition, one of the most interesting competitions I've seen in a while for sure. Thank you for sharing :)

\n

@cdeotte you are on fire! Great work yet again!

", "votes": 1, "replies": [ { "id": 2607863, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-01-18T13:36:52.977000", "content": "

Yes I love competitions where there are many successful modeling options. This competition will allow for lots of creativity. I'm excited. Enjoy!

", "votes": 1, "replies": [] } ] }, { "id": 2606501, "author_name": "Cody_Null", "author_url": "", "post_date": "2024-01-17T17:22:18.003000", "content": "

@cdeotte you are on fire! Great work yet again!

", "votes": 1, "replies": [] }, { "id": 2606484, "author_name": "supplejade", "author_url": "", "post_date": "2024-01-17T17:12:04.523000", "content": "

hi @cdeotte , Thank you for sharing. The discussion certainly makes a simpler way to start with the competition although I am sure there is more to it than meets the eye. Thanks!

", "votes": 1, "replies": [] }, { "id": 2606392, "author_name": "Bartley", "author_url": "", "post_date": "2024-01-17T15:53:34.420000", "content": "

Interesting post. Thanks for all the sharing you have done this competition.

\n

Why do you think this approach has a large CV/LB gap?

", "votes": 1, "replies": [ { "id": 2606436, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-01-17T16:18:04.030000", "content": "

Here are the CV LB of my 4 starter notebooks:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelDataCVLBGap
EfficientNetSpectrogram0.720.560.16
WaveNetEEG Waveform0.910.660.25
CatBoostSpectrogram0.670.820.15
MLPEEG Spectrogram1.030.770.26
\n

Kaggle provides Spectrograms and EEGs. It appears that there is a larger CV LB gap when using EEG features. This may because the EEG features are more \"raw\". Maybe Kaggle standardized the Spectrograms in a way to make them generalize better.

", "votes": 5, "replies": [ { "id": 2606991, "author_name": "Yan Teixeira", "author_url": "", "post_date": "2024-01-18T02:02:11.577000", "content": "

@cdeotte If I'm not mistaken, you had posted a discussion post and a notebook of your MLP. Did you delete it, or am I crazy?

", "votes": 1, "replies": [] }, { "id": 2606995, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-01-18T02:10:11.430000", "content": "

Kaggle deleted it. In the post, I asked for people to view my notebook and upvote my dataset which I learned is against Community Guidelines. I received a notification in my account and an email reminding me of the rules below:

\n
\n

Sharing self-promoting posts in other forums or in comments may result in the removal of that content and a warning may be issued to the author.

\n

Requesting upvotes or suggesting that other users should view or “check out” your work is considered upvote begging and is not allowed. Comments or posts of this nature may be removed and a warning may be issued to the author.

\n
", "votes": 3, "replies": [] }, { "id": 2612951, "author_name": "JamshaidSohail", "author_url": "", "post_date": "2024-01-21T18:12:18.643000", "content": "

Hi @cdeotte . Is it possible that you can publish that notebook again ? It would be very much great if you do it.

", "votes": 1, "replies": [] }, { "id": 2612965, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-01-21T18:20:03.083000", "content": "

The notebook still exists, only the discussion post referring to it was deleted. My MLP starter is in version 1-3 of my notebook here

", "votes": 2, "replies": [] }, { "id": 2613968, "author_name": "JamshaidSohail", "author_url": "", "post_date": "2024-01-22T11:23:26.220000", "content": "

Thank you so much for this.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2656264, "author_name": "Matej Duvnjak", "author_url": "", "post_date": "2024-02-17T15:06:10.127000", "content": "

Hi @cdeotte,

\n

I'm a beginner exploring your WaveNet implementation and have a question about dilated causal convolutions. While using conv1D, it seems the convolution process considers both past and future values around each point.

\n

However, the WaveNet paper specifies using dilated causal convolutions, where predictions at each point should only depend on dilated past points. I'm likely missing something in the implementation, and I'd love to hear your thinking about this.

\n

While we are using WaveNet for brain state classification (not next-point prediction), maybe dilated causal convolution doesn't need to be implemented because also the target is aggregated from multiple samples in preprocessing and it is same for each point in the wave.

\n

I am asking because through your code I want to understand model more clearly.

\n

Thank you for sharing your amazing resources!

", "votes": 2, "replies": [ { "id": 2656272, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-02-17T15:13:22.717000", "content": "

If we wanted to detect EEG events in real time (i.e. is there an EEG event happening now in the present moment), then yes we could only use causal convolutions and use past information. However in this competition, that is not what Kaggle is asking us to do.

\n

Kaggle is asking us to use both 5 minutes of before information and 5 minutes of after information to determine whether there is an EEG event in the middle 10 seconds. Since we have both past and future information, we will use it. Therefore in my WaveNet implementation, I use a convolution that uses both past and future information.

", "votes": 7, "replies": [] } ] }, { "id": 2617602, "author_name": "Shashank Shukla", "author_url": "", "post_date": "2024-01-24T10:43:54.583000", "content": "

@cdeotte sorry to bother you. Can you share the resource for wavenet where I can read more about it.

", "votes": 2, "replies": [ { "id": 2619435, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-01-25T12:49:46.550000", "content": "

Here is the research paper here

", "votes": 3, "replies": [] } ] }, { "id": 2613309, "author_name": "Roy Wei", "author_url": "", "post_date": "2024-01-22T02:15:07.617000", "content": "

@cdeotte Really appreciate your work and I have some questions. The raw waveform is sampled at 200Hz and dilation_rate is set to 20. I thus assume that 10 kernels will be chosen every second. Since every kernel correspond to one phase of a complete wave, I assume this thus corresponds to a 10Hz wave rather than a 5Hz wave. I must have misunderstood the illustration in the post😭

", "votes": 2, "replies": [ { "id": 2614167, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-01-22T13:29:01.033000", "content": "

Hi. In my diagram, the dilation=20 is the distance from the green mid point to the green end point. The diagram shows kernel_size=3 which has 1 mid point and two end points. Thus the full length of the convolution is:

\n
 * (kernel_size-) =  * (-) = \n
\n

So there are 5 = 200/40 complete kernels when place end to end in one second. So this kernel size = 3 with dilation = 20 will detect 5Hz. Note it will also detect 10Hz, 15Hz, 20Hz, etc, etc because all multiples of 5Hz will have the same phenomena as having same y value at inspection points.

", "votes": 9, "replies": [] } ] }, { "id": 2610489, "author_name": "Nishant Singhal", "author_url": "", "post_date": "2024-01-20T06:49:25.303000", "content": "

Thank you, Chris, for sharing your WaveNet Starter Notebook and its impressive results in leveraging raw EEG features. It's fascinating how you've utilized WaveNet's dilation property to effectively detect specific frequencies in EEG waveforms, and the addition of the Butterworth low-pass filter to refine the model's focus is a smart touch. Your approach offers valuable insights into the capabilities of convolutional neural networks in handling waveform data.

\n

I'm curious about the potential unexpected outcomes when applying this model in real-world scenarios. Have you encountered any funny or surprising interpretations by the model, especially given the complex and often unpredictable nature of EEG data? Sharing such anecdotes could not only provide a few laughs but also offer valuable lessons for the Kaggle community on the quirks and challenges of working with neural network models and EEG data.

\n

Best,
\nNishant

", "votes": 2, "replies": [ { "id": 2610522, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-01-20T07:21:07.450000", "content": "

When building models from raw waveforms (in this comp and past comps), i have found WaveNet to be powerful and successful.

", "votes": 2, "replies": [] } ] }, { "id": 2606457, "author_name": "Catharis", "author_url": "", "post_date": "2024-01-17T16:39:26.250000", "content": "

Wow, by reading your discussions I believe there are limitless ways to be creative in this competition, one of the most interesting competitions I've seen in a while for sure. Thank you for sharing :)

", "votes": 2, "replies": [ { "id": 2606462, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-01-17T16:43:31.317000", "content": "

Yes and this is only the beginning. These models are still simple. The winning model will take both Spectrograms and EGG Waveforms as input and use sophisticated self attention to allow the model to attend to related parts between Spectrogram and EEG Waveform.

", "votes": 4, "replies": [ { "id": 2618155, "author_name": "Edward Tuck", "author_url": "", "post_date": "2024-01-24T15:36:14.313000", "content": "

This contest is so interesting! maybe the machine learning will pick up subtleties in the raw data that the spectrogram doesn't show?

", "votes": 1, "replies": [] } ] } ] }, { "id": 2624942, "author_name": "", "author_url": "", "post_date": "2024-01-29T05:03:16.427000", "content": "", "votes": 0, "replies": [ { "id": 2624944, "author_name": "", "author_url": "", "post_date": "2024-01-29T05:07:35.883000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2624937, "author_name": "", "author_url": "", "post_date": "2024-01-29T05:01:41.167000", "content": "", "votes": 0, "replies": [] }, { "id": 2619361, "author_name": "", "author_url": "", "post_date": "2024-01-25T12:16:39.403000", "content": "", "votes": 1, "replies": [] }, { "id": 2606618, "author_name": "SeshuRaju 🧘‍♂️", "author_url": "", "post_date": "2024-01-17T18:51:31.273000", "content": "

@cdeotte Thanks for sharing wavnet

", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "2606338": "I am excited to have built a successful model which only trains with raw EEG features [here][2]. This model is WaveNet. WaveNet is a convolution neural network which uses dilations to spread out the convolutions and locate the presence of frequencies in waveforms. The research paper is [here][1]. WaveNet is a powerful model which has contributed to many Kaggle competition winning solutions.\n\n# Example: How WaveNet Finds 5 Hz\nWhen we apply a convolution with `kernel_size=3` and `dilation_rate=20` to a raw EEG waveform (that is sampled at 200Hz), then this convolution can detect 5Hz. \n\n tf.keras.layers.Conv1D(filters, kernel_size=3, dilation_rate=20)\n\nThis is because wherever we apply this convolution, the 3 inspection points of the convolution have the same value when the EEG waveform has 5Hz present. This is show in the two example images below:\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/wave6.png)\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/wave7.png)\n\n# Butterworth Low-Pass Filter\nIn this competition, we (most likely) are only concerned with brain waves of 20Hz and below. Therefore the presence of brain frequencies above 20Hz will confuse WaveNet. We solve this by applying a butterworth low-pass filter. Below shows examples of using a Butterworth filter with cutoff frequencies 16, 8, 4, 2, 1 for illustration purposes.\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/butter.png)\n\n# Example: How Spectrograms Find 5 Hz\nA spectrogram is an image which represents a raw EEG waveform. Each row of a spectrogram image represents a specific frequency. Therefore to detect 5Hz with a spectrogram image, we simply need to take the mean across the row that represents 5Hz:\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/wave3.png)\n\n# Starter Notebook\nThe TensorFlow WaveNet starter notebook is [here][2]. And the PyTorch version (converted by @alejopaullier ) is [here][4]\n\n# UPDATE\nIn version 5 and 6, we use 2 EEG features and achieve CV 0.91 LB 0.66. In version 7 and 8 we use 8 EEG features and achieve **CV 0.81 LB 0.52**, wow! 🥳. The motivation for version 7 and 8's features is the new Magic Formula described [here][3]. The Magic Formula implies that we should process each montage chain separately and then combine the results. The new architecture for WaveNet follows this paradigm.\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/wave-model.png)\n\n# Enjoy\nEnjoy and happy modeling!\n\n[1]: https://arxiv.org/abs/1609.03499\n[2]: https://www.kaggle.com/code/cdeotte/wavenet-starter-lb-0-70\n[3]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469760\n[4]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477610", "2616776": "**UPDATE** I added more features to WaveNet and updated the architecture. Version 7 and 8 now achieve CV 0.81 LB 0.52. Woohoo!", "2628498": "Thank you for the Wavenet starter, just improved it from 0.52 to 0.51. Now I'll take a look if I can improve the efficientnet starter. Hopefully I can try an ensemble soon to pierce into the medal region.", "2611716": "Is WaveNet limited to N frequencies, where N is the number of dilation rates?\n\nFor example, in the WaveNet start notebook [here](https://www.kaggle.com/code/cdeotte/wavenet-starter-lb-0-70), there are 12 dilation rates in the first WaveBlock. I calculated the matched frequencies using `def dil2hz(dil_rate): return (200*0.5)/dil_rate`. This gives us 12 frequencies.\n\nDilation rates: `[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048]`\nFrequencies: `[100.0, 50.0, 25.0, 12.5, 6.25, 3.125, 1.562, 0.781, 0.391, 0.195, 0.098, 0.049]`\n\nCan WaveNet identify frequencies other than the 12 above. I am struggling to see how it can identify other frequencies?", "2607480": "@cdeotte Thank you for sharing your thoughts. If we are only interested in frequency <= 20Hz, the sampling rate 200Hz seems much higher than we need. Do you think downsampling, say, to 40Hz is a good way to reduce the data size?", "2606473": "Inb4 final solution is stacked ensemble of 10 seed 10 fold wavenet, efnet, resnet, swim, catboost, transformer, lgb, xgb\n\n😅 nice to see lots of things work", "2606825": "Great post, thank you! Just curious how you know \"we are only concerned with brain waves of 20Hz and below\"?", "2666150": "HI @cdeotte ,\n\nThank you as usual for the great notebook.\nI have dabbled with WaveNet previously, but this is a new level of depth.\nAs a sanity check, in your 5 Hz example, how would you fetch the outputs at the \"3 inspection points of the convolution\" to indeed confirm that the values are the same? Do we have to inspect the kernel outputs themselves?\n\n\n\n", "2607687": "Wow, by reading your discussions I believe there are limitless ways to be creative in this competition, one of the most interesting competitions I've seen in a while for sure. Thank you for sharing :)\n\n@cdeotte you are on fire! Great work yet again!\n", "2606501": "@cdeotte you are on fire! Great work yet again!", "2606484": "hi @cdeotte , Thank you for sharing. The discussion certainly makes a simpler way to start with the competition although I am sure there is more to it than meets the eye. Thanks!", "2606392": "Interesting post. Thanks for all the sharing you have done this competition. \n\nWhy do you think this approach has a large CV/LB gap?", "2656264": "Hi @cdeotte,\n\nI'm a beginner exploring your WaveNet implementation and have a question about dilated causal convolutions. While using conv1D, it seems the convolution process considers both past and future values around each point.\n\nHowever, the WaveNet paper specifies using dilated causal convolutions, where predictions at each point should only depend on dilated past points. I'm likely missing something in the implementation, and I'd love to hear your thinking about this.\n\nWhile we are using WaveNet for brain state classification (not next-point prediction), maybe dilated causal convolution doesn't need to be implemented because also the target is aggregated from multiple samples in preprocessing and it is same for each point in the wave. \n\nI am asking because through your code I want to understand model more clearly.\n\nThank you for sharing your amazing resources!", "2617602": "@cdeotte sorry to bother you. Can you share the resource for wavenet where I can read more about it.", "2613309": "@cdeotte Really appreciate your work and I have some questions. The raw waveform is sampled at 200Hz and `dilation_rate` is set to `20`. I thus assume that 10 kernels will be chosen every second. Since every kernel correspond to one phase of a complete wave, I assume this thus corresponds to a **10Hz** wave rather than a **5Hz** wave. I must have misunderstood the illustration in the post😭", "2610489": "Thank you, Chris, for sharing your WaveNet Starter Notebook and its impressive results in leveraging raw EEG features. It's fascinating how you've utilized WaveNet's dilation property to effectively detect specific frequencies in EEG waveforms, and the addition of the Butterworth low-pass filter to refine the model's focus is a smart touch. Your approach offers valuable insights into the capabilities of convolutional neural networks in handling waveform data.\n\nI'm curious about the potential unexpected outcomes when applying this model in real-world scenarios. Have you encountered any funny or surprising interpretations by the model, especially given the complex and often unpredictable nature of EEG data? Sharing such anecdotes could not only provide a few laughs but also offer valuable lessons for the Kaggle community on the quirks and challenges of working with neural network models and EEG data.\n\nBest,\nNishant", "2606457": "Wow, by reading your discussions I believe there are limitless ways to be creative in this competition, one of the most interesting competitions I've seen in a while for sure. Thank you for sharing :) ", "2624942": "", "2624937": "", "2619361": "", "2606618": "@cdeotte Thanks for sharing wavnet" } }