![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2422331%2F93297936355b66b3d6b10b6f70a6016c%2Fprediction_pipeline.png?generation=1712796229936116&alt=media\")
Team : KTMUD ( @kazumax0720 @hiroitakafumi @maxchen303 @asaliquid1011 @dimanishi)
\nFirst of all, Thank you to the organizers and Congrats to all the participants and winners!
\nOur team merged with the UEMU&T.H.&kazumax teams and MaxChen303 and D.Imanishi during the competition.
\nWe ensemble the models developed by each team.
\nOur prediciction pipeline is this.
\nEach approaches are will added in the comments!
We borrowed the format from the 1st place solution.
\nThank you, team SONY!
Team : KTMUD ( @kazumax0720 @hiroitakafumi @maxchen303 @asaliquid1011 @dimanishi)
\nFirst of all, Thank you to the organizers and Congrats to all the participants and winners!
\nOur team merged with the UEMU&T.H.&kazumax teams and MaxChen303 and D.Imanishi during the competition.
\nWe ensemble the models developed by each team.
\nOur prediciction pipeline is this.
\nEach approaches are will added in the comments!
We borrowed the format from the 1st place solution.
\nThank you, team SONY!
Thanks to the organizers and congrats to all the winners! I think it was a very exciting and hard competition.
\nAnd I am very happy to have finally achieved my goal of becoming a grandmaster!
2D-CNN Models were trained for two types of input data and ensemble them.
\nThe input image of my model is like following. 19 signals are the same as competition's sample figure.
\n![](\"https://i.postimg.cc/BQHHgFyw/specgraph.png\")
\nI had also taken the approach of converting EEG to spectrogram, but did not use it in the end because the score was better without it in the process of team merging.
Congrats on achieving grandmaster @dimanishi, well deserved!
", "rawMarkdown": "Congrats on achieving grandmaster @dimanishi, well deserved!", "votes": 2 }, { "id": 2746172, "postDate": "2024-04-11T05:07:40.513Z", "content": "Congrats on achieving grandmaster @dimanishi, you deserve it!
", "rawMarkdown": "Congrats on achieving grandmaster @dimanishi, you deserve it!", "votes": 1 } ] }, { "id": 2745233, "postDate": "2024-04-10T13:35:32.363Z", "content": "This is the first time I have made a team with others on Kaggle. Thanks to my teammates for bringing the great experience.
\nThe main idea of my solution is to generate inputs similar to what the annotators see.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3964695%2Fef831958400ea7348dfe6ecebf6c511a%2FScreenshot%20from%202024-04-09%2012-53-16.png?generation=1712681638219917&alt=media\")
Due to a mistake in my HIGH vote CV implementation (accidentally included some low votes samples), I think I didn't fully optimize the performance in the last week.
\nAfter checking the provided EEG pattern images many times, I found that the provided raw EEGs look much cleaner. It was much easier to see the pattern in the provided image than my first few raw EEG plots.
\nIt turns out that simply applying a bandpass filter can make the raw EEGs look much closer to the provided examples.
\nI also noticed that the scipy sosfilt is preferred over the commonly used lfilter according to the document and the sosfiltfilt can reduce the effect of phase shift.
The red boxes in this diagram show the possible phase shifts:
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3964695%2Ff2e4493929bebfd7149582d9e23e8c0b%2FScreenshot%20from%202024-04-09%2012-56-41.png?generation=1712681817517725&alt=media\")
average # LOW votes / average # HIGH votes.Test time augmentation(TTA):
\nPretrained backbones from TIMM with different pooling layers.
\nUEMU & T.H. & kazumax created models as a mini team.
\nI was in charge of the raw EEG part.
\nCV was only calculated for the middle sample when samples from the same eeg_id were lined up in time order.
I developed a wavenet+2dCNN model using only raw EEG.
\nBelow is an outline of my pipeline.
\nI achieved a public LB = 0.25 (cv = 0.244) with a single model.
![\"th_pipeline\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2449267%2Ff01ca637106a7b304d97a892205fbabc%2Fth_pipeline.png?generation=1712759565877010&alt=media\")
thanks for sharing @hiroitakafumi and congrats on achieving 5th place. Is it possible you share your training and inference code?
", "rawMarkdown": "thanks for sharing @hiroitakafumi and congrats on achieving 5th place. Is it possible you share your training and inference code?" }, { "id": 2755394, "postDate": "2024-04-16T13:48:38.570Z", "content": "Thanks for sharing.
\nWhy did you add a CNN Block between WaveBlock?
\nI just want to know the logic to do that.
I wanted to reduce the width of the Wavenet output by including a CNN with stride=2.
\nWithout CNN, the output shape of Wavenet is [64, 10000], but with CNN it becomes [64, 625].
\nThe aim was to improve accuracy by lowering the downsampling rate due to pooling layer.
\nPublic LB improved by about 0.01.
UEMU & T.H. & kazumax created models as a mini team.
\nI was in charge of the spectrogram part.
\nCV was only calculated for the middle sample when samples from the same eeg_id were lined up in time order.
24 spectrograms
\nEEGSpec is a power spectrogram truncated at 20Hz not mel spectrogram.
\nspec = librosa.stft(\n y=x,\n hop_length=(x) // ,\n n_fft=,\n)\nspec = np.(spec) ** \n\n\n\n\nspec = spec[:, :] \n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2422331%2Fdc133f535c8f382e26ed9a77956e3f62%2Fmodel.png?generation=1712754127506152&alt=media\")
Our team (UEMU&kazumax&TH) developed models, each member focusing on a different data type.
\n However, in this competition, it's necessary to integrate the inference values of each data type, similar to what the annotators are doing.
\n Therefore, we created a model that integrates the models for each data type through a fully connected (FC) layer to provide a multifaceted perspective.
\n We created some models of each data type for ensemble, and we also developed this model for all combinations.
As mentioned in this discussion, there are some data where the eeg_id is different but the patient_id is the same in the test dataset.
\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/471890
We aggregated the prediction values for each patient_id to create new features.
\nUsing the original prediction values and these aggregated features, we built a another inference model.
\nThis model did not improve the score on its own, but by including this model in the 2_3.Weight Average, we achieved an improvement in CV of 0.002 to 0.003.
We took a weighted average using all the models(each single model,2_1.FC model and the 2_2.PostProcess, about 50 models).
\nThe weights were optimized based on CV. Although it seemed excessive, the correlation between the LB and CV was good.
The best scores(UEMU&kazumax&TH's part)
\nSimple weighted average of three teams.
\nThe weighted average was taken using the logits before applying the softmax function.
\nThe weights were decided based on the public leaderboard (LB), where the public best score was equal to the private best score.
The best scores(Team KTMUD)
\nCongratulations on achieving 5th place in this competition. Thanks for sharing the insights into your solution.
", "rawMarkdown": "Congratulations on achieving 5th place in this competition. Thanks for sharing the insights into your solution. " } ], "comments": [ { "id": 2745220, "author_name": "D.Imanishi", "author_url": "", "post_date": "2024-04-10T13:18:33.623000", "content": "Thanks to the organizers and congrats to all the winners! I think it was a very exciting and hard competition.
\nAnd I am very happy to have finally achieved my goal of becoming a grandmaster!
2D-CNN Models were trained for two types of input data and ensemble them.
\nThe input image of my model is like following. 19 signals are the same as competition's sample figure.
\n
\nI had also taken the approach of converting EEG to spectrogram, but did not use it in the end because the score was better without it in the process of team merging.
Congrats on achieving grandmaster @dimanishi, well deserved!
", "votes": 2, "replies": [] }, { "id": 2746172, "author_name": "Aaditya Porwal", "author_url": "", "post_date": "2024-04-11T05:07:40.513000", "content": "Congrats on achieving grandmaster @dimanishi, you deserve it!
", "votes": 1, "replies": [] } ] }, { "id": 2745233, "author_name": "MaxChen303", "author_url": "", "post_date": "2024-04-10T13:35:32.363000", "content": "This is the first time I have made a team with others on Kaggle. Thanks to my teammates for bringing the great experience.
\nThe main idea of my solution is to generate inputs similar to what the annotators see.
\n
Due to a mistake in my HIGH vote CV implementation (accidentally included some low votes samples), I think I didn't fully optimize the performance in the last week.
\nAfter checking the provided EEG pattern images many times, I found that the provided raw EEGs look much cleaner. It was much easier to see the pattern in the provided image than my first few raw EEG plots.
\nIt turns out that simply applying a bandpass filter can make the raw EEGs look much closer to the provided examples.
\nI also noticed that the scipy sosfilt is preferred over the commonly used lfilter according to the document and the sosfiltfilt can reduce the effect of phase shift.
The red boxes in this diagram show the possible phase shifts:
\n
average # LOW votes / average # HIGH votes.Test time augmentation(TTA):
\nPretrained backbones from TIMM with different pooling layers.
\nUEMU & T.H. & kazumax created models as a mini team.
\nI was in charge of the raw EEG part.
\nCV was only calculated for the middle sample when samples from the same eeg_id were lined up in time order.
I developed a wavenet+2dCNN model using only raw EEG.
\nBelow is an outline of my pipeline.
\nI achieved a public LB = 0.25 (cv = 0.244) with a single model.
thanks for sharing @hiroitakafumi and congrats on achieving 5th place. Is it possible you share your training and inference code?
", "votes": 0, "replies": [] }, { "id": 2755394, "author_name": "MBOOK", "author_url": "", "post_date": "2024-04-16T13:48:38.570000", "content": "Thanks for sharing.
\nWhy did you add a CNN Block between WaveBlock?
\nI just want to know the logic to do that.
I wanted to reduce the width of the Wavenet output by including a CNN with stride=2.
\nWithout CNN, the output shape of Wavenet is [64, 10000], but with CNN it becomes [64, 625].
\nThe aim was to improve accuracy by lowering the downsampling rate due to pooling layer.
\nPublic LB improved by about 0.01.
UEMU & T.H. & kazumax created models as a mini team.
\nI was in charge of the spectrogram part.
\nCV was only calculated for the middle sample when samples from the same eeg_id were lined up in time order.
24 spectrograms
\nEEGSpec is a power spectrogram truncated at 20Hz not mel spectrogram.
\nspec = librosa.stft(\n y=x,\n hop_length=(x) // ,\n n_fft=,\n)\nspec = np.(spec) ** \n\n\n\n\nspec = spec[:, :] \n
Our team (UEMU&kazumax&TH) developed models, each member focusing on a different data type.
\n However, in this competition, it's necessary to integrate the inference values of each data type, similar to what the annotators are doing.
\n Therefore, we created a model that integrates the models for each data type through a fully connected (FC) layer to provide a multifaceted perspective.
\n We created some models of each data type for ensemble, and we also developed this model for all combinations.
As mentioned in this discussion, there are some data where the eeg_id is different but the patient_id is the same in the test dataset.
\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/471890
We aggregated the prediction values for each patient_id to create new features.
\nUsing the original prediction values and these aggregated features, we built a another inference model.
\nThis model did not improve the score on its own, but by including this model in the 2_3.Weight Average, we achieved an improvement in CV of 0.002 to 0.003.
We took a weighted average using all the models(each single model,2_1.FC model and the 2_2.PostProcess, about 50 models).
\nThe weights were optimized based on CV. Although it seemed excessive, the correlation between the LB and CV was good.
The best scores(UEMU&kazumax&TH's part)
\nSimple weighted average of three teams.
\nThe weighted average was taken using the logits before applying the softmax function.
\nThe weights were decided based on the public leaderboard (LB), where the public best score was equal to the private best score.
The best scores(Team KTMUD)
\nCongratulations on achieving 5th place in this competition. Thanks for sharing the insights into your solution.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2745188": "Team : KTMUD ( @kazumax0720 @hiroitakafumi @maxchen303 @asaliquid1011 @dimanishi) \n\nFirst of all, Thank you to the organizers and Congrats to all the participants and winners!\n\n## Summary\nOur team merged with the UEMU&T.H.&kazumax teams and MaxChen303 and D.Imanishi during the competition. \nWe ensemble the models developed by each team. \nOur prediciction pipeline is this. \nEach approaches are will added in the comments!\n\n* UEMU's part : [Link](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492652#2745803)\n* T.H.'s part : [Link](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492652#2745305)\n* kazumax's part : [Link](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492652#2745192)\n* MaxChen303's part : [Link](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492652#2745233)\n* D.Imanishi's part : [Link](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492652#2745220)\n\n\n\nWe borrowed the format from the [1st place solution](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492560). \nThank you, team SONY!", "2745220": "## D.Imanishi's part\nThanks to the organizers and congrats to all the winners! I think it was a very exciting and hard competition.\nAnd I am very happy to have finally achieved my goal of becoming a grandmaster!\n\n### Approach\n2D-CNN Models were trained for two types of input data and ensemble them. \n- concatenation of waveform graph image and kaggle spectrogram image (public LB:0.25)\n- waveform graph only (public LB:0.25)\n\nThe input image of my model is like following. 19 signals are the same as competition's sample figure.\n\nI had also taken the approach of converting EEG to spectrogram, but did not use it in the end because the score was better without it in the process of team merging.\n\n### Training\n- 2 stage training (1st: all data, 2nd: votes>=10)\n- After training the first model, the second model was trained by applying the pseudo label to votes<10 data using OOF prediction. Instead of replacing it with the pseudo label, it was mixed with the original label according to the number of votes (for samples with fewer votes, increase the weight of the pseudo label).\n- 3 types of CNN backbones were ensembled (EfficientNetV2B0, EfficientNetV2B1, EfficientNetV1B0)\n\n### Augmentation\n- Swap left and right electrode (Fp1<=>Fp2, F3<=>F4, ...)\n- Positive and negative swap (data=-data)\n- Mixup\n- Flipping time axis (kaggle spectrogram only)\n\n### Inference\n- TTA (Same as training augmentations except mixup)\n- Logits averaging (not after softmax) for ensemble\n\n### Code\n- [Waveform image generation](https://www.kaggle.com/code/dimanishi/hms-waveform-image-generation)\n- [Graph model training](https://www.kaggle.com/code/dimanishi/hms-graph-model-training)\n- [Graph model inference](https://www.kaggle.com/code/dimanishi/hms-graph-model-inference)", "2745233": "## MaxChen303's part\nThis is the first time I have made a team with others on Kaggle. Thanks to my teammates for bringing the great experience.\n### Summary\nThe main idea of my solution is to generate inputs similar to what the annotators see.\n\n- The size of each part is configurable.\n - Final models use size 512x512.\n - The width of raw eeg part is 500 for 10s. (Should be enough for sampling 20Hz signals)\n - The height of raw eeg part is controlled by the number of repeat. E.g. 16 channels * 8 repeat = 128 height.\n- Removing the EEG spectrogram part may not affect the performance.\n\nDue to a mistake in my HIGH vote CV implementation (accidentally included some low votes samples), I think I didn't fully optimize the performance in the last week.\n- The best score of my part with ensemble (3 models) is Private 0.295527/Public 0.247586. \n- The best single model score: Private=0.303614, Public = 0.250533\n\n### EEG Preprocessing \nAfter checking the provided EEG pattern images many times, I found that the provided raw EEGs look much cleaner. It was much easier to see the pattern in the provided image than my first few raw EEG plots.\n\nIt turns out that simply applying a bandpass filter can make the raw EEGs look much closer to the provided examples.\nI also noticed that the scipy `sosfilt` is preferred over the commonly used `lfilter` according to the [document](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.lfilter.html) and the `sosfiltfilt` can reduce the effect of phase shift. \n\n- https://stackoverflow.com/questions/12093594/how-to-implement-band-pass-butterworth-filter-with-scipy-signal-butter/48677312#48677312\n- Thanks to @gunesevitan for sharing this [discussion](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/483839)\n\nThe red boxes in this diagram show the possible phase shifts:\n\n\n\n### Training\n- One stage training with lower weight for LOW vote samples.\n - Monitor CV score for both All and HIGH vote samples under different weight ratios.\n - The final weight selected is 0.25 for low vote samples (<=7) for best ALL and HIGH vote CV score.\n - The weight is very close to (3.04/14.29 ~= 0.21), which is `average # LOW votes / average # HIGH votes`.\n- Exponential Moving Average (EMA) did well in stabilizing the validation loss during training. \n\n### Augmentation\n- Flip Left and Right components\n- Horizontal flip for spectrograms\n- Time and Frequency mask for the spectrograms\n- Time and Channel mask for the raw EEG. (Randomly hide 4-6 channels in the raw EEG)\n- Time shift of 10s Raw EEG\n- Raw EEG * -1 \n- Small random contrast adjustment for the raw eeg.\n\nTest time augmentation(TTA):\n- Flip Left and Right components.\n\n### Model\nPretrained backbones from TIMM with different pooling layers.\n- Models used in final submission:\n - maxvit_small_tf_512, \n - maxvit_tiny_tf_512,\n - convnext_small\n- Average pooling performs the best in my experiments.\n- LayerNorm -> Dropout -> FC head\n", "2745305": "## T.H.'s part\nUEMU & T.H. & kazumax created models as a mini team. \nI was in charge of the raw EEG part. \nCV was only calculated for the middle sample when samples from the same eeg_id were lined up in time order.\n\n### pipeline\nI developed a wavenet+2dCNN model using only raw EEG. \nBelow is an outline of my pipeline. \nI achieved a public LB = 0.25 (cv = 0.244) with a single model.\n\n\n\n### Input\n\n- Compute a total of 16 signals by calculating 4 neighboring difference signals each for LL, RL, LP, RP.\n- 20Hz butter lowpass filter.\n- Clipping of signal intensity.\n\n### Model\n\n- Wavenet+2DCNN.\n- For Wavenet, I referred to this [notebook](https://www.kaggle.com/code/alejopaullier/hms-wavenet-pytorch-train \"HMS | WaveNet PyTorch [Train]\"). Thank you for sharing the code!\n- Input 16 signals separately into Wavenet and average the output feature maps within the groups of LL, RL, LP, RP.\n- Stack the four feature maps vertically to input into 2DCNN.\n- For the 2DCNN model, MaxxvitV2-nano was the most accurate. Maxxvit-small and effnetb4 were also used for the ensemble.\n\n### Training\n\n- 2-stage training:\n - 1st stage: train with all data for 50 epochs (lr=1e-3).\n - 2nd stage: fine-tune with data where n_votes>=10 for 30 epochs (lr=1e-4).\n- Optimizer: AdamW.\n- Scheduler: Cosine Annealing (+Warmup).\n- Augmentations (in order of most effective):\n - ChannelSwap ([LL/LP] <=> [RL/RP]).\n - Inversion (inversion of signal strength).\n - TimeMask.\n - Reverse (time reversal of the signal).", "2745192": "## kazumax's part\n\nUEMU & T.H. & kazumax created models as a mini team. \nI was in charge of the spectrogram part. \nCV was only calculated for the middle sample when samples from the same eeg_id were lined up in time order.\n### Input\n* 24 spectrograms\n * 4 KaggleSpec (LL,LR,RP,RL)\n * 20 EEGSpec \n * 16 specs: Adjacent Difference (Fp1-F7, F7-T3, …) \n * 4 specs (LL,LP,RP,RL) \n\n* EEGSpec is a power spectrogram truncated at 20Hz not mel spectrogram.\n ```python\n spec = librosa.stft(\n y=x,\n hop_length=len(x) // 256,\n n_fft=1024,\n )\n spec = np.abs(spec) ** 2\n\n # truncate at 20Hz.\n # frequency resoluion = sr / n_fft = 200[Hz] / 1024 = 0.1953[Hz]\n # 0.1953[Hz] * 100 ≃ 20[Hz]\n spec = spec[:100, :] \n ```\n\n### Model\n\n### Training\n* 2stage training\n * 1st stage: all data\n * 2nd stage: vote >= 10\n* Augmentation\n * TimeFreqMask(p=0.5)\n * swap left-right specs(p=0.5)\n * LL ↔ RL, LR ↔ RP\n * Fp1 - F7 ↔ FP2 - F8, ...\n* label smoothing\n * epsilon=0.1 for samples with vote <= 3", "2745803": "## UEMU's part\n\n### 1.Multi Input Model \n - input = Kaggle_Spec + EEG_Spec + Raw_RRG\n - model = large model(2DCNN + 1DCNN + MLP)\n - CV(single) = 0.2436\n (update later)\n\n### 2.UEMU&kazumax&TH Ensemble\n#### 2_1.Integrated model for Each Data Type\n Our team (UEMU&kazumax&TH) developed models, each member focusing on a different data type.\n However, in this competition, it's necessary to integrate the inference values of each data type, similar to what the annotators are doing.\n Therefore, we created a model that integrates the models for each data type through a fully connected (FC) layer to provide a multifaceted perspective.\n We created some models of each data type for ensemble, and we also developed this model for all combinations.\n - Feature\n - prediction of Raw EEG Models(T.H. Part)\n - prediction of Multi Spec Models(kazumax Part)\n - prediction of EEG Spec Models(UEMU Part)\n - prediction of Multi Input Models(UEMU Part)\n - Model\n - multi-layer fully connected (FC) layer (No activation function is used)\n \n#### 2_2.PostProcess\nAs mentioned in this discussion, there are some data where the eeg_id is different but the patient_id is the same in the test dataset.\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/471890\n\nWe aggregated the prediction values for each patient_id to create new features.\nUsing the original prediction values and these aggregated features, we built a another inference model.\nThis model did not improve the score on its own, but by including this model in the 2_3.Weight Average, we achieved an improvement in CV of 0.002 to 0.003.\n\n - Feature\n - Original prediction\n - Average prediction of Raw EEG Models(T.H. Part)\n - Average prediction of Multi Spec Models(kazumax Part)\n - Average prediction of Multi Input Models(UEMU Part)\n - Aggregated feature\n - Average/Max/Min/Std/Delta/Count of original prediction groupby patient_id\n - Model\n - multi-layer fully connected (FC) layer (No activation function is used)\n\n\n#### 2_3.Weight Average All Models\nWe took a weighted average using all the models(each single model,2_1.FC model and the 2_2.PostProcess, about 50 models).\nThe weights were optimized based on CV. Although it seemed excessive, the correlation between the LB and CV was good.\n\nThe best scores(UEMU&kazumax&TH's part)\n - CV = 0.204\n - public = 0.2382\n - private = 0.2859(≒10th place)\n\n### 3.Team KTMUD Ensemble\nSimple weighted average of three teams.\nThe weighted average was taken using the logits before applying the softmax function.\nThe weights were decided based on the public leaderboard (LB), where the public best score was equal to the private best score.\n - weights = [D.Imanishi , MaxChen, and UEMU&kazumax&TH] = [0.28, 0.28, 0.44] \n\nThe best scores(Team KTMUD)\n - public = 0.2258\n - private = 0.2761", "2748281": "Congratulations on achieving 5th place in this competition. Thanks for sharing the insights into your solution. " } }