{ "id": 492240, "title": "4th place solution", "url": "/competitions/hms-harmful-brain-activity-classification/writeups/aillis-go-bilzard-4th-place-solution", "author_name": "", "post_date": "2024-05-09T00:45:23.957Z", "votes": 99, "comment_count": 18, "views": 0, "content": "

First of all, we want to thank Kaggle and Harvard Medical School for hosting the competition and CCEMRC for providing such a interest dataset. I would also like to thank our team members. It was a great experience to develop with high level members.

\n

Summary

\n

We ensemble a total of 8 1D and 2D models from 4 members. The CV for 5939 data with vote_sum>=10 was 0.1983 for a weighted average. For the final ensemble, we used a single-layer nn.Linear and overfit the model to the training data using all data for training.

\n

Code

\n\n

Yuji

\n

2D Model

\n

Following chris's method (ref: https://www.kaggle.com/code/cdeotte/how-to-make-spectrogram-from-eeg ), a spectrogram was generated from eeg and used as input for swin transformer v2 large. I pretrained on the full data and then finetune on clean data with vote≥10. By looking at the data in chronological order within the same eeg_id and considering them as different data when the labels switched, the finetune training data went from 5939 to 6366. The CV of the single model reached 0.2318 through a series of minor efforts, such as how to make labels and spectrograms. (CV was calculated for 5939 cases, if there are multiple samples in one eeg_id, the first sample was used for validation)
\nFor diversity, four models were created with variations in spectrogram generation parameters, seconds(10/30/40/50), with and without concat of the kaggle spectrogram, etc. The CV of the 4model ensemble is 0.2135.

\n

Ensemble

\n

I had been ensemble using nelder-mead until midway through the competition and noticed that LB always improves when the weight is finer without rounding. It seemed ok to overfit the train data a bit more, so I trained a linear layer with 360 parameters with all data as train / all data as valid. (code: nn.Linear(48, 6)) # 48: 6 predictions for each of the 8models
\nThe probe results showed that test contained more Seizure/LRDA/GRDA than clean train, so LB was improved by upsampleing more of the three classes before training. Sub with nn.Linear was public 0.21 / private 0.27 compared to sub with nelder-mead which was public 0.22 / private 0.28.
\nThe more epochs I increased, the better the LB became until about 3000 epochs, so perhaps a method using metric learning, such as giving the same label to samples similar to the train sample, might have been effective. We did not have time to try this because I came up with the idea at the last stage of the competition.

\n

tattaka

\n

I developed four different models.
\nUnlike Yuji's pipeline, they are trained only on data with vote>7.
\nEEG is processed using butter_bandpass_filter(min=0.5, max=30) before conversion to spectrogram.

\n

model

\n

\n

\n

\n

\n

I used tiny_vit_21m_384, caformer_s18, convnext_large as 2D backbone.

\n

Training

\n\n

bilzard

\n

1. Outline

\n

Initially, my solution was an ensemble of 1D and 2D models. However, after merging teams and recognizing that my teammates had already developed strong 2D models, I shifted my focus primarily towards 1D models (i.e. the models process raw EEG signals directly).

\n

2. Basic Concept

\n

The approach to our 1D modeling is twofold:

\n
    \n
  1. L/R Symmetric Modeling: We aimed to maintain symmetry in the model to facilitate the detection of Laterality.
    2. \n
  2. Channel Quality Factor (CQF): This is used to evaluate the quality of EEG channels and identify any that are suboptimal.
    3. \n
\n

Detail solution of 1D model:
\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492388

\n

3. Result

\n

The cross-validation scores (CVs) of our 1D models in the final submissions are listed below. These CVs were calculated using samples with num_votes > 8.

\n
 .\n
\n

yu4u

\n

I developed a 1D CNN model, whose backbone relies heavily on @tatamikenn's excellent implementation.

\n

Preprocessing

\n\n

Model

\n

1D version of MobileNetV2 with a channel mixer is used.
\n
\nFeatures from LL/RL/LP/RP are processed independently by 1D inverted residual blocks, and then a channel mixer is used to exchange information across LL, RL, LP, and RP. The channel mixer is implemented with a convolution.

\n

Training

\n", "messages": [ { "id": "2742639", "postDate": "04/09/2024 02:20:32", "content": "

First of all, we want to thank Kaggle and Harvard Medical School for hosting the competition and CCEMRC for providing such a interest dataset. I would also like to thank our team members. It was a great experience to develop with high level members.

\n

Summary

\n

We ensemble a total of 8 1D and 2D models from 4 members. The CV for 5939 data with vote_sum>=10 was 0.1983 for a weighted average. For the final ensemble, we used a single-layer nn.Linear and overfit the model to the training data using all data for training.

\n

Code

\n\n

Yuji

\n

2D Model

\n

Following chris's method (ref: https://www.kaggle.com/code/cdeotte/how-to-make-spectrogram-from-eeg ), a spectrogram was generated from eeg and used as input for swin transformer v2 large. I pretrained on the full data and then finetune on clean data with vote≥10. By looking at the data in chronological order within the same eeg_id and considering them as different data when the labels switched, the finetune training data went from 5939 to 6366. The CV of the single model reached 0.2318 through a series of minor efforts, such as how to make labels and spectrograms. (CV was calculated for 5939 cases, if there are multiple samples in one eeg_id, the first sample was used for validation)
\nFor diversity, four models were created with variations in spectrogram generation parameters, seconds(10/30/40/50), with and without concat of the kaggle spectrogram, etc. The CV of the 4model ensemble is 0.2135.

\n

Ensemble

\n

I had been ensemble using nelder-mead until midway through the competition and noticed that LB always improves when the weight is finer without rounding. It seemed ok to overfit the train data a bit more, so I trained a linear layer with 360 parameters with all data as train / all data as valid. (code: nn.Linear(48, 6)) # 48: 6 predictions for each of the 8models
\nThe probe results showed that test contained more Seizure/LRDA/GRDA than clean train, so LB was improved by upsampleing more of the three classes before training. Sub with nn.Linear was public 0.21 / private 0.27 compared to sub with nelder-mead which was public 0.22 / private 0.28.
\nThe more epochs I increased, the better the LB became until about 3000 epochs, so perhaps a method using metric learning, such as giving the same label to samples similar to the train sample, might have been effective. We did not have time to try this because I came up with the idea at the last stage of the competition.

\n

tattaka

\n

I developed four different models.
\nUnlike Yuji's pipeline, they are trained only on data with vote>7.
\nEEG is processed using butter_bandpass_filter(min=0.5, max=30) before conversion to spectrogram.

\n

model

\n

\n

\n

\n

\n

I used tiny_vit_21m_384, caformer_s18, convnext_large as 2D backbone.

\n

Training

\n\n

bilzard

\n

1. Outline

\n

Initially, my solution was an ensemble of 1D and 2D models. However, after merging teams and recognizing that my teammates had already developed strong 2D models, I shifted my focus primarily towards 1D models (i.e. the models process raw EEG signals directly).

\n

2. Basic Concept

\n

The approach to our 1D modeling is twofold:

\n
    \n
  1. L/R Symmetric Modeling: We aimed to maintain symmetry in the model to facilitate the detection of Laterality.
    2. \n
  2. Channel Quality Factor (CQF): This is used to evaluate the quality of EEG channels and identify any that are suboptimal.
    3. \n
\n

Detail solution of 1D model:
\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492388

\n

3. Result

\n

The cross-validation scores (CVs) of our 1D models in the final submissions are listed below. These CVs were calculated using samples with num_votes > 8.

\n
 .\n
\n

yu4u

\n

I developed a 1D CNN model, whose backbone relies heavily on @tatamikenn's excellent implementation.

\n

Preprocessing

\n\n

Model

\n

1D version of MobileNetV2 with a channel mixer is used.
\n
\nFeatures from LL/RL/LP/RP are processed independently by 1D inverted residual blocks, and then a channel mixer is used to exchange information across LL, RL, LP, and RP. The channel mixer is implemented with a convolution.

\n

Training

\n", "rawMarkdown": "First of all, we want to thank Kaggle and Harvard Medical School for hosting the competition and CCEMRC for providing such a interest dataset. I would also like to thank our team members. It was a great experience to develop with high level members.\n\n# Summary\n\nWe ensemble a total of 8 1D and 2D models from 4 members. The CV for 5939 data with vote_sum>=10 was 0.1983 for a weighted average. For the final ensemble, we used a single-layer nn.Linear and overfit the model to the training data using all data for training.\n\n# Code\n\n- training code: https://github.com/bilzard/hms-4th-place-solution\n- inference code: https://www.kaggle.com/code/yujiariyasu/4th-place-solution\n\n# Yuji\n\n### 2D Model\nFollowing chris's method (ref: https://www.kaggle.com/code/cdeotte/how-to-make-spectrogram-from-eeg ), a spectrogram was generated from eeg and used as input for swin transformer v2 large. I pretrained on the full data and then finetune on clean data with vote≥10. By looking at the data in chronological order within the same eeg_id and considering them as different data when the labels switched, the finetune training data went from 5939 to 6366. The CV of the single model reached 0.2318 through a series of minor efforts, such as how to make labels and spectrograms. (CV was calculated for 5939 cases, if there are multiple samples in one eeg_id, the first sample was used for validation)\nFor diversity, four models were created with variations in spectrogram generation parameters, seconds(10/30/40/50), with and without concat of the kaggle spectrogram, etc. The CV of the 4model ensemble is 0.2135.\n\n### Ensemble\nI had been ensemble using nelder-mead until midway through the competition and noticed that LB always improves when the weight is finer without rounding. It seemed ok to overfit the train data a bit more, so I trained a linear layer with 360 parameters with all data as train / all data as valid. (code: nn.Linear(48, 6)) # 48: 6 predictions for each of the 8models\nThe probe results showed that test contained more Seizure/LRDA/GRDA than clean train, so LB was improved by upsampleing more of the three classes before training. Sub with nn.Linear was public 0.21 / private 0.27 compared to sub with nelder-mead which was public 0.22 / private 0.28.\nThe more epochs I increased, the better the LB became until about 3000 epochs, so perhaps a method using metric learning, such as giving the same label to samples similar to the train sample, might have been effective. We did not have time to try this because I came up with the idea at the last stage of the competition.\n\n# tattaka\nI developed four different models.\nUnlike Yuji's pipeline, they are trained only on data with vote>7.\nEEG is processed using `butter_bandpass_filter(min=0.5, max=30)` before conversion to spectrogram.\n\n### model\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3460291%2F817f86660bb00936dca44203adcafaa4%2F312515380-e2767514-c6eb-4921-aaef-99fce3432c58.png?generation=1712628961328663&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3460291%2F794c519821e0788f35cd50dd6792091c%2F313414930-6d0cd3a9-0489-4c42-ab85-fe94efb89912.png?generation=1712628976951075&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3460291%2F44b0356b9b9c75f3ce9f9f72e3f58e2c%2F320515916-b8f6976b-8c90-4fa3-b3b3-71ec27c5628e.png?generation=1712629033096178&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3460291%2Fae9bbea1bf5603d10a5ea9a0f97f9c41%2F320516088-467e87fd-f12d-4405-9a7f-f2c9c6a57418.png?generation=1712629047992602&alt=media)\n\nI used `tiny_vit_21m_384`, `caformer_s18`, `convnext_large` as 2D backbone.\n\n### Training\n* 50 epoch\n* 1e-4 or 5e-4 or 2e-4 to 0 dropped by cosine_schedule\n * Warmup 1/50step of the whole\n* EMA(decay=0.998)\n* drop_path_rate = 0.2 or 0.3\n* augmentation:\n * **mixup of eeg**\n * p=0.5, alpha=0.2\n * XYMasking (p = 0.5, num_masks_x = 8, num_masks_y = 8, mask_x_length = 1 / 16, mask_y_length = 1 / 16)\n * Apply independently before concat to kaggle spectrogram and eeg spectrogram\n * hflip of eeg and kaggle spectrogram\n * Apply independently before concat to kaggle spectrogram and eeg spectrogram\n * shuffle of eeg and kaggle spectrogram\n * \"FpX\", \"FX\", \"TX\", \"TX\", \"OX\" shuffle without breaking the order.\n * Flip for model_b and model_d\n * **random center crop for eeg**\n * Use the central 30 or 40s of eeg for inference, and a random +-10 length for training\n\n# bilzard\n\n### 1. Outline\n\nInitially, my solution was an ensemble of 1D and 2D models. However, after merging teams and recognizing that my teammates had already developed strong 2D models, I shifted my focus primarily towards 1D models (i.e. the models process raw EEG signals directly).\n\n### 2. Basic Concept\n\nThe approach to our 1D modeling is twofold:\n\n1. **L/R Symmetric Modeling**: We aimed to maintain symmetry in the model to facilitate the detection of Laterality.\n2. **Channel Quality Factor (CQF)**: This is used to evaluate the quality of EEG channels and identify any that are suboptimal.\n\nDetail solution of 1D model:\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492388\n\n### 3. Result\n\nThe cross-validation scores (CVs) of our 1D models in the final submissions are listed below. These CVs were calculated using samples with `num_votes > 8`.\n\n```\nv5_eeg_24ep_cutmix 0.2477\n```\n\n# yu4u\nI developed a 1D CNN model, whose backbone relies heavily on @tatamikenn's excellent implementation.\n\n### Preprocessing\n* Extract LL, RL, LP, and RP features (4x4x10000) from raw EEG signals.\n* Apply bandpass filter (0.5Hz-20Hz).\n* Randomly crop 9600 samples and downsample them to 1/5 (1920).\n\n### Model\n1D version of MobileNetV2 with a channel mixer is used.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3460291%2Fa0f14f1648c39de25ff8db6f1e7fc6fc%2F1dcnn.png?generation=1712628591375689&alt=media)\nFeatures from LL/RL/LP/RP are processed independently by 1D inverted residual blocks, and then a channel mixer is used to exchange information across LL, RL, LP, and RP. The channel mixer is implemented with a convolution.\n\n### Training\n* 2-stage training: in the first 32 epochs, all data are used for training, and in the latter 32 epochs, those with vote_num > 7 are used.\n* Optimizer: AdamW with lr=2e-3 to 0.0, weight_decay=1e-5, batch_size=96.\n* Augmentations: cutout, mixup (p=0.5, alpha=1.0), shuffle LL/RL/LP/RP, flip in time axis.", "votes": null }, { "id": "2742651", "postDate": "04/09/2024 02:39:37", "content": "

I'm curious about Channel Quality Factor.
\nIs there any code for this?

", "rawMarkdown": "I'm curious about Channel Quality Factor.\nIs there any code for this?", "votes": null }, { "id": "2742659", "postDate": "04/09/2024 02:45:45", "content": "

Congrats on 4th place! Thanks for sharing your solution.

", "rawMarkdown": "Congrats on 4th place! Thanks for sharing your solution.", "votes": null }, { "id": "2742679", "postDate": "04/09/2024 02:56:28", "content": "

Congratulations and thanks for Sharing !

", "rawMarkdown": "Congratulations and thanks for Sharing !", "votes": null }, { "id": "2742703", "postDate": "04/09/2024 03:14:09", "content": "

Congrats!
\nWait, Swin transformer v2 large!?
\nIn my case, ResNet 101 even worsen than MobileNetv3Large, let along ViTs lol.

", "rawMarkdown": "Congrats!\nWait, Swin transformer v2 large!?\nIn my case, ResNet 101 even worsen than MobileNetv3Large, let along ViTs lol.", "votes": null }, { "id": "2742824", "postDate": "04/09/2024 05:30:13", "content": "

Congrats! fantastic work

", "rawMarkdown": "Congrats! fantastic work", "votes": null }, { "id": "2742913", "postDate": "04/09/2024 06:20:53", "content": "

Congratulation,Yuji. It's a huge engineering, and your team did the work like in a studio. Your whole pipeline almost reached the top, and can be utilized in industry. BTW, I wish I had enough hardware like your team.

", "rawMarkdown": "Congratulation,Yuji. It's a huge engineering, and your team did the work like in a studio. Your whole pipeline almost reached the top, and can be utilized in industry. BTW, I wish I had enough hardware like your team.", "votes": null }, { "id": "2743197", "postDate": "04/09/2024 09:22:22", "content": "

Congratulations for your fine work.
\nThe diagrams look great in this post, what tool have you used to create them?

", "rawMarkdown": "Congratulations for your fine work. \nThe diagrams look great in this post, what tool have you used to create them?", "votes": null }, { "id": "2743350", "postDate": "04/09/2024 12:00:32", "content": "

My diagram was created with Google Slides

", "rawMarkdown": "My diagram was created with Google Slides", "votes": null }, { "id": "2743406", "postDate": "04/09/2024 12:46:24", "content": "

@yujiariyasu thanks for putting it all together. That's a great team work. Who carried the most in the team? How did you guys organize the team work and communication?

", "rawMarkdown": "yujiariyasu thanks for putting it all together. That's a great team work. Who carried the most in the team? How did you guys organize the team work and communication?", "votes": null }, { "id": "2743522", "postDate": "04/09/2024 13:56:38", "content": "

Very diverse solution and congrats on the strong finish!

\n

@ren4yu Would you be willing to share code/pseudocode for the channel mixer part of the 1D version of MobileNetV2? This sounds really interesting.

", "rawMarkdown": "Very diverse solution and congrats on the strong finish!\n\n@ren4yu Would you be willing to share code/pseudocode for the channel mixer part of the `1D version of MobileNetV2`? This sounds really interesting.", "votes": null }, { "id": "2743549", "postDate": "04/09/2024 14:08:27", "content": "

Just using the swin transformer improved score by about 5%!

", "rawMarkdown": "Just using the swin transformer improved score by about 5%!", "votes": null }, { "id": "2743655", "postDate": "04/09/2024 15:07:19", "content": "

I shared on this thread.
\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492388#2743601

", "rawMarkdown": "I shared on this thread.\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492388#2743601", "votes": null }, { "id": "2743768", "postDate": "04/09/2024 15:59:22", "content": "

thanks for your sharing!

", "rawMarkdown": "thanks for your sharing!", "votes": null }, { "id": "2743982", "postDate": "04/09/2024 17:31:26", "content": "

Congrats for winning the prize!

", "rawMarkdown": "Congrats for winning the prize!", "votes": null }, { "id": "2744080", "postDate": "04/09/2024 18:37:34", "content": "

Thank you for the explanations. Your team did great!

", "rawMarkdown": "Thank you for the explanations. Your team did great!", "votes": null }, { "id": "2744410", "postDate": "04/09/2024 22:26:34", "content": "

@brendanartley
\nYou can think channel mixer is just 2D conv with kernel size (k, 1) with input shape of (B, d, ch, T).

\n

One thing to note is our 1D model is actually 1.5D, i.e. input is 2D and process with EEG channel and time series alternately.
\nE.g. for processing time series, we use 2D conv with kernel size (1, k) and for processing EEG channel mixing, with (k, 1).
\nYou can also consider our model as factorized version of 2D model which processes vertically concatenated EEG channels.

\n

Our detailed architecture is shared here on (A3. Backbone 1D CNN Architecture):
\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492388

", "rawMarkdown": "brendanartley \nYou can think channel mixer is just 2D conv with kernel size (k, 1) with input shape of (B, d, ch, T).\n\nOne thing to note is our 1D model is actually 1.5D, i.e. input is 2D and process with EEG channel and time series alternately.\nE.g. for processing time series, we use 2D conv with kernel size (1, k) and for processing EEG channel mixing, with (k, 1).\nYou can also consider our model as factorized version of 2D model which processes vertically concatenated EEG channels.\n\nOur detailed architecture is shared here on (A3. Backbone 1D CNN Architecture):\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492388", "votes": null }, { "id": "2744698", "postDate": "04/10/2024 04:27:57", "content": "

Congratulations on winning the 4th prize in this competition. Thanks for sharing writeup on your solution with diagrams.

", "rawMarkdown": "Congratulations on winning the 4th prize in this competition. Thanks for sharing writeup on your solution with diagrams.", "votes": null }, { "id": "2745098", "postDate": "04/10/2024 11:25:34", "content": "

Guys, we are happy to announce we open-sourced part of our source code (bilzard part).
\nNow you can access full resource of our experiments including best 1D model.
\nEnjoy Kaggling!

\n

License: Apache 2.0
\nLink:
\nhttps://github.com/bilzard/kaggle-hms-bilzard/tree/main

", "rawMarkdown": "Guys, we are happy to announce we open-sourced part of our source code (bilzard part).\nNow you can access full resource of our experiments including best 1D model.\nEnjoy Kaggling!\n\nLicense: Apache 2.0\nLink:\nhttps://github.com/bilzard/kaggle-hms-bilzard/tree/main", "votes": null } ], "comments": [ { "id": 2742651, "author_name": "horikitasaku", "author_url": "", "post_date": "04/09/2024 02:39:37", "content": "

I'm curious about Channel Quality Factor.
\nIs there any code for this?

", "votes": null, "replies": [ { "id": 2743655, "author_name": "tatamikenn", "author_url": "", "post_date": "04/09/2024 15:07:19", "content": "

I shared on this thread.
\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492388#2743601

", "votes": null, "replies": [ { "id": 2743768, "author_name": "horikitasaku", "author_url": "", "post_date": "04/09/2024 15:59:22", "content": "

thanks for your sharing!

", "votes": null, "replies": [] } ] } ] }, { "id": 2742659, "author_name": "zijiangyang1116", "author_url": "", "post_date": "04/09/2024 02:45:45", "content": "

Congrats on 4th place! Thanks for sharing your solution.

", "votes": null, "replies": [] }, { "id": 2742679, "author_name": "autitoreezio", "author_url": "", "post_date": "04/09/2024 02:56:28", "content": "

Congratulations and thanks for Sharing !

", "votes": null, "replies": [] }, { "id": 2742703, "author_name": "jdwayy", "author_url": "", "post_date": "04/09/2024 03:14:09", "content": "

Congrats!
\nWait, Swin transformer v2 large!?
\nIn my case, ResNet 101 even worsen than MobileNetv3Large, let along ViTs lol.

", "votes": null, "replies": [ { "id": 2743549, "author_name": "yujiariyasu", "author_url": "", "post_date": "04/09/2024 14:08:27", "content": "

Just using the swin transformer improved score by about 5%!

", "votes": null, "replies": [] } ] }, { "id": 2742824, "author_name": "midcarryhz", "author_url": "", "post_date": "04/09/2024 05:30:13", "content": "

Congrats! fantastic work

", "votes": null, "replies": [] }, { "id": 2742913, "author_name": "sweetyheehee", "author_url": "", "post_date": "04/09/2024 06:20:53", "content": "

Congratulation,Yuji. It's a huge engineering, and your team did the work like in a studio. Your whole pipeline almost reached the top, and can be utilized in industry. BTW, I wish I had enough hardware like your team.

", "votes": null, "replies": [] }, { "id": 2743197, "author_name": "nartaa", "author_url": "", "post_date": "04/09/2024 09:22:22", "content": "

Congratulations for your fine work.
\nThe diagrams look great in this post, what tool have you used to create them?

", "votes": null, "replies": [ { "id": 2743350, "author_name": "tattaka", "author_url": "", "post_date": "04/09/2024 12:00:32", "content": "

My diagram was created with Google Slides

", "votes": null, "replies": [] } ] }, { "id": 2743406, "author_name": "sergiosaharovskiy", "author_url": "", "post_date": "04/09/2024 12:46:24", "content": "

@yujiariyasu thanks for putting it all together. That's a great team work. Who carried the most in the team? How did you guys organize the team work and communication?

", "votes": null, "replies": [] }, { "id": 2743522, "author_name": "brendanartley", "author_url": "", "post_date": "04/09/2024 13:56:38", "content": "

Very diverse solution and congrats on the strong finish!

\n

@ren4yu Would you be willing to share code/pseudocode for the channel mixer part of the 1D version of MobileNetV2? This sounds really interesting.

", "votes": null, "replies": [ { "id": 2744410, "author_name": "tatamikenn", "author_url": "", "post_date": "04/09/2024 22:26:34", "content": "

@brendanartley
\nYou can think channel mixer is just 2D conv with kernel size (k, 1) with input shape of (B, d, ch, T).

\n

One thing to note is our 1D model is actually 1.5D, i.e. input is 2D and process with EEG channel and time series alternately.
\nE.g. for processing time series, we use 2D conv with kernel size (1, k) and for processing EEG channel mixing, with (k, 1).
\nYou can also consider our model as factorized version of 2D model which processes vertically concatenated EEG channels.

\n

Our detailed architecture is shared here on (A3. Backbone 1D CNN Architecture):
\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492388

", "votes": null, "replies": [] } ] }, { "id": 2743982, "author_name": "haohantsao", "author_url": "", "post_date": "04/09/2024 17:31:26", "content": "

Congrats for winning the prize!

", "votes": null, "replies": [] }, { "id": 2744080, "author_name": "djtrainwreckx", "author_url": "", "post_date": "04/09/2024 18:37:34", "content": "

Thank you for the explanations. Your team did great!

", "votes": null, "replies": [] }, { "id": 2744698, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "04/10/2024 04:27:57", "content": "

Congratulations on winning the 4th prize in this competition. Thanks for sharing writeup on your solution with diagrams.

", "votes": null, "replies": [] }, { "id": 2745098, "author_name": "tatamikenn", "author_url": "", "post_date": "04/10/2024 11:25:34", "content": "

Guys, we are happy to announce we open-sourced part of our source code (bilzard part).
\nNow you can access full resource of our experiments including best 1D model.
\nEnjoy Kaggling!

\n

License: Apache 2.0
\nLink:
\nhttps://github.com/bilzard/kaggle-hms-bilzard/tree/main

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2742639": "First of all, we want to thank Kaggle and Harvard Medical School for hosting the competition and CCEMRC for providing such a interest dataset. I would also like to thank our team members. It was a great experience to develop with high level members.\n\n# Summary\n\nWe ensemble a total of 8 1D and 2D models from 4 members. The CV for 5939 data with vote_sum>=10 was 0.1983 for a weighted average. For the final ensemble, we used a single-layer nn.Linear and overfit the model to the training data using all data for training.\n\n# Code\n\n- training code: https://github.com/bilzard/hms-4th-place-solution\n- inference code: https://www.kaggle.com/code/yujiariyasu/4th-place-solution\n\n# Yuji\n\n### 2D Model\nFollowing chris's method (ref: https://www.kaggle.com/code/cdeotte/how-to-make-spectrogram-from-eeg ), a spectrogram was generated from eeg and used as input for swin transformer v2 large. I pretrained on the full data and then finetune on clean data with vote≥10. By looking at the data in chronological order within the same eeg_id and considering them as different data when the labels switched, the finetune training data went from 5939 to 6366. The CV of the single model reached 0.2318 through a series of minor efforts, such as how to make labels and spectrograms. (CV was calculated for 5939 cases, if there are multiple samples in one eeg_id, the first sample was used for validation)\nFor diversity, four models were created with variations in spectrogram generation parameters, seconds(10/30/40/50), with and without concat of the kaggle spectrogram, etc. The CV of the 4model ensemble is 0.2135.\n\n### Ensemble\nI had been ensemble using nelder-mead until midway through the competition and noticed that LB always improves when the weight is finer without rounding. It seemed ok to overfit the train data a bit more, so I trained a linear layer with 360 parameters with all data as train / all data as valid. (code: nn.Linear(48, 6)) # 48: 6 predictions for each of the 8models\nThe probe results showed that test contained more Seizure/LRDA/GRDA than clean train, so LB was improved by upsampleing more of the three classes before training. Sub with nn.Linear was public 0.21 / private 0.27 compared to sub with nelder-mead which was public 0.22 / private 0.28.\nThe more epochs I increased, the better the LB became until about 3000 epochs, so perhaps a method using metric learning, such as giving the same label to samples similar to the train sample, might have been effective. We did not have time to try this because I came up with the idea at the last stage of the competition.\n\n# tattaka\nI developed four different models.\nUnlike Yuji's pipeline, they are trained only on data with vote>7.\nEEG is processed using `butter_bandpass_filter(min=0.5, max=30)` before conversion to spectrogram.\n\n### model\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3460291%2F817f86660bb00936dca44203adcafaa4%2F312515380-e2767514-c6eb-4921-aaef-99fce3432c58.png?generation=1712628961328663&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3460291%2F794c519821e0788f35cd50dd6792091c%2F313414930-6d0cd3a9-0489-4c42-ab85-fe94efb89912.png?generation=1712628976951075&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3460291%2F44b0356b9b9c75f3ce9f9f72e3f58e2c%2F320515916-b8f6976b-8c90-4fa3-b3b3-71ec27c5628e.png?generation=1712629033096178&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3460291%2Fae9bbea1bf5603d10a5ea9a0f97f9c41%2F320516088-467e87fd-f12d-4405-9a7f-f2c9c6a57418.png?generation=1712629047992602&alt=media)\n\nI used `tiny_vit_21m_384`, `caformer_s18`, `convnext_large` as 2D backbone.\n\n### Training\n* 50 epoch\n* 1e-4 or 5e-4 or 2e-4 to 0 dropped by cosine_schedule\n * Warmup 1/50step of the whole\n* EMA(decay=0.998)\n* drop_path_rate = 0.2 or 0.3\n* augmentation:\n * **mixup of eeg**\n * p=0.5, alpha=0.2\n * XYMasking (p = 0.5, num_masks_x = 8, num_masks_y = 8, mask_x_length = 1 / 16, mask_y_length = 1 / 16)\n * Apply independently before concat to kaggle spectrogram and eeg spectrogram\n * hflip of eeg and kaggle spectrogram\n * Apply independently before concat to kaggle spectrogram and eeg spectrogram\n * shuffle of eeg and kaggle spectrogram\n * \"FpX\", \"FX\", \"TX\", \"TX\", \"OX\" shuffle without breaking the order.\n * Flip for model_b and model_d\n * **random center crop for eeg**\n * Use the central 30 or 40s of eeg for inference, and a random +-10 length for training\n\n# bilzard\n\n### 1. Outline\n\nInitially, my solution was an ensemble of 1D and 2D models. However, after merging teams and recognizing that my teammates had already developed strong 2D models, I shifted my focus primarily towards 1D models (i.e. the models process raw EEG signals directly).\n\n### 2. Basic Concept\n\nThe approach to our 1D modeling is twofold:\n\n1. **L/R Symmetric Modeling**: We aimed to maintain symmetry in the model to facilitate the detection of Laterality.\n2. **Channel Quality Factor (CQF)**: This is used to evaluate the quality of EEG channels and identify any that are suboptimal.\n\nDetail solution of 1D model:\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492388\n\n### 3. Result\n\nThe cross-validation scores (CVs) of our 1D models in the final submissions are listed below. These CVs were calculated using samples with `num_votes > 8`.\n\n```\nv5_eeg_24ep_cutmix 0.2477\n```\n\n# yu4u\nI developed a 1D CNN model, whose backbone relies heavily on @tatamikenn's excellent implementation.\n\n### Preprocessing\n* Extract LL, RL, LP, and RP features (4x4x10000) from raw EEG signals.\n* Apply bandpass filter (0.5Hz-20Hz).\n* Randomly crop 9600 samples and downsample them to 1/5 (1920).\n\n### Model\n1D version of MobileNetV2 with a channel mixer is used.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3460291%2Fa0f14f1648c39de25ff8db6f1e7fc6fc%2F1dcnn.png?generation=1712628591375689&alt=media)\nFeatures from LL/RL/LP/RP are processed independently by 1D inverted residual blocks, and then a channel mixer is used to exchange information across LL, RL, LP, and RP. The channel mixer is implemented with a convolution.\n\n### Training\n* 2-stage training: in the first 32 epochs, all data are used for training, and in the latter 32 epochs, those with vote_num > 7 are used.\n* Optimizer: AdamW with lr=2e-3 to 0.0, weight_decay=1e-5, batch_size=96.\n* Augmentations: cutout, mixup (p=0.5, alpha=1.0), shuffle LL/RL/LP/RP, flip in time axis.", "2742651": "I'm curious about Channel Quality Factor.\nIs there any code for this?", "2742659": "Congrats on 4th place! Thanks for sharing your solution.", "2742679": "Congratulations and thanks for Sharing !", "2742703": "Congrats!\nWait, Swin transformer v2 large!?\nIn my case, ResNet 101 even worsen than MobileNetv3Large, let along ViTs lol.", "2742824": "Congrats! fantastic work", "2742913": "Congratulation,Yuji. It's a huge engineering, and your team did the work like in a studio. Your whole pipeline almost reached the top, and can be utilized in industry. BTW, I wish I had enough hardware like your team.", "2743197": "Congratulations for your fine work. \nThe diagrams look great in this post, what tool have you used to create them?", "2743350": "My diagram was created with Google Slides", "2743406": "yujiariyasu thanks for putting it all together. That's a great team work. Who carried the most in the team? How did you guys organize the team work and communication?", "2743522": "Very diverse solution and congrats on the strong finish!\n\n@ren4yu Would you be willing to share code/pseudocode for the channel mixer part of the `1D version of MobileNetV2`? This sounds really interesting.", "2743549": "Just using the swin transformer improved score by about 5%!", "2743655": "I shared on this thread.\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492388#2743601", "2743768": "thanks for your sharing!", "2743982": "Congrats for winning the prize!", "2744080": "Thank you for the explanations. Your team did great!", "2744410": "brendanartley \nYou can think channel mixer is just 2D conv with kernel size (k, 1) with input shape of (B, d, ch, T).\n\nOne thing to note is our 1D model is actually 1.5D, i.e. input is 2D and process with EEG channel and time series alternately.\nE.g. for processing time series, we use 2D conv with kernel size (1, k) and for processing EEG channel mixing, with (k, 1).\nYou can also consider our model as factorized version of 2D model which processes vertically concatenated EEG channels.\n\nOur detailed architecture is shared here on (A3. Backbone 1D CNN Architecture):\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492388", "2744698": "Congratulations on winning the 4th prize in this competition. Thanks for sharing writeup on your solution with diagrams.", "2745098": "Guys, we are happy to announce we open-sourced part of our source code (bilzard part).\nNow you can access full resource of our experiments including best 1D model.\nEnjoy Kaggling!\n\nLicense: Apache 2.0\nLink:\nhttps://github.com/bilzard/kaggle-hms-bilzard/tree/main" }, "source": "meta" }