![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4712534%2F7ba613bbd6fb92058ca98ceb1327bc8a%2FScreenshot%202024-03-13%20at%2012.29.48%20PM.png?generation=1710313207400643&alt=media\")
I was recently exploring foundational models which are specifically pretrained on EEG/Brain signals. I found couple of really good reads which can be helpful for the competition.
\n\n\nBrain Bert: https://arxiv.org/abs/2302.14367 [ICLR 2023]\n
\n
Github Code: https://github.com/czlwang/BrainBERT
\nPretrained weights: https://www.kaggle.com/datasets/nischaydnk/brainbertweights
you can load the model with a configuration like this:
\n torch.nn nn\n\n (nn.Module): \n ():\n (SpecPredictionHead, self).__init__()\n cfg.hidden_dim = \n self.hidden_layer = nn.Linear(cfg.hidden_dim, cfg.hidden_dim)\n self.act_fn = \n\n self.layer_norm = nn.LayerNorm(cfg.hidden_dim)\n self.output = nn.Linear(cfg.hidden_dim, num_classes)\n\n ():\n h = self.hidden_layer(hidden)\n h = self.layer_norm(h)\n h = self.output(h)\n h\n\n (torch.nn.Module):\n\n ():\n ().__init__()\n ckpt_path = \n cfg = OmegaConf.create({: ckpt_path})\n self.model = build_model(cfg)\n init_state = torch.load(ckpt_path)\n load_model_weights(self.model, init_state[], )\n\n self.head = SpecPredictionHead(cfg, num_classes)\n\n ():\n x = x.squeeze()\n mask = mask.squeeze()\n\n h = self.model(x, mask)[] \n h = h[:,-,:]\n h = self.head(h)\n h\n
\nBRANT: https://openreview.net/forum?id=DDkl9vaJyE [Neurips 2023]\n
\n
Github Code: https://zju-brainnet.github.io/Brant.github.io/
\nModel weights: https://drive.google.com/file/d/1QzxTNBvgcJBRxa8W2mNq2Tj967GtlDLF/view
\nBRANT-2: https://arxiv.org/abs/2402.10251 [March 2024]\n
\n
Github: https://github.com/yzz673/Brant-2
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4712534%2F831ed03fdc0a7630b41035a8b739eb41%2FScreenshot%202024-03-13%20at%2012.35.26%20PM.png?generation=1710313785016789&alt=media\")
\nSimMTM: https://arxiv.org/abs/2302.00861[Neurips 2023]\n
\n
Github: https://github.com/thuml/SimMTM
\nI tried various architectures published, but no success in getting great scores. Maybe transformers aren't the best architecture for this task.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4712534%2F864851a6468fe48bf10d9e4d9e2657f1%2FScreenshot%202024-03-13%20at%2012.42.21%20PM.png?generation=1710314112558110&alt=media\")
Comparison of all these pretrained models, shown in Brant-2 paper 👆
", "messages": [ { "id": 2694638, "postDate": "2024-03-13T07:16:58.637Z", "content": "I was recently exploring foundational models which are specifically pretrained on EEG/Brain signals. I found couple of really good reads which can be helpful for the competition.
\n\n\nBrain Bert: https://arxiv.org/abs/2302.14367 [ICLR 2023]\n
\n
Github Code: https://github.com/czlwang/BrainBERT
\nPretrained weights: https://www.kaggle.com/datasets/nischaydnk/brainbertweights
you can load the model with a configuration like this:
\n torch.nn nn\n\n (nn.Module): \n ():\n (SpecPredictionHead, self).__init__()\n cfg.hidden_dim = \n self.hidden_layer = nn.Linear(cfg.hidden_dim, cfg.hidden_dim)\n self.act_fn = \n\n self.layer_norm = nn.LayerNorm(cfg.hidden_dim)\n self.output = nn.Linear(cfg.hidden_dim, num_classes)\n\n ():\n h = self.hidden_layer(hidden)\n h = self.layer_norm(h)\n h = self.output(h)\n h\n\n (torch.nn.Module):\n\n ():\n ().__init__()\n ckpt_path = \n cfg = OmegaConf.create({: ckpt_path})\n self.model = build_model(cfg)\n init_state = torch.load(ckpt_path)\n load_model_weights(self.model, init_state[], )\n\n self.head = SpecPredictionHead(cfg, num_classes)\n\n ():\n x = x.squeeze()\n mask = mask.squeeze()\n\n h = self.model(x, mask)[] \n h = h[:,-,:]\n h = self.head(h)\n h\n
\nBRANT: https://openreview.net/forum?id=DDkl9vaJyE [Neurips 2023]\n
\n
Github Code: https://zju-brainnet.github.io/Brant.github.io/
\nModel weights: https://drive.google.com/file/d/1QzxTNBvgcJBRxa8W2mNq2Tj967GtlDLF/view
\nBRANT-2: https://arxiv.org/abs/2402.10251 [March 2024]\n
\n
Github: https://github.com/yzz673/Brant-2
\n
\nSimMTM: https://arxiv.org/abs/2302.00861[Neurips 2023]\n
\n
Github: https://github.com/thuml/SimMTM
\nI tried various architectures published, but no success in getting great scores. Maybe transformers aren't the best architecture for this task.
\n
Comparison of all these pretrained models, shown in Brant-2 paper 👆
", "rawMarkdown": "I was recently exploring foundational models which are specifically pretrained on EEG/Brain signals. I found couple of really good reads which can be helpful for the competition. \n\n\n\nBrain Bert: https://arxiv.org/abs/2302.14367 [ICLR 2023]\nGithub Code: https://github.com/czlwang/BrainBERT\nPretrained weights: https://www.kaggle.com/datasets/nischaydnk/brainbertweights\n\n**you can load the model with a configuration like this:**\n```\nimport torch.nn as nn\n\nclass SpecPredictionHead(nn.Module): \n def __init__(self, cfg, num_classes):\n super(SpecPredictionHead, self).__init__()\n cfg.hidden_dim = 768\n self.hidden_layer = nn.Linear(cfg.hidden_dim, cfg.hidden_dim)\n self.act_fn = None\n\n self.layer_norm = nn.LayerNorm(cfg.hidden_dim)\n self.output = nn.Linear(cfg.hidden_dim, num_classes)\n\n def forward(self, hidden):\n h = self.hidden_layer(hidden)\n h = self.layer_norm(h)\n h = self.output(h)\n return h\n\nclass HMSModel(torch.nn.Module):\n\n def __init__(\n self,\n in_channels: int,\n num_classes: int,\n ):\n super().__init__()\n ckpt_path = \"stft_large_pretrained.pth\"\n cfg = OmegaConf.create({\"upstream_ckpt\": ckpt_path})\n self.model = build_model(cfg)\n init_state = torch.load(ckpt_path)\n load_model_weights(self.model, init_state['model'], False)\n \n self.head = SpecPredictionHead(cfg, num_classes)\n \n def forward(self, x, mask):\n x = x.squeeze(1)\n mask = mask.squeeze(1)\n \n h = self.model(x, mask)[0] \n h = h[:,-1,:]\n h = self.head(h)\n return h\n\n```\n\n\n\n\n\n
\n
\nBRANT: https://openreview.net/forum?id=DDkl9vaJyE [Neurips 2023]\nGithub Code: https://zju-brainnet.github.io/Brant.github.io/
\n
\nBRANT-2: https://arxiv.org/abs/2402.10251 [March 2024]\nGithub: https://github.com/yzz673/Brant-2\n\n\n\n\n\n
\n
\nSimMTM: https://arxiv.org/abs/2302.00861[Neurips 2023]\nGithub: https://github.com/thuml/SimMTM\n\n\n\nI tried various architectures published, but no success in getting great scores. Maybe transformers aren't the best architecture for this task. \n\n\n\nComparison of all these pretrained models, shown in Brant-2 paper 👆\n", "votes": 75 }, { "id": 2697348, "postDate": "2024-03-14T20:36:21.660Z", "content": "
\n
I used ViTMAE backbone, weights pre-trained on ImageNet. I have re-run the pre-train on spectrograms, achieved quite good reconstruction of the masked spectrograms (by visual comparison & cross-entropy loss).
\nBut when using the weights to predict labels, my best LB was only 0.44. As comparison, my EfficientNet hit a LB score of 0.37.
Great insight, thanks!!
", "rawMarkdown": "Great insight, thanks!!" } ] }, { "id": 2694740, "postDate": "2024-03-13T08:39:23.437Z", "content": "have you looked at LARGE BRAIN MODEL FOR LEARNING GENERIC REPRESENTATIONS WITH TREMENDOUS EEG DATA IN BCI ( mentioned in this post)
\ncode is available at https://github.com/935963004/LaBraM
some of the review papers considered next steps for future work to include using pretrained BCI models, e.g. for emotion recognition, perhaps will be more in this space over time.
", "rawMarkdown": "have you looked at [LARGE BRAIN MODEL FOR LEARNING GENERIC REPRESENTATIONS WITH TREMENDOUS EEG DATA IN BCI](https://openreview.net/pdf?id=QzTpTRVtrP) ( [mentioned in this post](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/481700#2682313)) \ncode is available at https://github.com/935963004/LaBraM\n\nsome of the review papers considered next steps for future work to include using pretrained BCI models, e.g. for emotion recognition, perhaps will be more in this space over time.", "votes": 2 }, { "id": 2694686, "postDate": "2024-03-13T07:53:09.943Z", "content": "Thank you for sharing!
\n\n\nI tried various architectures published, but no success in getting great scores. Maybe transformers aren't the best architecture for this task.
\n
Do you use pretrained weights and finetune these models? Is it possible to open approximate scores you got? I wonder if it is better than 1D EEGNet you have shared already.
", "rawMarkdown": "Thank you for sharing!\n>I tried various architectures published, but no success in getting great scores. Maybe transformers aren't the best architecture for this task. \n\nDo you use pretrained weights and finetune these models? Is it possible to open approximate scores you got? I wonder if it is better than [1D EEGNet you have shared already](https://www.kaggle.com/code/nischaydnk/lightning-1d-eegnet-training-pipeline-hbs).", "votes": 2, "replies": [ { "id": 2694753, "postDate": "2024-03-13T08:59:48.787Z", "content": "Hi. Haven't experimented much with these models, the closest I could get was around ~0.80 CV in comparison to 0.72 CV of 1D Resnet baseline, I used BRANT.
", "rawMarkdown": "Hi. Haven't experimented much with these models, the closest I could get was around ~0.80 CV in comparison to 0.72 CV of 1D Resnet baseline, I used BRANT. ", "votes": 1, "replies": [ { "id": 2694835, "postDate": "2024-03-13T10:12:13.463Z", "content": "Thank you for your reply. I have seen so many 'SOTA' models/techniques not work well in kaggle😅.
", "rawMarkdown": "Thank you for your reply. I have seen so many 'SOTA' models/techniques not work well in kaggle😅.", "votes": 5 } ] } ] }, { "id": 2700328, "postDate": "2024-03-16T13:19:46.637Z", "content": "Thank your for the advice and the given model. The discussion inspired me a lot.
", "rawMarkdown": "Thank your for the advice and the given model. The discussion inspired me a lot. " }, { "id": 2697455, "postDate": "2024-03-14T22:41:16.683Z", "content": "interesting that you found using BrainBERT didn't give great results. could you post the code you used for your test please? would be interesting to see what you tried
", "rawMarkdown": "interesting that you found using BrainBERT didn't give great results. could you post the code you used for your test please? would be interesting to see what you tried" }, { "id": 2695173, "postDate": "2024-03-13T14:32:31.793Z", "content": "So pretrain does not help?
", "rawMarkdown": "So pretrain does not help?", "replies": [ { "id": 2695558, "postDate": "2024-03-13T18:25:35.613Z", "content": "Atleast for now, we couldn't make any of these model to perform well. But definitely, these papers bring some novel ideas to implement which may help in the competition.
", "rawMarkdown": "Atleast for now, we couldn't make any of these model to perform well. But definitely, these papers bring some novel ideas to implement which may help in the competition. ", "votes": 2, "replies": [ { "id": 2699855, "postDate": "2024-03-16T07:42:27.977Z", "content": "Indeed, the ideas are novel and a good read.
", "rawMarkdown": "Indeed, the ideas are novel and a good read. " } ] } ] }, { "id": 2700123, "postDate": "2024-03-16T11:03:12.767Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2696151, "postDate": "2024-03-14T05:32:33.647Z", "content": "thanks for discussion
", "rawMarkdown": "thanks for discussion", "votes": 1 }, { "id": 2695036, "postDate": "2024-03-13T12:29:08.420Z", "content": "thanks for sharing @nischaydnk
", "rawMarkdown": "thanks for sharing @nischaydnk ", "votes": 1 }, { "id": 2694766, "postDate": "2024-03-13T09:08:26.370Z", "content": "Thanks for share.
", "rawMarkdown": "Thanks for share.", "votes": 1 }, { "id": 2708458, "postDate": "2024-03-21T03:16:59.493Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing" }, { "id": 2706337, "postDate": "2024-03-19T20:22:31.263Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing." }, { "id": 2704944, "postDate": "2024-03-19T02:31:04.620Z", "content": "thanks for sharing
", "rawMarkdown": "thanks for sharing" }, { "id": 2700210, "postDate": "2024-03-16T11:59:26.617Z", "content": "thanks for sharing🥳
", "rawMarkdown": "thanks for sharing🥳" } ], "comments": [ { "id": 2697348, "author_name": "SLi", "author_url": "", "post_date": "2024-03-14T20:36:21.660000", "content": "I used ViTMAE backbone, weights pre-trained on ImageNet. I have re-run the pre-train on spectrograms, achieved quite good reconstruction of the masked spectrograms (by visual comparison & cross-entropy loss).
\nBut when using the weights to predict labels, my best LB was only 0.44. As comparison, my EfficientNet hit a LB score of 0.37.
Great insight, thanks!!
", "votes": 0, "replies": [] } ] }, { "id": 2694740, "author_name": "something4kag", "author_url": "", "post_date": "2024-03-13T08:39:23.437000", "content": "have you looked at LARGE BRAIN MODEL FOR LEARNING GENERIC REPRESENTATIONS WITH TREMENDOUS EEG DATA IN BCI ( mentioned in this post)
\ncode is available at https://github.com/935963004/LaBraM
some of the review papers considered next steps for future work to include using pretrained BCI models, e.g. for emotion recognition, perhaps will be more in this space over time.
", "votes": 2, "replies": [] }, { "id": 2694686, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2024-03-13T07:53:09.943000", "content": "Thank you for sharing!
\n\n\nI tried various architectures published, but no success in getting great scores. Maybe transformers aren't the best architecture for this task.
\n
Do you use pretrained weights and finetune these models? Is it possible to open approximate scores you got? I wonder if it is better than 1D EEGNet you have shared already.
", "votes": 2, "replies": [ { "id": 2694753, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2024-03-13T08:59:48.787000", "content": "Hi. Haven't experimented much with these models, the closest I could get was around ~0.80 CV in comparison to 0.72 CV of 1D Resnet baseline, I used BRANT.
", "votes": 1, "replies": [ { "id": 2694835, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2024-03-13T10:12:13.463000", "content": "Thank you for your reply. I have seen so many 'SOTA' models/techniques not work well in kaggle😅.
", "votes": 5, "replies": [] } ] } ] }, { "id": 2700328, "author_name": "WU YUHUI 902", "author_url": "", "post_date": "2024-03-16T13:19:46.637000", "content": "Thank your for the advice and the given model. The discussion inspired me a lot.
", "votes": 0, "replies": [] }, { "id": 2697455, "author_name": "Mandeep", "author_url": "", "post_date": "2024-03-14T22:41:16.683000", "content": "interesting that you found using BrainBERT didn't give great results. could you post the code you used for your test please? would be interesting to see what you tried
", "votes": 0, "replies": [] }, { "id": 2695173, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-03-13T14:32:31.793000", "content": "So pretrain does not help?
", "votes": 0, "replies": [ { "id": 2695558, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2024-03-13T18:25:35.613000", "content": "Atleast for now, we couldn't make any of these model to perform well. But definitely, these papers bring some novel ideas to implement which may help in the competition.
", "votes": 2, "replies": [ { "id": 2699855, "author_name": "Muhammad Bilal Khan", "author_url": "", "post_date": "2024-03-16T07:42:27.977000", "content": "Indeed, the ideas are novel and a good read.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2700123, "author_name": "", "author_url": "", "post_date": "2024-03-16T11:03:12.767000", "content": "", "votes": 0, "replies": [] }, { "id": 2696151, "author_name": "XKG-", "author_url": "", "post_date": "2024-03-14T05:32:33.647000", "content": "thanks for discussion
", "votes": 1, "replies": [] }, { "id": 2695036, "author_name": "Adnan Alaref", "author_url": "", "post_date": "2024-03-13T12:29:08.420000", "content": "thanks for sharing @nischaydnk
", "votes": 1, "replies": [] }, { "id": 2694766, "author_name": "Ángel Jacinto Sánchez Ruiz", "author_url": "", "post_date": "2024-03-13T09:08:26.370000", "content": "Thanks for share.
", "votes": 1, "replies": [] }, { "id": 2708458, "author_name": "Li Peilin", "author_url": "", "post_date": "2024-03-21T03:16:59.493000", "content": "Thanks for sharing
", "votes": 0, "replies": [] }, { "id": 2706337, "author_name": "Yuqi Li_7629", "author_url": "", "post_date": "2024-03-19T20:22:31.263000", "content": "Thanks for sharing.
", "votes": 0, "replies": [] }, { "id": 2704944, "author_name": "theAAx", "author_url": "", "post_date": "2024-03-19T02:31:04.620000", "content": "thanks for sharing
", "votes": 0, "replies": [] }, { "id": 2700210, "author_name": "RogerOcean", "author_url": "", "post_date": "2024-03-16T11:59:26.617000", "content": "thanks for sharing🥳
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2694638": "I was recently exploring foundational models which are specifically pretrained on EEG/Brain signals. I found couple of really good reads which can be helpful for the competition. \n\n\n\nBrain Bert: https://arxiv.org/abs/2302.14367 [ICLR 2023]\nGithub Code: https://github.com/czlwang/BrainBERT\nPretrained weights: https://www.kaggle.com/datasets/nischaydnk/brainbertweights\n\n**you can load the model with a configuration like this:**\n```\nimport torch.nn as nn\n\nclass SpecPredictionHead(nn.Module): \n def __init__(self, cfg, num_classes):\n super(SpecPredictionHead, self).__init__()\n cfg.hidden_dim = 768\n self.hidden_layer = nn.Linear(cfg.hidden_dim, cfg.hidden_dim)\n self.act_fn = None\n\n self.layer_norm = nn.LayerNorm(cfg.hidden_dim)\n self.output = nn.Linear(cfg.hidden_dim, num_classes)\n\n def forward(self, hidden):\n h = self.hidden_layer(hidden)\n h = self.layer_norm(h)\n h = self.output(h)\n return h\n\nclass HMSModel(torch.nn.Module):\n\n def __init__(\n self,\n in_channels: int,\n num_classes: int,\n ):\n super().__init__()\n ckpt_path = \"stft_large_pretrained.pth\"\n cfg = OmegaConf.create({\"upstream_ckpt\": ckpt_path})\n self.model = build_model(cfg)\n init_state = torch.load(ckpt_path)\n load_model_weights(self.model, init_state['model'], False)\n \n self.head = SpecPredictionHead(cfg, num_classes)\n \n def forward(self, x, mask):\n x = x.squeeze(1)\n mask = mask.squeeze(1)\n \n h = self.model(x, mask)[0] \n h = h[:,-1,:]\n h = self.head(h)\n return h\n\n```\n\n\n\n\n\n
\n
\nBRANT: https://openreview.net/forum?id=DDkl9vaJyE [Neurips 2023]\nGithub Code: https://zju-brainnet.github.io/Brant.github.io/
\n
\nBRANT-2: https://arxiv.org/abs/2402.10251 [March 2024]\nGithub: https://github.com/yzz673/Brant-2\n\n\n\n\n\n
\n
\nSimMTM: https://arxiv.org/abs/2302.00861[Neurips 2023]\nGithub: https://github.com/thuml/SimMTM\n\n\n\nI tried various architectures published, but no success in getting great scores. Maybe transformers aren't the best architecture for this task. \n\n\n\nComparison of all these pretrained models, shown in Brant-2 paper 👆\n", "2697348": "I used ViTMAE backbone, weights pre-trained on ImageNet. I have re-run the pre-train on spectrograms, achieved quite good reconstruction of the masked spectrograms (by visual comparison & cross-entropy loss). \nBut when using the weights to predict labels, my best LB was only 0.44. As comparison, my EfficientNet hit a LB score of 0.37.", "2694740": "have you looked at [LARGE BRAIN MODEL FOR LEARNING GENERIC REPRESENTATIONS WITH TREMENDOUS EEG DATA IN BCI](https://openreview.net/pdf?id=QzTpTRVtrP) ( [mentioned in this post](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/481700#2682313)) \ncode is available at https://github.com/935963004/LaBraM\n\nsome of the review papers considered next steps for future work to include using pretrained BCI models, e.g. for emotion recognition, perhaps will be more in this space over time.", "2694686": "Thank you for sharing!\n>I tried various architectures published, but no success in getting great scores. Maybe transformers aren't the best architecture for this task. \n\nDo you use pretrained weights and finetune these models? Is it possible to open approximate scores you got? I wonder if it is better than [1D EEGNet you have shared already](https://www.kaggle.com/code/nischaydnk/lightning-1d-eegnet-training-pipeline-hbs).", "2700328": "Thank your for the advice and the given model. The discussion inspired me a lot. ", "2697455": "interesting that you found using BrainBERT didn't give great results. could you post the code you used for your test please? would be interesting to see what you tried", "2695173": "So pretrain does not help?", "2700123": "", "2696151": "thanks for discussion", "2695036": "thanks for sharing @nischaydnk ", "2694766": "Thanks for share.", "2708458": "Thanks for sharing", "2706337": "Thanks for sharing.", "2704944": "thanks for sharing", "2700210": "thanks for sharing🥳" } }
\n