Hi Everyone,
\nHas anyone here taken a look at transformers for this problem? There are a couple scenarios you could imagine applying them in (ViT, spatial temporal transformers on EEGs, etc), and I'm wondering if it's something that's worked out for folks yet.
\nTransformers are still a little new for me, so I'm hoping to use the last month of this competition as an excuse to learn them a bit better and try some things. The rest of this month I'm mostly focused on trying to improve my convnet approach.
", "messages": [ { "id": "2658855", "postDate": "02/19/2024 13:00:50", "content": "Hi Everyone,
\nHas anyone here taken a look at transformers for this problem? There are a couple scenarios you could imagine applying them in (ViT, spatial temporal transformers on EEGs, etc), and I'm wondering if it's something that's worked out for folks yet.
\nTransformers are still a little new for me, so I'm hoping to use the last month of this competition as an excuse to learn them a bit better and try some things. The rest of this month I'm mostly focused on trying to improve my convnet approach.
", "rawMarkdown": "Hi Everyone,\n\nHas anyone here taken a look at transformers for this problem? There are a couple scenarios you could imagine applying them in (ViT, spatial temporal transformers on EEGs, etc), and I'm wondering if it's something that's worked out for folks yet.\n\nTransformers are still a little new for me, so I'm hoping to use the last month of this competition as an excuse to learn them a bit better and try some things. The rest of this month I'm mostly focused on trying to improve my convnet approach.", "votes": null }, { "id": "2658962", "postDate": "02/19/2024 14:17:56", "content": "you could have a look in https://torcheeg.readthedocs.io/en/latest/torcheeg.models.html#transformer e.g. ATCNet
\nPaper: H. Altaheri, G. Muhammad and M. Alsulaiman, “Physics-Informed Attention Temporal Convolutional Network for EEG-Based Motor Imagery Classification,” in IEEE Transactions on Industrial Informatics, vol. 19, no. 2, pp. 2249-2258, Feb. 2023, doi: 10.1109/TII.2022.3197419.
\nURL: https://github.com/Altaheri/EEG-ATCNet
\ntorcheeg has other model ideas here as well.
", "rawMarkdown": "you could have a look in https://torcheeg.readthedocs.io/en/latest/torcheeg.models.html#transformer e.g. ATCNet\n\nPaper: H. Altaheri, G. Muhammad and M. Alsulaiman, “Physics-Informed Attention Temporal Convolutional Network for EEG-Based Motor Imagery Classification,” in IEEE Transactions on Industrial Informatics, vol. 19, no. 2, pp. 2249-2258, Feb. 2023, doi: 10.1109/TII.2022.3197419.\n\nURL: https://github.com/Altaheri/EEG-ATCNet\n\ntorcheeg has other model ideas here as well.", "votes": null }, { "id": "2659452", "postDate": "02/19/2024 20:56:41", "content": "I tried combining CNN as a feature extractor combined with a transformer encoder for the raw EEG but this did decrease performance so far…
", "rawMarkdown": "I tried combining CNN as a feature extractor combined with a transformer encoder for the raw EEG but this did decrease performance so far...", "votes": null }, { "id": "2659509", "postDate": "02/19/2024 22:41:27", "content": "I just tried that too. Kaggle only spectrograms: CV0.82, LB 0.57 (CNN to Encoder) vs CV 0.77, LB 0.56 (CNN only). I wouldn't say it's a definitive experiment, but it wasn't a quick win.
\nI have also noticed that reducing my model size (shallower, less filters, etc) doesn't always help LB score, but tends to help my CV. I think overfitting is a real problem in this competition and it's probably not great to have too strong a model unless you can find some nice augmentations. So far, augmentation has not impacted my LB score and has only had a marginal impact on my CVs.
", "rawMarkdown": "I just tried that too. Kaggle only spectrograms: CV0.82, LB 0.57 (CNN to Encoder) vs CV 0.77, LB 0.56 (CNN only). I wouldn't say it's a definitive experiment, but it wasn't a quick win. \n\nI have also noticed that reducing my model size (shallower, less filters, etc) doesn't always help LB score, but tends to help my CV. I think overfitting is a real problem in this competition and it's probably not great to have too strong a model unless you can find some nice augmentations. So far, augmentation has not impacted my LB score and has only had a marginal impact on my CVs.", "votes": null }, { "id": "2659510", "postDate": "02/19/2024 22:41:41", "content": "Thanks for the references :)
", "rawMarkdown": "Thanks for the references :)", "votes": null }, { "id": "2659580", "postDate": "02/20/2024 02:09:24", "content": "I did try the ViT but it did not have good results. ResNext gave me better both in CV and LB with the EfficientNet_B0 currently giving the best.
", "rawMarkdown": "I did try the ViT but it did not have good results. ResNext gave me better both in CV and LB with the EfficientNet_B0 currently giving the best.", "votes": null }, { "id": "2660232", "postDate": "02/20/2024 13:21:43", "content": "I can support your findings but augmentations really helped my CV and LB score!
", "rawMarkdown": "I can support your findings but augmentations really helped my CV and LB score!", "votes": null }, { "id": "2661048", "postDate": "02/21/2024 01:24:28", "content": "Good to know! I just need to find the right ones.
", "rawMarkdown": "Good to know! I just need to find the right ones.", "votes": null }, { "id": "2661094", "postDate": "02/21/2024 03:07:21", "content": "As my experiment,
\nAmong the transformers, tried maxvit_base_tf_512/convnext_base, which has a realistic number and size of parameters for inference time, but efficientnet was better.
As a side note, blends of CNN+Transformer are often seen in top solution in CV competitions, so I have hope for Transformers.
", "rawMarkdown": "As my experiment,\nAmong the transformers, tried maxvit_base_tf_512/convnext_base, which has a realistic number and size of parameters for inference time, but efficientnet was better.\n\nAs a side note, blends of CNN+Transformer are often seen in top solution in CV competitions, so I have hope for Transformers.", "votes": null }, { "id": "2662898", "postDate": "02/22/2024 07:43:46", "content": "Not sure about the transformers but RNN and LSTM didn't gave any results for me, I don't think seq2seq models will be giving good results here, if they are giving I will be keen to learn that
", "rawMarkdown": "Not sure about the transformers but RNN and LSTM didn't gave any results for me, I don't think seq2seq models will be giving good results here, if they are giving I will be keen to learn that", "votes": null }, { "id": "2662911", "postDate": "02/22/2024 08:05:20", "content": "I mostly wrote my own after trying lots of available augmentations that did not help CV.
\nBest,
\nJan
Ah, thank you very much for the tip!
", "rawMarkdown": "Ah, thank you very much for the tip!", "votes": null }, { "id": "2663393", "postDate": "02/22/2024 13:21:32", "content": "I'm very keen to see what folks end up using in this comp! There are a lot of ways to approach it
", "rawMarkdown": "I'm very keen to see what folks end up using in this comp! There are a lot of ways to approach it", "votes": null }, { "id": "2670485", "postDate": "02/26/2024 22:11:20", "content": "With the right augmentations I was now able to train a CNN-encoder combination on the raw EEG data that achieves 0.58 CV and 0.38 LB.
\nSo as long as you can provide enough data, it might work…
\nBest,
\nJan
Nice result! Finding the right augmentations seems to be a very important part of this competition. Life has gotten in the way a bit for me so I haven't had much chance to work on this comp recently, but I am hoping to come back this weekend!
", "rawMarkdown": "Nice result! Finding the right augmentations seems to be a very important part of this competition. Life has gotten in the way a bit for me so I haven't had much chance to work on this comp recently, but I am hoping to come back this weekend!", "votes": null } ], "comments": [ { "id": 2658962, "author_name": "something4kag", "author_url": "", "post_date": "02/19/2024 14:17:56", "content": "you could have a look in https://torcheeg.readthedocs.io/en/latest/torcheeg.models.html#transformer e.g. ATCNet
\nPaper: H. Altaheri, G. Muhammad and M. Alsulaiman, “Physics-Informed Attention Temporal Convolutional Network for EEG-Based Motor Imagery Classification,” in IEEE Transactions on Industrial Informatics, vol. 19, no. 2, pp. 2249-2258, Feb. 2023, doi: 10.1109/TII.2022.3197419.
\nURL: https://github.com/Altaheri/EEG-ATCNet
\ntorcheeg has other model ideas here as well.
", "votes": null, "replies": [ { "id": 2659510, "author_name": "chemdatafarmer", "author_url": "", "post_date": "02/19/2024 22:41:41", "content": "Thanks for the references :)
", "votes": null, "replies": [] } ] }, { "id": 2659452, "author_name": "janbrederecke", "author_url": "", "post_date": "02/19/2024 20:56:41", "content": "I tried combining CNN as a feature extractor combined with a transformer encoder for the raw EEG but this did decrease performance so far…
", "votes": null, "replies": [ { "id": 2659509, "author_name": "chemdatafarmer", "author_url": "", "post_date": "02/19/2024 22:41:27", "content": "I just tried that too. Kaggle only spectrograms: CV0.82, LB 0.57 (CNN to Encoder) vs CV 0.77, LB 0.56 (CNN only). I wouldn't say it's a definitive experiment, but it wasn't a quick win.
\nI have also noticed that reducing my model size (shallower, less filters, etc) doesn't always help LB score, but tends to help my CV. I think overfitting is a real problem in this competition and it's probably not great to have too strong a model unless you can find some nice augmentations. So far, augmentation has not impacted my LB score and has only had a marginal impact on my CVs.
", "votes": null, "replies": [ { "id": 2660232, "author_name": "janbrederecke", "author_url": "", "post_date": "02/20/2024 13:21:43", "content": "I can support your findings but augmentations really helped my CV and LB score!
", "votes": null, "replies": [ { "id": 2661048, "author_name": "chemdatafarmer", "author_url": "", "post_date": "02/21/2024 01:24:28", "content": "Good to know! I just need to find the right ones.
", "votes": null, "replies": [ { "id": 2662911, "author_name": "janbrederecke", "author_url": "", "post_date": "02/22/2024 08:05:20", "content": "I mostly wrote my own after trying lots of available augmentations that did not help CV.
\nBest,
\nJan
Ah, thank you very much for the tip!
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2670485, "author_name": "janbrederecke", "author_url": "", "post_date": "02/26/2024 22:11:20", "content": "With the right augmentations I was now able to train a CNN-encoder combination on the raw EEG data that achieves 0.58 CV and 0.38 LB.
\nSo as long as you can provide enough data, it might work…
\nBest,
\nJan
Nice result! Finding the right augmentations seems to be a very important part of this competition. Life has gotten in the way a bit for me so I haven't had much chance to work on this comp recently, but I am hoping to come back this weekend!
", "votes": null, "replies": [] } ] } ] }, { "id": 2659580, "author_name": "mohib94", "author_url": "", "post_date": "02/20/2024 02:09:24", "content": "I did try the ViT but it did not have good results. ResNext gave me better both in CV and LB with the EfficientNet_B0 currently giving the best.
", "votes": null, "replies": [] }, { "id": 2661094, "author_name": "hideyukizushi", "author_url": "", "post_date": "02/21/2024 03:07:21", "content": "As my experiment,
\nAmong the transformers, tried maxvit_base_tf_512/convnext_base, which has a realistic number and size of parameters for inference time, but efficientnet was better.
As a side note, blends of CNN+Transformer are often seen in top solution in CV competitions, so I have hope for Transformers.
", "votes": null, "replies": [ { "id": 2663393, "author_name": "chemdatafarmer", "author_url": "", "post_date": "02/22/2024 13:21:32", "content": "I'm very keen to see what folks end up using in this comp! There are a lot of ways to approach it
", "votes": null, "replies": [] } ] }, { "id": 2662898, "author_name": "anexperiencedidiot", "author_url": "", "post_date": "02/22/2024 07:43:46", "content": "Not sure about the transformers but RNN and LSTM didn't gave any results for me, I don't think seq2seq models will be giving good results here, if they are giving I will be keen to learn that
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2658855": "Hi Everyone,\n\nHas anyone here taken a look at transformers for this problem? There are a couple scenarios you could imagine applying them in (ViT, spatial temporal transformers on EEGs, etc), and I'm wondering if it's something that's worked out for folks yet.\n\nTransformers are still a little new for me, so I'm hoping to use the last month of this competition as an excuse to learn them a bit better and try some things. The rest of this month I'm mostly focused on trying to improve my convnet approach.", "2658962": "you could have a look in https://torcheeg.readthedocs.io/en/latest/torcheeg.models.html#transformer e.g. ATCNet\n\nPaper: H. Altaheri, G. Muhammad and M. Alsulaiman, “Physics-Informed Attention Temporal Convolutional Network for EEG-Based Motor Imagery Classification,” in IEEE Transactions on Industrial Informatics, vol. 19, no. 2, pp. 2249-2258, Feb. 2023, doi: 10.1109/TII.2022.3197419.\n\nURL: https://github.com/Altaheri/EEG-ATCNet\n\ntorcheeg has other model ideas here as well.", "2659452": "I tried combining CNN as a feature extractor combined with a transformer encoder for the raw EEG but this did decrease performance so far...", "2659509": "I just tried that too. Kaggle only spectrograms: CV0.82, LB 0.57 (CNN to Encoder) vs CV 0.77, LB 0.56 (CNN only). I wouldn't say it's a definitive experiment, but it wasn't a quick win. \n\nI have also noticed that reducing my model size (shallower, less filters, etc) doesn't always help LB score, but tends to help my CV. I think overfitting is a real problem in this competition and it's probably not great to have too strong a model unless you can find some nice augmentations. So far, augmentation has not impacted my LB score and has only had a marginal impact on my CVs.", "2659510": "Thanks for the references :)", "2659580": "I did try the ViT but it did not have good results. ResNext gave me better both in CV and LB with the EfficientNet_B0 currently giving the best.", "2660232": "I can support your findings but augmentations really helped my CV and LB score!", "2661048": "Good to know! I just need to find the right ones.", "2661094": "As my experiment,\nAmong the transformers, tried maxvit_base_tf_512/convnext_base, which has a realistic number and size of parameters for inference time, but efficientnet was better.\n\nAs a side note, blends of CNN+Transformer are often seen in top solution in CV competitions, so I have hope for Transformers.", "2662898": "Not sure about the transformers but RNN and LSTM didn't gave any results for me, I don't think seq2seq models will be giving good results here, if they are giving I will be keen to learn that", "2662911": "I mostly wrote my own after trying lots of available augmentations that did not help CV.\nBest,\nJan", "2663389": "Ah, thank you very much for the tip!", "2663393": "I'm very keen to see what folks end up using in this comp! There are a lot of ways to approach it", "2670485": "With the right augmentations I was now able to train a CNN-encoder combination on the raw EEG data that achieves 0.58 CV and 0.38 LB.\nSo as long as you can provide enough data, it might work...\nBest,\nJan", "2671310": "Nice result! Finding the right augmentations seems to be a very important part of this competition. Life has gotten in the way a bit for me so I haven't had much chance to work on this comp recently, but I am hoping to come back this weekend!" }, "source": "meta" }