{ "id": 480674, "title": "Need for New Insights", "url": "/competitions/hms-harmful-brain-activity-classification/discussion/480674", "author_name": "", "post_date": "2024-02-29T14:51:30.753947100Z", "votes": 28, "comment_count": 13, "views": 0, "content": "

The competition is already halfway through, and many high scoring notebooks have been shared.
\nI would like to share some of my own experimental results, hoping to provide you with reference.

\n
    \n
  1. My notebook is based on:
    \nhttps://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train
    \nhttps://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-inference
    \nand the baseline lb is 0.43.
    2. \n
  2. Add mixup: lb 0.41
    3. \n
  3. Use vit: lb 0.4
    4. \n
  4. Add augmentation based on https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/479776: lb 0.39
    5. \n
  5. Use 2 stages training based on https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461: lb 0.35
    6. \n
  6. Ensemble with the same weights: lb 0.34
    7. \n
\n

So far, I have used most of the methods discussed. I think the key lies in data processing and understanding of labels. We need new insights rather than blindly changing models or ensemble weights.
\nWelcome to share your ideas!

", "messages": [ { "id": "2674811", "postDate": "02/29/2024 14:51:30", "content": "

The competition is already halfway through, and many high scoring notebooks have been shared.
\nI would like to share some of my own experimental results, hoping to provide you with reference.

\n
    \n
  1. My notebook is based on:
    \nhttps://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train
    \nhttps://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-inference
    \nand the baseline lb is 0.43.
    2. \n
  2. Add mixup: lb 0.41
    3. \n
  3. Use vit: lb 0.4
    4. \n
  4. Add augmentation based on https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/479776: lb 0.39
    5. \n
  5. Use 2 stages training based on https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461: lb 0.35
    6. \n
  6. Ensemble with the same weights: lb 0.34
    7. \n
\n

So far, I have used most of the methods discussed. I think the key lies in data processing and understanding of labels. We need new insights rather than blindly changing models or ensemble weights.
\nWelcome to share your ideas!

", "rawMarkdown": "The competition is already halfway through, and many high scoring notebooks have been shared.\nI would like to share some of my own experimental results, hoping to provide you with reference.\n\n1. My notebook is based on:\nhttps://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train\nhttps://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-inference\nand the **baseline** lb is **0.43**.\n2. Add **mixup**: lb **0.41**\n3. Use **vit**: lb **0.4**\n4. Add **augmentation** based on https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/479776: lb **0.39**\n5. Use **2 stages training** based on https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461: lb **0.35**\n6. **Ensemble** with the same weights: lb **0.34**\n\nSo far, I have used most of the methods discussed. I think the key lies in data processing and understanding of labels. We need new insights rather than blindly changing models or ensemble weights. \nWelcome to share your ideas!", "votes": null }, { "id": "2675091", "postDate": "02/29/2024 17:41:27", "content": "

Thank you for sharing this, it's very helpful! BTW, I would like to ask is your ensemble model composed of ViT and EfficientNet?

", "rawMarkdown": "Thank you for sharing this, it's very helpful! BTW, I would like to ask is your ensemble model composed of ViT and EfficientNet?", "votes": null }, { "id": "2675096", "postDate": "02/29/2024 17:44:03", "content": "

Hi, here is my ideas:

\n

I'm training models with a weighted sampler, but I haven't done the submission yet.

\n

I also think about a compound loss, maybe (KL + weighted CE with weights=expert votes).

\n

I'm experimenting with weight_decay against overfitting.

\n

Also I'm in the process of finding strong and justifiable augmentations for our data.

\n

And I also have questions regarding the preprocessing of data into 2d spectrograms, I would like to know how they preprocess and what kind of 2d data people from top LB receive for their models.

", "rawMarkdown": "Hi, here is my ideas:\n\nI'm training models with a weighted sampler, but I haven't done the submission yet.\n\nI also think about a compound loss, maybe (KL + weighted CE with weights=expert votes).\n\nI'm experimenting with weight_decay against overfitting.\n\nAlso I'm in the process of finding strong and justifiable augmentations for our data.\n\nAnd I also have questions regarding the preprocessing of data into 2d spectrograms, I would like to know how they preprocess and what kind of 2d data people from top LB receive for their models.", "votes": null }, { "id": "2675230", "postDate": "02/29/2024 19:04:17", "content": "

Good added information, I saw that you shared some links, cool, was the use of vit using timm pythorch models?

", "rawMarkdown": "Good added information, I saw that you shared some links, cool, was the use of vit using timm pythorch models?", "votes": null }, { "id": "2676313", "postDate": "03/01/2024 12:13:19", "content": "

Thank you for sharing. It is surprising that the improving way and its score above is almost same as me except using vit.

", "rawMarkdown": "Thank you for sharing. It is surprising that the improving way and its score above is almost same as me except using vit.", "votes": null }, { "id": "2676677", "postDate": "03/01/2024 16:50:45", "content": "

I Think the next potential improvement could involve incorporating a self-attention layer. This addition has the potential to significantly boost performance, potentially aligning with techniques used by top 0.2x Kagglers.
\nI may be mistaken, experimenting with this approach is a priority for me.

", "rawMarkdown": "I Think the next potential improvement could involve incorporating a **self-attention** layer. This addition has the potential to significantly boost performance, potentially aligning with techniques used by top 0.2x Kagglers. \nI may be mistaken, experimenting with this approach is a priority for me.", "votes": null }, { "id": "2678745", "postDate": "03/03/2024 03:03:19", "content": "

I'm in the same situation. My models are based on kaggle and eeg spectrograms. Ensemble of them can boost LB, but the effect seems to be limited (0.02~0.04).
\nWe do not have public notebooks which achieve high LB with raw eegs, but in discussion there are some comments on high-LB models using them. Is using raw eegs a key? I will try.

", "rawMarkdown": "I'm in the same situation. My models are based on kaggle and eeg spectrograms. Ensemble of them can boost LB, but the effect seems to be limited (0.02~0.04). \nWe do not have public notebooks which achieve high LB with raw eegs, but in discussion there are some comments on high-LB models using them. Is using raw eegs a key? I will try.", "votes": null }, { "id": "2680098", "postDate": "03/04/2024 01:35:25", "content": "

good insights

", "rawMarkdown": "good insights", "votes": null }, { "id": "2687244", "postDate": "03/08/2024 11:43:48", "content": "

I am currently improving my raw eeg models and so far reach 0.37 LB with them. I have not even tweaked the model I use so far. so I think, that there is a lot to gain from it.

\n

Best,
\nJan

", "rawMarkdown": "I am currently improving my raw eeg models and so far reach 0.37 LB with them. I have not even tweaked the model I use so far. so I think, that there is a lot to gain from it.\n\nBest,\nJan", "votes": null }, { "id": "2687859", "postDate": "03/08/2024 20:22:52", "content": "

@Jan Brederecke what sort of models have you played with if you dont mind me asking?

", "rawMarkdown": "Jan Brederecke what sort of models have you played with if you dont mind me asking?", "votes": null }, { "id": "2690779", "postDate": "03/10/2024 19:24:34", "content": "

So far I have used very simple 1D CNNs with ResNet architecture. I tried combining them with an encoder but it did not really improve my results so far. I will play with this more in the upcoming days and if you remind me, I might share some more insights if successful!
\nBest,
\nJan

", "rawMarkdown": "So far I have used very simple 1D CNNs with ResNet architecture. I tried combining them with an encoder but it did not really improve my results so far. I will play with this more in the upcoming days and if you remind me, I might share some more insights if successful! \nBest,\nJan", "votes": null }, { "id": "2701981", "postDate": "03/17/2024 10:43:22", "content": "

maybe you can add more spec data from eeg with different kind of converting methods😀

", "rawMarkdown": "maybe you can add more spec data from eeg with different kind of converting methods😀", "votes": null }, { "id": "2704139", "postDate": "03/18/2024 15:30:05", "content": "

Hey, did you succeed with the weighted sampling? I didn’t 😅. That’s why I ask…

\n

Best,
\nJan

", "rawMarkdown": "Hey, did you succeed with the weighted sampling? I didn’t 😅. That’s why I ask…\n\nBest,\nJan", "votes": null }, { "id": "2704568", "postDate": "03/18/2024 19:57:00", "content": "

Hi, this did not give me any profit, but I think it increased the stability of the solution

", "rawMarkdown": "Hi, this did not give me any profit, but I think it increased the stability of the solution", "votes": null } ], "comments": [ { "id": 2675091, "author_name": "zachary666", "author_url": "", "post_date": "02/29/2024 17:41:27", "content": "

Thank you for sharing this, it's very helpful! BTW, I would like to ask is your ensemble model composed of ViT and EfficientNet?

", "votes": null, "replies": [] }, { "id": 2675096, "author_name": "egorgij21", "author_url": "", "post_date": "02/29/2024 17:44:03", "content": "

Hi, here is my ideas:

\n

I'm training models with a weighted sampler, but I haven't done the submission yet.

\n

I also think about a compound loss, maybe (KL + weighted CE with weights=expert votes).

\n

I'm experimenting with weight_decay against overfitting.

\n

Also I'm in the process of finding strong and justifiable augmentations for our data.

\n

And I also have questions regarding the preprocessing of data into 2d spectrograms, I would like to know how they preprocess and what kind of 2d data people from top LB receive for their models.

", "votes": null, "replies": [ { "id": 2704139, "author_name": "janbrederecke", "author_url": "", "post_date": "03/18/2024 15:30:05", "content": "

Hey, did you succeed with the weighted sampling? I didn’t 😅. That’s why I ask…

\n

Best,
\nJan

", "votes": null, "replies": [ { "id": 2704568, "author_name": "egorgij21", "author_url": "", "post_date": "03/18/2024 19:57:00", "content": "

Hi, this did not give me any profit, but I think it increased the stability of the solution

", "votes": null, "replies": [] } ] } ] }, { "id": 2675230, "author_name": "rafaelzimmermann1", "author_url": "", "post_date": "02/29/2024 19:04:17", "content": "

Good added information, I saw that you shared some links, cool, was the use of vit using timm pythorch models?

", "votes": null, "replies": [] }, { "id": 2676313, "author_name": "clearwaterkzk", "author_url": "", "post_date": "03/01/2024 12:13:19", "content": "

Thank you for sharing. It is surprising that the improving way and its score above is almost same as me except using vit.

", "votes": null, "replies": [] }, { "id": 2676677, "author_name": "nartaa", "author_url": "", "post_date": "03/01/2024 16:50:45", "content": "

I Think the next potential improvement could involve incorporating a self-attention layer. This addition has the potential to significantly boost performance, potentially aligning with techniques used by top 0.2x Kagglers.
\nI may be mistaken, experimenting with this approach is a priority for me.

", "votes": null, "replies": [] }, { "id": 2678745, "author_name": "symz00", "author_url": "", "post_date": "03/03/2024 03:03:19", "content": "

I'm in the same situation. My models are based on kaggle and eeg spectrograms. Ensemble of them can boost LB, but the effect seems to be limited (0.02~0.04).
\nWe do not have public notebooks which achieve high LB with raw eegs, but in discussion there are some comments on high-LB models using them. Is using raw eegs a key? I will try.

", "votes": null, "replies": [ { "id": 2687244, "author_name": "janbrederecke", "author_url": "", "post_date": "03/08/2024 11:43:48", "content": "

I am currently improving my raw eeg models and so far reach 0.37 LB with them. I have not even tweaked the model I use so far. so I think, that there is a lot to gain from it.

\n

Best,
\nJan

", "votes": null, "replies": [ { "id": 2687859, "author_name": "tashin47", "author_url": "", "post_date": "03/08/2024 20:22:52", "content": "

@Jan Brederecke what sort of models have you played with if you dont mind me asking?

", "votes": null, "replies": [ { "id": 2690779, "author_name": "janbrederecke", "author_url": "", "post_date": "03/10/2024 19:24:34", "content": "

So far I have used very simple 1D CNNs with ResNet architecture. I tried combining them with an encoder but it did not really improve my results so far. I will play with this more in the upcoming days and if you remind me, I might share some more insights if successful!
\nBest,
\nJan

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2680098, "author_name": "doxgxxn", "author_url": "", "post_date": "03/04/2024 01:35:25", "content": "

good insights

", "votes": null, "replies": [] }, { "id": 2701981, "author_name": "roger92", "author_url": "", "post_date": "03/17/2024 10:43:22", "content": "

maybe you can add more spec data from eeg with different kind of converting methods😀

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2674811": "The competition is already halfway through, and many high scoring notebooks have been shared.\nI would like to share some of my own experimental results, hoping to provide you with reference.\n\n1. My notebook is based on:\nhttps://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train\nhttps://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-inference\nand the **baseline** lb is **0.43**.\n2. Add **mixup**: lb **0.41**\n3. Use **vit**: lb **0.4**\n4. Add **augmentation** based on https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/479776: lb **0.39**\n5. Use **2 stages training** based on https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461: lb **0.35**\n6. **Ensemble** with the same weights: lb **0.34**\n\nSo far, I have used most of the methods discussed. I think the key lies in data processing and understanding of labels. We need new insights rather than blindly changing models or ensemble weights. \nWelcome to share your ideas!", "2675091": "Thank you for sharing this, it's very helpful! BTW, I would like to ask is your ensemble model composed of ViT and EfficientNet?", "2675096": "Hi, here is my ideas:\n\nI'm training models with a weighted sampler, but I haven't done the submission yet.\n\nI also think about a compound loss, maybe (KL + weighted CE with weights=expert votes).\n\nI'm experimenting with weight_decay against overfitting.\n\nAlso I'm in the process of finding strong and justifiable augmentations for our data.\n\nAnd I also have questions regarding the preprocessing of data into 2d spectrograms, I would like to know how they preprocess and what kind of 2d data people from top LB receive for their models.", "2675230": "Good added information, I saw that you shared some links, cool, was the use of vit using timm pythorch models?", "2676313": "Thank you for sharing. It is surprising that the improving way and its score above is almost same as me except using vit.", "2676677": "I Think the next potential improvement could involve incorporating a **self-attention** layer. This addition has the potential to significantly boost performance, potentially aligning with techniques used by top 0.2x Kagglers. \nI may be mistaken, experimenting with this approach is a priority for me.", "2678745": "I'm in the same situation. My models are based on kaggle and eeg spectrograms. Ensemble of them can boost LB, but the effect seems to be limited (0.02~0.04). \nWe do not have public notebooks which achieve high LB with raw eegs, but in discussion there are some comments on high-LB models using them. Is using raw eegs a key? I will try.", "2680098": "good insights", "2687244": "I am currently improving my raw eeg models and so far reach 0.37 LB with them. I have not even tweaked the model I use so far. so I think, that there is a lot to gain from it.\n\nBest,\nJan", "2687859": "Jan Brederecke what sort of models have you played with if you dont mind me asking?", "2690779": "So far I have used very simple 1D CNNs with ResNet architecture. I tried combining them with an encoder but it did not really improve my results so far. I will play with this more in the upcoming days and if you remind me, I might share some more insights if successful! \nBest,\nJan", "2701981": "maybe you can add more spec data from eeg with different kind of converting methods😀", "2704139": "Hey, did you succeed with the weighted sampling? I didn’t 😅. That’s why I ask…\n\nBest,\nJan", "2704568": "Hi, this did not give me any profit, but I think it increased the stability of the solution" }, "source": "meta" }