![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9041364%2F17977936ff43f954b6ee62fca3e706d4%2FUntitled%20drawing%20(8).png?generation=1712622281054757&alt=media\")
Huge thanks to the Hosts and Kaggle, this competition was by far one of the most fun competitions I've competed in awhile.
\nTL;DR:
\nScores:
\nI selected two models, one for best CV on the whole dataset and one for best CV on samples >10 in case there was a shakeup. No shakeup, trusted my personal CV and it payed off :). Given that my original goal was ~0.29 when I join this competition a month ago (a bit late, really had to grind), I am very happy with my performance.
\nI feel if I had a bit more time, ~0.25 would be achievable (local CV / public LB) with my current solution, but that something was missing between what I have now and a sub 0.25 CV/LB score (probably some form of data psuedo-labeling, preprocessing, and/or selection). Can't wait to read everyone elses solutions and see how sub 0.25 scores were achieved.
\nFor the EEG, I used the banana montage (including the middle chain and the EKG). Nothing here is too different from what I saw other people doing, other than the fact that I normalize all 1D EEG signals by the MAD for robustness.
\nProprocessing (EEG):
\nProprocessing (EEG Spectrogram):
\nProprocessing (Spectrogram):
\nIn this competition especially, I found it very important to make sure that your normalize your data in a way that was robust to artifacts. And after playing around with quite a lot of different normalizations and visualizing the raw data, it was clear to me that the traditional standard deviation was NOT a robust measure and that it was extremely susceptible to artifacts in the case of the 1D EEGs. To this end, I instead chose to divide by the mean absolute deviation (MAD) and clip values between -10 and 10. This spead up convergence quite a bit, reduce overfitting, and also seemed to generalize better to my validation set.
\n1D EEGs:
\n2D Spectroggrams:
\nI tried working with test-time-augmentations (TTA) like flipping the orientation of chains, but it neither increased or decreased my CV/LB score in my later models. I think this is because I treat each node as an independent sample and used the same TTAs as augmentations during training. Which means by model tends to be robust to these shifts.
\nWhen I started this competition, I had a couple of goals for my model architecture that I wanted to achieve.
\nTo achieve these goals, my model architecture for each modality followed the same set of encoding/decoding principles:
\n
My training stages were an absolute disaster. I had 5-6 stages where I would start with easy samples to get the model to converge quick, then moved on to hard samples, and finalized with training on just the high confidence (>10 votes) samples.
\nSince the beginning of this competition, I've put a bit of thought into the future directions of this project after the competition and have some ideas for the competition hosts regarding how I could imagine technology like this being used in practice.
\nOne of a couple ideas that I've had would be to design an interpretable model with a kNN decoder. For example, imagine you deploy a model like this in the real world as an aid to doctors who are labeling or interpreting this data. Then, upon making a prediction, the model not only gives you a probabilistic output of what it thinks is true, but also looks up the nearest N neighbors neighbors in the embedding space (of the training set) and shows you those N samples as to say \"here is my prediction, but here are also 10 (for example) other EEGs and spectrograms that have similar patterns to this one and hence why I labeled it this way.\"
\nI have a lot of ideas like this that I'm going to be exploring over this summer as a side project (including reading/implementing other unique top submissions). I'll also be open sourcing it, so feel free to monitor my GitHub for updates (as well as the code for this project, which should be up in 2-3 days).
\nCongrats to the winners!
\nThanks Kaggle, and Hosts! This competition was by far one of the most fun I have joined. I learned a ton.
\nGitHub Training Code: https://github.com/ryanirl/hbac
\nKaggle Final Inference: https://www.kaggle.com/code/ryanirl/hms-baseline-sub
Ryan Peters
\nRyanIRL@icloud.com
Huge thanks to the Hosts and Kaggle, this competition was by far one of the most fun competitions I've competed in awhile.
\nTL;DR:
\nScores:
\nI selected two models, one for best CV on the whole dataset and one for best CV on samples >10 in case there was a shakeup. No shakeup, trusted my personal CV and it payed off :). Given that my original goal was ~0.29 when I join this competition a month ago (a bit late, really had to grind), I am very happy with my performance.
\nI feel if I had a bit more time, ~0.25 would be achievable (local CV / public LB) with my current solution, but that something was missing between what I have now and a sub 0.25 CV/LB score (probably some form of data psuedo-labeling, preprocessing, and/or selection). Can't wait to read everyone elses solutions and see how sub 0.25 scores were achieved.
\nFor the EEG, I used the banana montage (including the middle chain and the EKG). Nothing here is too different from what I saw other people doing, other than the fact that I normalize all 1D EEG signals by the MAD for robustness.
\nProprocessing (EEG):
\nProprocessing (EEG Spectrogram):
\nProprocessing (Spectrogram):
\nIn this competition especially, I found it very important to make sure that your normalize your data in a way that was robust to artifacts. And after playing around with quite a lot of different normalizations and visualizing the raw data, it was clear to me that the traditional standard deviation was NOT a robust measure and that it was extremely susceptible to artifacts in the case of the 1D EEGs. To this end, I instead chose to divide by the mean absolute deviation (MAD) and clip values between -10 and 10. This spead up convergence quite a bit, reduce overfitting, and also seemed to generalize better to my validation set.
\n1D EEGs:
\n2D Spectroggrams:
\nI tried working with test-time-augmentations (TTA) like flipping the orientation of chains, but it neither increased or decreased my CV/LB score in my later models. I think this is because I treat each node as an independent sample and used the same TTAs as augmentations during training. Which means by model tends to be robust to these shifts.
\nWhen I started this competition, I had a couple of goals for my model architecture that I wanted to achieve.
\nTo achieve these goals, my model architecture for each modality followed the same set of encoding/decoding principles:
\n
My training stages were an absolute disaster. I had 5-6 stages where I would start with easy samples to get the model to converge quick, then moved on to hard samples, and finalized with training on just the high confidence (>10 votes) samples.
\nSince the beginning of this competition, I've put a bit of thought into the future directions of this project after the competition and have some ideas for the competition hosts regarding how I could imagine technology like this being used in practice.
\nOne of a couple ideas that I've had would be to design an interpretable model with a kNN decoder. For example, imagine you deploy a model like this in the real world as an aid to doctors who are labeling or interpreting this data. Then, upon making a prediction, the model not only gives you a probabilistic output of what it thinks is true, but also looks up the nearest N neighbors neighbors in the embedding space (of the training set) and shows you those N samples as to say \"here is my prediction, but here are also 10 (for example) other EEGs and spectrograms that have similar patterns to this one and hence why I labeled it this way.\"
\nI have a lot of ideas like this that I'm going to be exploring over this summer as a side project (including reading/implementing other unique top submissions). I'll also be open sourcing it, so feel free to monitor my GitHub for updates (as well as the code for this project, which should be up in 2-3 days).
\nCongrats to the winners!
\nThanks Kaggle, and Hosts! This competition was by far one of the most fun I have joined. I learned a ton.
\nGitHub Training Code: https://github.com/ryanirl/hbac
\nKaggle Final Inference: https://www.kaggle.com/code/ryanirl/hms-baseline-sub
Ryan Peters
\nRyanIRL@icloud.com
Designed a custom kNN knowledge distillation scheme to pseudo-label samples with <10 votes. I won't go into it here, but I choose kNN so the entropy of labels would match almost exactly. I couldn't get this to increase my personal CV, but it did speed up training/convergence a lot which allowed me to rapidly test out ideas.
\nWhat does this mean?
", "rawMarkdown": "Designed a custom kNN knowledge distillation scheme to pseudo-label samples with <10 votes. I won't go into it here, but I choose kNN so the entropy of labels would match almost exactly. I couldn't get this to increase my personal CV, but it did speed up training/convergence a lot which allowed me to rapidly test out ideas.\n\nWhat does this mean? ", "votes": 1, "replies": [ { "id": 2766481, "postDate": "2024-04-21T18:00:28.650Z", "content": "Sorry for the late response. Basically what I did for pseudo-labeling was the following:
\n\n\n x, y validation_dataloader: \n embed = model.generate_embedding(x)\n db.((embed, y))\n x, y train_dataloader:\n embed = model.generate\n db_ys, db_embeds = db.k\n new_y = db_ys.mean(axis = ) # `new_y` becomes our pseudo-label.\nThe key here is that when generating the embeddings in the first step (validation step)… we ONLY use the samples with count >= 10. This ensures that we are only sampling from high quality labels.
\nThe reason I chose kNN pseudo-labeling over traditional knowledge-distillation was two-fold:
\nGreat work!
", "rawMarkdown": "Great work!", "votes": 1 }, { "id": 2742704, "postDate": "2024-04-09T03:14:29.767Z", "content": "great work!
", "rawMarkdown": "great work!", "votes": 1 }, { "id": 2742622, "postDate": "2024-04-09T01:58:40.993Z", "content": "this is an amazing solution, congrats
", "rawMarkdown": "this is an amazing solution, congrats", "votes": 1 }, { "id": 2742582, "postDate": "2024-04-09T01:15:53.693Z", "content": "Great work!
\n\n\nNotice that when vertically flipping, you must also reverse the sign of the data because the order of subtraction is changed.
\n
What does this mean?
\n\n\nUnfreeze with very small LR and retrain again
\n
How do you achieve this?
", "rawMarkdown": "Great work!\n \n> Notice that when vertically flipping, you must also reverse the sign of the data because the order of subtraction is changed.\n\nWhat does this mean?\n\n> Unfreeze with very small LR and retrain again\n\nHow do you achieve this?", "votes": 1, "replies": [ { "id": 2742612, "postDate": "2024-04-09T01:46:47.457Z", "content": "\n\nNotice that when vertically flipping, you must also reverse the sign of the data because the order of subtraction is changed.
\n
I am using the bipolar double banana montage which means that for each channel in the EEG data (let's say Fp1 and F7), we normalize each channel by another channel to get a relative value. This is done by subtracting them to get F7-Fp1 (or the other way around). But when you vertically flip the montage, the ordering of subtraction is also reversed (because I normalize vertically). So after vertically flipping, Fp7-Fp1 becomes Fp1-F7. So to fix this, you just flip the sign. At the end of the day, I don't think the flipping of signs is really all that crutial. You could probably even have an augmentation that reverses the sign of the whole montage and it would most likely still work out to be about the same. I just did it for consistency. Here is also an image to demonstrate the idea.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9041364%2F0852e7fba6dc02ae97821bc3c168285f%2FUntitled%20drawing%20(10).png?generation=1712626749233600&alt=media\")
\n\nUnfreeze with very small LR and retrain again.
\n
Not sure exactly what your asking, but I probably didn't word this too well 😅. First, I trained 3 models for each of the modalities that I am considering (EEG, EEG spectrograms, and the Kaggle provided spectrograms). Then I take each of the 3 models that are pretrained and I combine them into my final multimodal model explained in my post. But I only train the decoder head (attention layer on the end), as I freeze the weights of the 3 models I previously trained. The reason being that I want the model to focus on learning how to combine the embeddings in the final attention layer and not learn to encode the data (which is the frozen and pretrained backbones job). Once the final attention head is performing well, I unfreeze the whole model (meaning training the whole model now and no longer freezing the backbone) and retrain on the data again but with a really small learning rate (LR) so the model doesn't make any dramatic changes as this is just a fine-tuning step.
", "rawMarkdown": "> Notice that when vertically flipping, you must also reverse the sign of the data because the order of subtraction is changed.\n\nI am using the [bipolar double banana montage](https://www.learningeeg.com/montages-and-technical-components) which means that for each channel in the EEG data (let's say Fp1 and F7), we normalize each channel by another channel to get a relative value. This is done by subtracting them to get F7-Fp1 (or the other way around). But when you vertically flip the montage, the ordering of subtraction is also reversed (because I normalize vertically). So after vertically flipping, Fp7-Fp1 becomes Fp1-F7. So to fix this, you just flip the sign. At the end of the day, I don't think the flipping of signs is really all that crutial. You could probably even have an augmentation that reverses the sign of the whole montage and it would most likely still work out to be about the same. I just did it for consistency. Here is also an image to demonstrate the idea.\n\n.png?generation=1712626749233600&alt=media)\n\n> Unfreeze with very small LR and retrain again.\n\nNot sure exactly what your asking, but I probably didn't word this too well 😅. First, I trained 3 models for each of the modalities that I am considering (EEG, EEG spectrograms, and the Kaggle provided spectrograms). Then I take each of the 3 models that are pretrained and I combine them into my final multimodal model explained in my post. But I only train the decoder head (attention layer on the end), as I freeze the weights of the 3 models I previously trained. The reason being that I want the model to focus on learning how to combine the embeddings in the final attention layer and not learn to encode the data (which is the frozen and pretrained backbones job). Once the final attention head is performing well, I unfreeze the whole model (meaning training the whole model now and no longer freezing the backbone) and retrain on the data again but with a really small learning rate (LR) so the model doesn't make any dramatic changes as this is just a fine-tuning step.", "replies": [ { "id": 2742618, "postDate": "2024-04-09T01:54:52.853Z", "content": "Oh I see. Thanks for clarification!
", "rawMarkdown": "Oh I see. Thanks for clarification!", "votes": 1 } ] } ] }, { "id": 2742504, "postDate": "2024-04-09T00:22:14.533Z", "content": "That was fast! Thank you so much for this quick sharing! This is really cool multimodal model. Congrats, well deserved!
", "rawMarkdown": "That was fast! Thank you so much for this quick sharing! This is really cool multimodal model. Congrats, well deserved!", "votes": 1, "replies": [ { "id": 2742511, "postDate": "2024-04-09T00:27:06.450Z", "content": "Thanks! I've been writing it for the past couple of hours now 😅.
", "rawMarkdown": "Thanks! I've been writing it for the past couple of hours now 😅.", "votes": 1, "replies": [ { "id": 2742533, "postDate": "2024-04-09T00:40:03.793Z", "content": "Thanks for the great writeup. I am a little curious of how do you make such a beautiful graph of the model architecture?🤩
", "rawMarkdown": "Thanks for the great writeup. I am a little curious of how do you make such a beautiful graph of the model architecture?🤩" }, { "id": 2742537, "postDate": "2024-04-09T00:43:08.277Z", "content": "Just spent a couple of hours in google drawings 😂.
", "rawMarkdown": "Just spent a couple of hours in google drawings 😂." } ] } ] }, { "id": 2742503, "postDate": "2024-04-09T00:22:06.693Z", "content": "Congratulations, impressive solution! I'd love to see how you accelerated it with KNN. How can I follow you on GitHub? Thank you.
", "rawMarkdown": "Congratulations, impressive solution! I'd love to see how you accelerated it with KNN. How can I follow you on GitHub? Thank you.", "votes": 1, "replies": [ { "id": 2742514, "postDate": "2024-04-09T00:28:46.777Z", "content": "Here's the link: https://github.com/ryanirl
\nI'll have all of the code posted within 2-3 days, as soon as I finish migrating all of the code from my private repo to a public one.
", "rawMarkdown": "Here's the link: https://github.com/ryanirl\n\nI'll have all of the code posted within 2-3 days, as soon as I finish migrating all of the code from my private repo to a public one.", "votes": 2, "replies": [ { "id": 2742676, "postDate": "2024-04-09T02:53:22.180Z", "content": "Big thanks
", "rawMarkdown": "Big thanks" }, { "id": 2756661, "postDate": "2024-04-17T06:03:56.337Z", "content": "Looking forward to！Thank you.
", "rawMarkdown": "Looking forward to！Thank you." } ] } ] }, { "id": 3159058, "postDate": "2025-03-25T08:04:27.267Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2744531, "author_name": "MeiYuxin", "author_url": "", "post_date": "2024-04-10T01:18:27.393000", "content": "Designed a custom kNN knowledge distillation scheme to pseudo-label samples with <10 votes. I won't go into it here, but I choose kNN so the entropy of labels would match almost exactly. I couldn't get this to increase my personal CV, but it did speed up training/convergence a lot which allowed me to rapidly test out ideas.
\nWhat does this mean?
", "votes": 1, "replies": [ { "id": 2766481, "author_name": "ryan", "author_url": "", "post_date": "2024-04-21T18:00:28.650000", "content": "Sorry for the late response. Basically what I did for pseudo-labeling was the following:
\n\n\n x, y validation_dataloader: \n embed = model.generate_embedding(x)\n db.((embed, y))\n x, y train_dataloader:\n embed = model.generate\n db_ys, db_embeds = db.k\n new_y = db_ys.mean(axis = ) # `new_y` becomes our pseudo-label.\nThe key here is that when generating the embeddings in the first step (validation step)… we ONLY use the samples with count >= 10. This ensures that we are only sampling from high quality labels.
\nThe reason I chose kNN pseudo-labeling over traditional knowledge-distillation was two-fold:
\nGreat work!
", "votes": 1, "replies": [] }, { "id": 2742704, "author_name": "MiHu", "author_url": "", "post_date": "2024-04-09T03:14:29.767000", "content": "great work!
", "votes": 1, "replies": [] }, { "id": 2742622, "author_name": "MengYe", "author_url": "", "post_date": "2024-04-09T01:58:40.993000", "content": "this is an amazing solution, congrats
", "votes": 1, "replies": [] }, { "id": 2742582, "author_name": "Roy Wei", "author_url": "", "post_date": "2024-04-09T01:15:53.693000", "content": "Great work!
\n\n\nNotice that when vertically flipping, you must also reverse the sign of the data because the order of subtraction is changed.
\n
What does this mean?
\n\n\nUnfreeze with very small LR and retrain again
\n
How do you achieve this?
", "votes": 1, "replies": [ { "id": 2742612, "author_name": "ryan", "author_url": "", "post_date": "2024-04-09T01:46:47.457000", "content": "\n\nNotice that when vertically flipping, you must also reverse the sign of the data because the order of subtraction is changed.
\n
I am using the bipolar double banana montage which means that for each channel in the EEG data (let's say Fp1 and F7), we normalize each channel by another channel to get a relative value. This is done by subtracting them to get F7-Fp1 (or the other way around). But when you vertically flip the montage, the ordering of subtraction is also reversed (because I normalize vertically). So after vertically flipping, Fp7-Fp1 becomes Fp1-F7. So to fix this, you just flip the sign. At the end of the day, I don't think the flipping of signs is really all that crutial. You could probably even have an augmentation that reverses the sign of the whole montage and it would most likely still work out to be about the same. I just did it for consistency. Here is also an image to demonstrate the idea.
\n
\n\nUnfreeze with very small LR and retrain again.
\n
Not sure exactly what your asking, but I probably didn't word this too well 😅. First, I trained 3 models for each of the modalities that I am considering (EEG, EEG spectrograms, and the Kaggle provided spectrograms). Then I take each of the 3 models that are pretrained and I combine them into my final multimodal model explained in my post. But I only train the decoder head (attention layer on the end), as I freeze the weights of the 3 models I previously trained. The reason being that I want the model to focus on learning how to combine the embeddings in the final attention layer and not learn to encode the data (which is the frozen and pretrained backbones job). Once the final attention head is performing well, I unfreeze the whole model (meaning training the whole model now and no longer freezing the backbone) and retrain on the data again but with a really small learning rate (LR) so the model doesn't make any dramatic changes as this is just a fine-tuning step.
", "votes": 0, "replies": [ { "id": 2742618, "author_name": "Roy Wei", "author_url": "", "post_date": "2024-04-09T01:54:52.853000", "content": "Oh I see. Thanks for clarification!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2742504, "author_name": "Steven_Y", "author_url": "", "post_date": "2024-04-09T00:22:14.533000", "content": "That was fast! Thank you so much for this quick sharing! This is really cool multimodal model. Congrats, well deserved!
", "votes": 1, "replies": [ { "id": 2742511, "author_name": "ryan", "author_url": "", "post_date": "2024-04-09T00:27:06.450000", "content": "Thanks! I've been writing it for the past couple of hours now 😅.
", "votes": 1, "replies": [ { "id": 2742533, "author_name": "Roy Wei", "author_url": "", "post_date": "2024-04-09T00:40:03.793000", "content": "Thanks for the great writeup. I am a little curious of how do you make such a beautiful graph of the model architecture?🤩
", "votes": 0, "replies": [] }, { "id": 2742537, "author_name": "ryan", "author_url": "", "post_date": "2024-04-09T00:43:08.277000", "content": "Just spent a couple of hours in google drawings 😂.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2742503, "author_name": "kerry sun", "author_url": "", "post_date": "2024-04-09T00:22:06.693000", "content": "Congratulations, impressive solution! I'd love to see how you accelerated it with KNN. How can I follow you on GitHub? Thank you.
", "votes": 1, "replies": [ { "id": 2742514, "author_name": "ryan", "author_url": "", "post_date": "2024-04-09T00:28:46.777000", "content": "Here's the link: https://github.com/ryanirl
\nI'll have all of the code posted within 2-3 days, as soon as I finish migrating all of the code from my private repo to a public one.
", "votes": 2, "replies": [ { "id": 2742676, "author_name": "kerry sun", "author_url": "", "post_date": "2024-04-09T02:53:22.180000", "content": "Big thanks
", "votes": 0, "replies": [] }, { "id": 2756661, "author_name": "Kris_Lan", "author_url": "", "post_date": "2024-04-17T06:03:56.337000", "content": "Looking forward to！Thank you.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3159058, "author_name": "", "author_url": "", "post_date": "2025-03-25T08:04:27.267000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2742494": "Huge thanks to the Hosts and Kaggle, this competition was by far one of the most fun competitions I've competed in awhile. \n\n